Ammar Sabilarrohman
In an era where major technology conglomerates are increasingly criticized for "shredding" rare books—purchasing physical copies only to destroy them in the process of scanning for proprietary AI training datasets—a quiet, grassroots movement in Pakistan has offered a profound counter-narrative. For a decade, a trio of friends dedicated their personal time, resources, and technical ingenuity to the Ibteda Digital Library, a monumental effort to digitize out-of-print Urdu literature, much of which existed only in fragile, lithographic form.
This project, born from a simple love for language and culture, has resulted in the preservation of hundreds of thousands of pages. While the team officially concluded their decade-long labor of love in April 2026, their legacy is not merely the archive itself, but a sophisticated, machine-learning-driven process that could redefine how historians and librarians approach the digitization of rare, non-Latin scripts.
The Genesis of an Archival Mission: Chronology of the Effort
The Ibteda Digital Library did not begin in a high-tech laboratory; it began in the homes of three friends around 2015. With no external funding, no institutional backing, and no formal roadmap, the team relied entirely on their own pockets and a deep commitment to preserving Urdu heritage.
The Manual Era (2015–2020)
The early years were characterized by "brute force" archival techniques. The team utilized basic consumer-grade equipment: a Nikon D5300 camera, a DIY lighting setup consisting of LED bulbs, and a repurposed glass sheet salvaged from an old photocopier. This glass served a vital function—holding fragile, yellowing pages flat against a surface to ensure the sharpest possible image quality.
The process was entirely manual. Every page turn was performed by hand; every framing adjustment was made by eye. Following the photography phase, the team spent countless hours in Adobe Photoshop, performing meticulous post-processing to clean up images, correct perspective distortion, and ensure uniform margins. This period was defined by the sheer volume of labor: the Nikon D5300 alone eventually clocked over 576,000 shutter counts, a testament to the immense scale of the work.
The Scale-Up and the "Corner Case" Reality (2020–2025)
As the project grew, so did the complexity. The team added a Nikon D3300 to their inventory, which would eventually accumulate another 326,000 shutter counts. By the time the project reached its conclusion, the team had processed over 526,000 dual-page images.
The primary hurdle was the nature of the material. Unlike standardized modern paperbacks, these works ranged from centuries-old lithographs to handwritten notes and rare pamphlets. Each book presented unique challenges: varied paper thickness, inconsistent gutter widths, and the specific demands of the Urdu Nastaliq script.
The Technical Landscape: Why Urdu Digitization Is Unique
To understand the magnitude of this achievement, one must understand the linguistic and physical obstacles inherent in the Urdu script. Nastaliq is an elegant, flowing, and calligraphic style of writing. Unlike the rigid, modular nature of Latin-alphabet printing, Nastaliq’s beauty lies in its complexity.
The Challenge of Nastaliq
The script features an intricate system of dots, diacritics, and small symbols. In the context of digital scanning, these features are easily confused with "noise"—dirt, dust, paper blemishes, or ink bleed-through. When digitizing old lithographs, where the ink has faded or spread, distinguishing a crucial diacritic mark from a stray smudge becomes a high-stakes task. If an automated process is too aggressive, it risks erasing the very language it intends to preserve.
The "Homography" Breakthrough
As the sheer volume of photos became impossible to process manually, the team pivoted toward automation. The researcher leading the technical side turned to OpenCV, a standard library for computer vision. However, standard methods failed; a logic flow that worked perfectly for a 19th-century poem collection failed entirely when applied to a 20th-century political tract.
The breakthrough came when the team realized they were sitting on a "gold mine" of training data: their own years of manual, pixel-perfect Photoshop work. By treating their manually finished pages as "labels," they were able to train a neural network to calculate the homography—the geometric transformation—required to align and crop each page.
Data Points: The Scale of the Ibteda Project
The statistics behind the Ibteda Digital Library provide a staggering look at what a small, dedicated team can achieve:
- Total Shutter Counts: Over 900,000 between two primary cameras (D5300 and D3300).
- Archival Volume: Over 526,000 dual-page captures.
- Duration: 10 years of consistent, part-time, self-funded effort.
- Storage Infrastructure: The collection is now hosted on a ZFS storage pool, utilizing BLAKE3 cryptographic manifests to ensure data integrity and detect any potential bit-rot over the long term.
Perhaps most fascinatingly, the team discovered that "more is not always better" when training their model. While one might assume that feeding more books into the AI would improve accuracy, the researchers found that because every book had unique, editor-specific cropping margins, adding too much data actually confused the model. They ultimately arrived at a hybrid workflow: ten manual calibration crops are performed for each new book, after which the AI handles the bulk of the remaining work.
A Contrast to Corporate AI Practices
The Ibteda project stands in stark, moral contrast to the current trend in the tech industry. Recently, reports have surfaced indicating that major AI companies are engaging in the mass acquisition of rare books for the sole purpose of feeding them into LLMs (Large Language Models). In many cases, these books are shredded or destroyed after being scanned in high-speed, automated facilities, removing them from circulation forever.
While these corporations treat books as disposable raw material for data extraction, the Ibteda team treated them as cultural artifacts. Their focus was not on "ingesting" data for a chatbot, but on "preserving" the original visual and literary integrity of the works for future generations to read, study, and cherish.
Implications for Global Archiving
The methodology developed by the Ibteda team—specifically the use of manually processed "ground truth" data to train neural networks for perspective correction and cropping—holds significant promise for international digital humanities.
Many cultures possess historical archives in scripts that are not well-supported by mainstream, Western-centric OCR (Optical Character Recognition) or digitization tools. By open-sourcing the logic behind their journey, the Ibteda team has provided a roadmap for others:
- Low-Cost Hardware: Demonstrating that professional-grade results are possible with entry-level DSLR cameras and DIY lighting.
- Model Efficiency: Showing that "small data" (high-quality, human-curated examples) can outperform massive, generalized datasets when dealing with specific, non-standard layouts.
- Preservation Philosophy: Advocating for the long-term stewardship of data (ZFS, BLAKE3) rather than the ephemeral, "train-and-discard" model currently favored by Silicon Valley.
Conclusion: The Living Archive
As of April 2026, the active phase of the Ibteda Digital Library has come to an end, but the library itself remains a vibrant, living resource. Hosted at the Internet Archive, the collection is a testament to what human passion, when coupled with technical ingenuity, can achieve in the face of indifference.
For those interested in the technical minutiae, the team has published a detailed account of their journey. For the rest of the world, the Ibteda Digital Library offers something more profound: a chance to read the beautiful, flowing script of Urdu poetry and prose that might otherwise have been lost to the ravages of time or the greed of industrial AI expansion. The trio of friends may have put down their cameras, but the stories they rescued are now safe for the next hundred years.
Jia Lissa
Lina Hope
