TamilSynth: A Synthetic Line Corpus for Tamil OCR and the Three Defaults That Decide Whether It Helps
This paper introduces TamilSynth, a large-scale synthetic corpus for Tamil OCR, and demonstrates that correcting three specific, silent defaults in the standard Tesseract training workflow—rather than the corpus design itself—is critical to achieving significant error reductions on real-world documents.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine trying to teach a computer to read a language where the letters change shape depending on how they are grouped together. In many writing systems, a single sound is made by combining a consonant with a vowel mark that sits above, below, or beside it. To a human eye, this combination looks like one single letter. To a computer, however, it is often just a string of separate code points sitting next to each other. This is the challenge of Optical Character Recognition, or OCR, for scripts like Tamil. When a machine scans a page, it must not only see the ink but also understand how these separate pieces fit together to form a word. For languages with vast amounts of digital text, this is a solved problem. But for low-resource languages, where there are not enough scanned books to train the software, researchers often turn to a clever shortcut: they create synthetic data. Instead of scanning real pages, they use computers to generate thousands of perfect images of text, complete with the exact answer key for what each image says. It is a standard trick in the field, but it assumes that if you feed the computer enough examples, it will learn to read.
A researcher named SyedKhaleel Jageer set out to test this assumption for Tamil. He built a massive library of synthetic text, creating over 218,000 lines of printed Tamil words in 29 different fonts. The goal was to use this library to teach a standard reading engine, known as Tesseract, to recognize Tamil text better than it could on its own. The expectation was that more data would equal better results. However, when Jageer first tried to use his new library, the results were disastrous. The computer performed worse than the original, untrained version of the software. It was a confusing failure. The data looked perfect, the images were clear, and the answers were correct, yet the machine could not learn. The problem was not the data itself, but three silent, hidden settings in the software used to train the machine. These settings, which most users never change, were fundamentally broken for a writing system like Tamil. When Jageer fixed these three specific defaults, the same library of images suddenly became incredibly powerful, transforming the software from a failure into a tool that reads real-world documents with significantly higher accuracy.
The story of this discovery begins with the construction of the training library. Jageer gathered text from five open sources, including Tamil versions of Wikipedia and news sites, and arranged them into lines of text. He did not just copy and paste; he designed the layout to mimic how real paragraphs look, ensuring that lines could be short or long, just like on a real page. He then rendered these lines into images using 29 different typefaces, creating a dataset of over 218,000 unique lines. This was a huge resource, far larger than most previous attempts for any single Indian language. The plan was to take this library and fine-tune the Tesseract engine, a widely used tool for reading text, to specialize in Tamil. The process involves showing the engine the synthetic images and telling it what they say, allowing it to adjust its internal math to get better at the task.
When Jageer ran the standard training process, the results were shocking. The new model, trained on his massive library, scored worse than the stock version of Tesseract that came with the software. In fact, it was so bad that it never improved, even after running for a very long time. The researcher initially suspected the library itself was the problem. He thought perhaps the fonts were too strange or the text was not diverse enough. He spent time redesigning the library, but the results barely changed. The model remained stuck. The breakthrough came when he realized the issue was not the data, but the instructions given to the training software. The standard setup for training Tesseract has three default behaviors that work fine for simple alphabets but fail completely for syllabic scripts like Tamil.
The first hidden setting involves how the software breaks down a line of text into smaller pieces for learning. In Tamil, a consonant and its attached vowel sign form a single unit of meaning, a grapheme. The standard software, however, was splitting these units apart, treating the consonant and the vowel sign as two separate items. This forced the computer to learn a broken version of the language. The second setting involved the internal code the software uses to store its knowledge. When the training started, the software was rebuilding this code from scratch based on the new data, effectively throwing away the knowledge it had already learned about Tamil. It was like trying to learn a new language by erasing your memory of the first one before you started. The third setting was a timing error. The software was programmed to stop training when it made the fewest mistakes on the practice data. But because the practice data was synthetic and perfect, the software kept training until it memorized the practice set, ignoring the fact that it was getting worse at reading real, messy documents.
Once Jageer corrected these three defaults, the outcome changed instantly. He told the software to keep the consonant-vowel units together, to preserve the existing knowledge base instead of rebuilding it, and to stop training based on how well it did on real-world test pages rather than the perfect practice pages. With these adjustments, the same 218,000 images that had previously failed became a goldmine. The new model reduced the number of errors it made on real documents by 24 percent compared to the standard version. It was able to read newsprint, books, and forms with much greater reliability. The improvement was so consistent that in a test of 10,000 different scenarios, the new model was better every single time.
The study also revealed that the common belief about what causes errors in Tamil reading was wrong. Many experts thought the biggest problem was confusing similar-looking consonants. However, the analysis showed that the real trouble was with the vowel signs, which are small marks that change the sound of a letter. The software was most likely to miss these small marks or misidentify their length. This finding suggests that future improvements should focus on how the software handles these specific details rather than trying to distinguish between similar consonants.
This work serves as a powerful reminder that in the world of artificial intelligence, having more data is not always the answer. The quality of the process matters just as much as the quantity of the material. Jageer's research demonstrates that for complex writing systems, the standard tools used to train computers often contain hidden traps that can ruin the results. By simply adjusting three settings that most people overlook, he turned a failed experiment into a significant success. The library of synthetic text he created is now available for others to use, but the true value of his work lies in the discovery of these three silent defaults. For anyone trying to teach a computer to read a language where letters are built from multiple parts, the lesson is clear: check the settings before you blame the data.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.