DohaScript: A Large-Scale Multi-Writer Dataset for Continuous Handwritten Hindi Text

This paper introduces DohaScript, a large-scale, multi-writer dataset of continuous handwritten Hindi text derived from 531 contributors transcribing identical traditional couplets, designed to address the scarcity of high-quality benchmarks for Devanagari script analysis and support diverse tasks like recognition, writer identification, and style modeling.

The Gist

Imagine you are trying to teach a robot to read handwritten notes. For languages like English, you have millions of practice books filled with different people's handwriting. But for Hindi, which uses the Devanagari script, the robot is like a student trying to learn to read with only a few torn scraps of paper.

This paper introduces DohaScript, a massive new "library" designed to fix that problem. Here is the story of how they built it and why it matters, explained simply.

1. The Problem: The "Missing Library"

Hindi is spoken by hundreds of millions of people, yet there is almost no high-quality, large-scale data for computers to learn from.

• The Old Way: Existing datasets were like a box of loose LEGO bricks. They had individual letters or short words, but they didn't show how words connect in a sentence.
• The Hindi Challenge: In Devanagari, letters aren't just sitting next to each other; they are glued together by a "roof" (called a shirorekha) that runs across the top of the word. It's like a train where the cars are fused together. If you try to teach a computer using isolated bricks, it won't understand how to read the train.
• The Result: Computers struggle to read continuous Hindi handwriting because they lack a big, diverse dataset to practice on.

2. The Solution: The "Doha" Experiment

The researchers created DohaScript. Think of this as a massive "copycat" game, but with a twist.

• The Script: They asked 531 different people to write the exact same six poems (called dohas, which are traditional rhyming couplets).
• Why Poems? These poems are famous, taught in schools, and contain every single letter and sound in the Hindi alphabet. It's like asking everyone to write the same alphabet soup, but in a sentence.
• The Magic: Because everyone wrote the same words, the researchers can now compare the handwriting styles directly. It's like having 531 different artists paint the exact same landscape. You can see exactly how Person A's brushstrokes differ from Person B's, without the confusion of different subject matter.

3. The "Quality Control" Filter

Collecting 531 handwritten pages is great, but what if some are blurry, smudged, or taken in the dark?

• The Robot Inspector: The team built a smart AI "inspector" (a CNN classifier) to grade every page.
• The Grading System: They didn't just throw away bad photos. Instead, they sorted them into buckets:
  • The "Gold Standard" Bucket: Crisp, clear, high-quality pages perfect for teaching a computer to read.
  • The "Real World" Bucket: Pages that are a bit blurry, shaky, or messy.
• Why keep the messy ones? Because real life is messy! By keeping the "bad" samples, the researchers can teach computers how to read handwriting even when the lighting is poor or the pen is running out of ink. It's like training a driver not just on a perfect race track, but also on rainy, pothole-filled streets.

4. The "Tangled Thread" Challenge

Even if the handwriting is clear, reading it is hard because of how the lines are arranged.

• The Analogy: Imagine trying to read a book where the lines of text are sometimes perfectly straight, sometimes wavy, and sometimes the letters from one line dip down and touch the line below.
• The Annotation: The researchers analyzed every page and labeled them by difficulty:
  • Easy: Neat lines, easy to separate.
  • Medium: A little wobbly.
  • Complex: A tangled mess where lines overlap and bleed into each other.
• The Goal: This helps researchers build systems that can untangle the "thread" of text, even when the writer didn't use a ruler.

5. Why This Matters (The "So What?")

This dataset is a game-changer for several reasons:

• For Reading (OCR): It helps computers finally read handwritten Hindi forms, medical prescriptions, and historical documents accurately.
• For Identity: Since everyone wrote the same words, the computer can learn to recognize who wrote it based on their unique style (like a digital fingerprint).
• For Art: It allows AI to generate new handwriting that looks like a specific person's style.
• For Fairness: It includes people from all over India, different ages, and genders, ensuring the technology works for everyone, not just a select few.

The Bottom Line

DohaScript is like giving the world of AI a massive, organized, and diverse training camp for reading Hindi. It moves us from trying to learn with a few scattered puzzle pieces to having the whole picture, complete with instructions on how to handle the messy, blurry, and difficult parts of real-life handwriting.

The dataset is now publicly available, meaning any researcher can download it and start building better tools to understand and preserve the written word of hundreds of millions of people.

Technical Summary

1. Problem Statement

Despite Hindi being the fourth most widely spoken language globally with hundreds of millions of Devanagari script users, handwritten Devanagari text remains severely underrepresented in public benchmark datasets. Existing resources suffer from several critical limitations:

• Scale and Scope: Most datasets focus on isolated characters or short words (e.g., IIIT-HW-Words, CALAM) rather than continuous text, failing to capture the complex structural dependencies of Devanagari (e.g., the shirorekha headline and ligatures).
• Lack of Diversity: Existing datasets often lack sufficient writer diversity and controlled lexical content, making it difficult to isolate handwriting style from linguistic content.
• Data Scarcity for Deep Learning: The absence of large-scale, diverse, and well-annotated datasets hinders the training of robust deep learning models for Optical Character Recognition (OCR), Handwritten Text Recognition (HTR), writer identification, and generative modeling.
• Real-world Complexity: Current resources do not adequately reflect the challenges of continuous, page-level handwriting, such as variable spacing, baseline drift, and layout irregularities.

2. Methodology

A. Data Collection Protocol

• Corpus Design: The authors selected six traditional Hindi dohas (couplets) by renowned poets (Kabir, Rahim, Tulsidas). This text was chosen because it is culturally familiar, copyright-free, and linguistically complex enough to cover the full Devanagari character set.
• Coverage: The corpus contains 89 words and 361 characters (excluding spaces), covering 55 distinct characters, including all consonants (across five articulation categories), vowels, diacritics (mātrās), halants, and common conjuncts.
• Participants: Data was collected from 531 unique contributors across India (varying by age, gender, and region).
• Task: Each participant transcribed the exact same set of six dohas on a single A4 sheet. This "parallel stylistic corpus" design allows for the systematic analysis of writer-specific variation independent of linguistic content.
• Demographics: Non-identifiable metadata (age, gender, state) was collected. The cohort includes 396 males and 135 females, with ages ranging from students to adults.

B. Quality Curation Pipeline

To ensure the dataset is suitable for modern learning architectures, a rigorous, automated quality assessment pipeline was developed:

1. Initial Quantification: A Laplacian variance metric was used to measure image sharpness (blur score). Scores ranged widely (9.7 to 13,865), reflecting real-world acquisition variations (mobile photos vs. scanners).
2. CNN-Based Classification: Recognizing that blur scores alone are insufficient (as they miss ink density, noise, and continuity), the authors trained Convolutional Neural Networks (CNNs) to classify image quality.
   • Binary Classifier: Distinguishes between "Low/Medium" (<3000 blur score) and "High" (≥3000) quality.
   • Four-Class Classifier: Stratifies samples into Low, Medium, Good, and Excellent.
3. Filtering: A high-quality core subset of 288 images (54.2% of the total) was retained for primary training, while the remaining 243 degraded samples were kept to serve as a challenging subset for testing model robustness.

C. Layout and Segmentation Analysis

To address the difficulty of line segmentation in continuous Devanagari text:

• Heuristic Evaluation: A hybrid heuristic pipeline was applied to all 531 pages to detect text lines.
• Difficulty Annotation: Based on line count accuracy and structural regularity, pages were annotated with three difficulty levels: Easy (20.7%), Medium (26.6%), and Complex (52.7%).
• Findings: Even visually clear images often present segmentation challenges due to intrinsic handwriting properties like overlapping strokes, irregular baselines, and shirorekha continuity between lines.

3. Key Contributions

1. DohaScript Dataset: The first large-scale, multi-writer dataset for continuous handwritten Hindi text, comprising 531 unique writer samples of identical text.
2. Parallel Stylistic Corpus: A unique design where lexical content is fixed across all writers, enabling precise isolation of handwriting style for tasks like writer identification and style transfer.
3. Comprehensive Metadata: Inclusion of non-identifiable demographic data (age, gender, region) and page-level difficulty annotations (Easy/Medium/Complex).
4. Automated Quality Framework: A novel pipeline combining Laplacian variance and CNN-based classification to objectively filter and stratify data quality, providing both a "clean" benchmark and a "degraded" real-world test set.
5. Open Access: Full release of the dataset (Google Drive) and all code for preprocessing, quality assessment, and experimentation (GitHub) to ensure reproducibility.

4. Results and Evaluation

• Quality Classification Performance:
  • The binary CNN classifier achieved 96.26% test accuracy with balanced precision and recall (F1 ≈ 0.96).
  • The four-class model achieved 85.98% accuracy, with the "Medium" class showing lower performance due to boundary ambiguity with the "Good" class.
• Dataset Statistics:
  • Total Samples: 531 pages.
  • Word Instances: 47,259 total words across the dataset.
  • Quality Split: The filtered high-quality subset contains 157 "Good" and 131 "Excellent" samples.
• Segmentation Difficulty:
  • Only 29.57% of pages achieved perfect line segmentation (12/12 lines).
  • The majority (52.7%) fell into the "Complex" category, highlighting that structural variability (spacing, baseline drift) remains a significant challenge even after quality filtering.

5. Significance and Impact

• Benchmarking Standard: DohaScript establishes a standardized, reproducible benchmark for continuous Devanagari handwriting, addressing the fragmentation of previous research efforts.
• Advancing Low-Resource Research: It provides the necessary scale and diversity to train deep learning models that generalize well across different writers and writing styles, a critical need for Indic scripts.
• Enabling New Research Directions: The dataset supports advanced tasks previously difficult for Hindi, including:
  • Writer Identification & Biometrics: Leveraging identical text to isolate biometric features.
  • Generative Modeling: Facilitating handwriting synthesis and style transfer due to repeated lexical content.
  • Document Layout Analysis: Providing a stratified benchmark for line segmentation and layout understanding in complex scripts.
• Real-World Applicability: By including a "degraded" subset and difficulty annotations, the dataset allows researchers to develop and test models robust to real-world acquisition noise and structural irregularities.

In conclusion, DohaScript fills a critical gap in the field of Indic OCR and HTR by providing a large-scale, controlled, and richly annotated resource that moves beyond isolated character recognition to the complexities of continuous, multi-writer document analysis.