Naamah: A Large Scale Synthetic Sanskrit NER Corpus via DBpedia Seeding and LLM Generation

1. Problem Statement

The paper addresses the critical scarcity of annotated resources for Named Entity Recognition (NER) in Classical Sanskrit, a language central to South Asian intellectual history but underrepresented in modern NLP.

• Linguistic Complexity: Sanskrit is morphologically rich, relying on complex case markers (vibhakti) and phonetic fusion (sandhi). A single entity (e.g., "Rama") can have over 24 surface forms depending on syntactic role. Standard rule-based systems or string matching fail to capture these variations, especially when entities are merged with adjacent words via sandhi.
• Data Scarcity: Manual annotation requires high-level domain expertise, making gold-standard corpora expensive and slow to create. Existing datasets are small, domain-specific, or suffer from class imbalance.
• Limitations of Current Methods:
  • Cross-lingual Projection: Projecting labels from high-resource languages (like English) introduces alignment noise due to structural mismatches (e.g., word order differences).
  • Generic LLMs: Using general-purpose Large Language Models for data generation often results in grammatical errors because they lack grounding in Indic scripts and classical grammar.

2. Methodology

The authors propose a hybrid pipeline to generate a high-quality "silver standard" dataset, named Naamah, bypassing the need for manual annotation while ensuring grammatical naturalness.

A. Entity Seeding (DBpedia Mining)

To ensure lexical diversity and prevent the model from simply memorizing a few traditional names, the authors extracted entity seeds from DBpedia using SPARQL queries.

• Categories: Person (PER), Location (LOC), and Organization (ORG).
• Diversity Strategy: The extraction included a mix of classical Indian entities and global entities (e.g., modern international locations, foreign political figures) transliterated into Devanagari. This forces the model to learn syntactic patterns and case markers rather than relying on lexical familiarity.

B. Generative Pipeline

Instead of using rigid, pre-programmed morphological templates, the authors utilized Sarvam-M, a 24-billion-parameter hybrid reasoning model optimized for Indic languages.

• Process: The model was prompted to incorporate the DBpedia entity seeds into semantically coherent Sanskrit sentences.
• Advantage: This approach allows for the natural emergence of appropriate case endings and phrasing, generating a wide variety of syntactic structures and inflectional realizations that mimic authentic text.

C. Preprocessing and Filtering

Raw outputs underwent a Python-based heuristic preprocessing layer to ensure "silver standard" quality:

• Checks: Rule-based filtering for token-label consistency, malformed output, and ambiguous boundaries.
• Result: 102,942 high-quality sentences were retained.

D. Tokenization Strategy (De-Sandhi)

A key methodological choice was avoiding explicit de-sandhi preprocessing (splitting fused words before tagging).

• Approach: The models were trained on raw, unsplit text.
• Alignment: A "label-first" strategy was employed where the BIO tag is assigned only to the first subtoken of an entity, while subsequent subtokens (resulting from sandhi or agglutination) are masked with the ignore index (-100). This forces the model to learn to predict entity types based on the root stem while implicitly handling suffix structures.

3. Key Contributions

1. The Naamah Corpus: A large-scale, silver-standard dataset comprising 102,942 Sanskrit sentences with 127,397 total entities. It uses the standard BIO tagging scheme (B-PER, I-PER, B-LOC, etc.).
2. Hybrid Generation Methodology: A novel pipeline combining structured DBpedia mining with a domain-optimized LLM to generate syntactically diverse and grammatically natural data, overcoming the limitations of rigid templates and generic LLMs.
3. Benchmarking Insights: A comparative analysis demonstrating that for classical languages, domain-aligned tokenization is more critical than raw model scale.

4. Experimental Results

The authors benchmarked two transformer architectures on a fixed split (92,647 train / 10,295 validation):

• XLM-RoBERTa Base: A massive multilingual model (250k token vocab).
• IndicBERTv2: A parameter-efficient, Indic-focused model (~130 MB).

Performance Metrics (Validation Set):

Metric	XLM-RoBERTa	IndicBERTv2
F1 Score	0.9506	0.9615
Precision	0.9498	0.9596
Recall	0.9514	0.9633
Accuracy	0.9857	0.9889

Key Findings:

• IndicBERTv2 Outperforms: Despite being significantly smaller, IndicBERTv2 achieved a higher F1 score (0.9615 vs. 0.9506).
• Tokenizer Fragmentation: Qualitative analysis revealed a critical failure mode in XLM-RoBERTa. Its generic tokenizer fractures complex agglutinated Sanskrit terms (e.g., splitting Kuruksetre into Kuruksetra and e). The model often misclassified the suffix fragment as a separate entity (e.g., labeling the suffix as an "Organization").
• Semantic Integrity: IndicBERTv2, with its domain-adapted tokenizer, preserved the semantic integrity of the entire entity span, correctly handling subword transitions and suffixes.

5. Significance and Conclusion

• Bridging the Data Gap: Naamah provides a practical, scalable path to creating labeled data for classical languages where expert annotation is scarce.
• Model Architecture Insight: The study challenges the assumption that larger multilingual models are always superior for low-resource languages. It demonstrates that tokenizer alignment with the specific script and morphology of the target language is a primary determinant of success.
• Future Impact: The dataset serves as a foundational resource for downstream tasks like knowledge graph construction and digital philology. The authors plan to transition from this silver standard to a gold standard by incorporating expert-validated subsets and testing on complex sandhi resolution in authentic literature.

Availability: The dataset is publicly available on Hugging Face.