LEVOS: Leveraging Vocabulary Overlap with Sanskrit to Generate Technical Lexicons in Indian Languages

1. Problem Statement

The paper addresses the challenge of translating technical terms from English into low-resource Indian languages. Key difficulties include:

• Limited Parallel Data: There is a scarcity of high-quality, parallel corpora for technical domains in many Indian languages.
• Linguistic Complexity: Indian languages often form compound words through sandhi (phonetic changes at word boundaries) and morphological variations, making simple translation difficult.
• Resource Disparity: While Hindi has relatively abundant digital resources, other regional languages (e.g., Marathi, Kannada, Odia) suffer from a "digital data divide," hindering the creation of technical lexicons for education and professional use.
• Ambiguity: Technical terms often have multiple translations depending on context (e.g., "mass" in physics vs. "mass" in demographics), which standard Neural Machine Translation (NMT) models struggle to resolve without sufficient domain-specific training data.

2. Methodology

The authors propose LEVOS, a framework that leverages the high vocabulary overlap between Sanskrit and modern Indian languages (both Indo-Aryan and Dravidian) to inform translation. The methodology consists of two main components:

A. Sanskrit Word Segmentation (CharSS)

To handle the complexity of Sanskrit compounds and sandhi, the authors developed a Character-level Transformer model (CharSS) based on ByT5 (Byte-Level Text-to-Text Transfer Transformer).

• Task: The model performs Sanskrit Word Segmentation (SWS), splitting compound words into meaningful subword units (morphemes).
• Architecture: Unlike traditional tokenizers, ByT5 processes text as byte sequences, eliminating the need for language-specific tokenization. This allows it to handle diverse scripts and rare words effectively.
• Input/Output:
  • Input: A single Sanskrit compound word (unigram).
  • Output: Segmented sub-tokens concatenated with a + symbol, bounded by & (start) and $ (end) to delineate morpheme boundaries precisely.
• Training: The model was fine-tuned on benchmark datasets (UoH+SandhiKosh, SIGHUM, and a hackathon dataset) using the SLP1 script (a Romanized transliteration standard for Sanskrit).

B. Sanskrit-Based Input Augmentation for Translation

The core innovation is using the segmented Sanskrit subwords to augment the input for translating technical terms into low-resource languages.

• Workflow:
  1. Normalization: English technical terms are mapped to Hindi (a resource-rich language) using existing dictionaries.
  2. Lemmatization: Hindi-specific affixes are removed to extract the root lemma.
  3. Segmentation: The CharSS model segments the Hindi lemma into Sanskrit-based subword units.
  4. Augmentation: These segmented subwords are appended to the original English input.
• Rationale: Since many Indian languages share a significant lexical heritage with Sanskrit, providing the model with the underlying Sanskrit morphemes helps disambiguate the domain and guides the generation of linguistically accurate terms in the target language.

3. Key Contributions

1. CharSS Model: Introduction of a character-level Transformer (ByT5) for Sanskrit Word Segmentation that effectively handles sandhi splitting and morphological complexity, outperforming existing lexicon-driven and neural baselines.
2. Linguistically Informed Augmentation: A novel method to generate technical lexicons for low-resource languages by leveraging Hindi data and Sanskrit subword segmentation, effectively bridging the resource gap.
3. Empirical Validation: Comprehensive experiments demonstrating that this approach significantly improves translation quality (measured by chrF++) across multiple domains (Administrative, Biotechnology, Chemistry) and languages.

4. Experimental Results

The authors evaluated their approach using chrF++ (character n-gram F-score) and human evaluation.

• Sanskrit Word Segmentation (SWS):

  • On the UoH+SandhiKosh dataset, CharSS achieved 97.2% Location Prediction Accuracy (LPA) and 93.5% Split Prediction Accuracy (SPA), outperforming baselines like DD-RNN and Sandhi Prakarana.
  • On SIGHUM and Hackathon datasets, the model achieved competitive or superior results compared to state-of-the-art models like TransLIST.

• Technical Term Translation:

  • Setting 1 (Standard): Fine-tuning NLLB (No Language Left Behind) on 6 Indian languages.
    • Result: The "NLLB + Sanskrit" model achieved an average improvement of 8.46 chrF++ points over the baseline "NLLB" (English-only) model.
    • Domain Impact: Significant gains were observed in Biotechnology and Chemistry (e.g., Biotechnology average improved from 26.20 to 39.56), where technical vocabulary relies heavily on Sanskrit roots.
  • Setting 2 (Zero-Shot): Training on Hindi, Gujarati, and Tamil; testing on unseen languages (Marathi, Kannada, Odia).
    • Result: The augmented model consistently outperformed the baseline across all domains, proving the method's ability to capture multilingual nuances without direct training data for the target language.

• Human Evaluation (Post-hoc):

  • Evaluated on Punjabi and Malayalam.
  • The augmented approach achieved higher R@1 (Recall at 1) scores (0.53 for Punjabi, 0.46 for Malayalam) compared to the baseline LexGen method, indicating better selection of the most contextually appropriate term.
  • Qualitative analysis showed the augmented input helped the model disambiguate between synonyms (e.g., distinguishing "mass" as population vs. matter).

5. Significance and Implications

• Educational Inclusivity: The work directly supports the Indian government's initiative to introduce regional languages in higher education by enabling the creation of high-quality, accessible technical textbooks.
• Resource Efficiency: It offers a cost-effective solution for low-resource language communities by utilizing the "lexical bridge" of Sanskrit and the available resources of Hindi, rather than requiring massive new parallel corpora.
• Linguistic Preservation: By integrating morphological alignment and Sanskrit roots, the method promotes translations that are not just literal but linguistically rooted and culturally appropriate.

6. Limitations

• Dependency on Hindi: The method relies on the availability of Hindi data for the initial normalization step. While Hindi is resource-rich compared to other Indian languages, it is still less rich than English.
• Language Coverage: The approach is most effective for languages with significant Sanskrit vocabulary overlap. Languages with minimal Sanskrit influence may see limited benefits.
• Archaic Terms: In administrative domains, the model sometimes generates archaic Sanskrit-derived words that are no longer in common usage, which can lower automated metric scores despite being linguistically correct.