💬 NLP

From Lexicon to AI: A Structured-Data Pipeline for Specialized Conversational Systems in Low-Resource Languages

This paper presents a resource-efficient pipeline that transforms structured linguistic resources like Hindi WordNet into specialized conversational AI systems, demonstrating that expert-curated lexical databases can effectively replace massive training corpora to achieve superior pedagogical performance in low-resource languages.

Original authors: Siddhant Hitesh Mantri, Dhara Gorasiya, Malhar Kulkarni, Pushpak Bhattacharya

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Siddhant Hitesh Mantri, Dhara Gorasiya, Malhar Kulkarni, Pushpak Bhattacharya

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to build a super-smart tutor for a language that doesn't have many books, websites, or digital textbooks available. Usually, to teach an AI, you need to feed it a massive library of text—like millions of books and websites. But for many languages, that "library" simply doesn't exist.

This paper presents a clever workaround. Instead of trying to find a massive library, the researchers built a specialized AI tutor using a single, high-quality "dictionary of connections" (called a WordNet) as their foundation.

Here is the story of how they did it, broken down into simple concepts:

1. The Problem: The "Empty Library"

Think of building an AI like training a dog. To teach a dog complex tricks, you usually need thousands of practice sessions with different people and situations. For languages like Hindi (and hundreds of others), there aren't enough "practice sessions" (digital text) available to train a general AI to be a good teacher.

2. The Solution: The "Master Blueprint"

Instead of a messy library, the researchers used a WordNet.

The Analogy: Imagine a standard dictionary just lists words and definitions. A WordNet is more like a giant, organized family tree for words. It knows that a "dog" is a type of "animal," that "hot" is the opposite of "cold," and that a "wheel" is part of a "car."
The Innovation: The researchers took the Hindi WordNet (which has over 100,000 words and their relationships) and used a robot to turn it into 1.25 million practice questions and answers. They didn't just copy the dictionary; they turned the "family tree" into a conversation.
- Example: Instead of just saying "Dog is an animal," the AI learns to say, "If you ask me about a dog, I can tell you it's an animal, it has a tail, and it's the opposite of a cat in some ways."

3. The Training: "Fine-Tuning with a Tiny Spoon"

Usually, training a big AI requires a massive computer (like a supercomputer) and huge amounts of data.

The Analogy: Imagine you have a giant, all-knowing encyclopedia (a large AI model). Instead of rewriting the whole encyclopedia, the researchers used a tiny, precise tool (called LoRA) to add just the right "sticky notes" to the pages they needed.
They used a technique called 4-bit quantization, which is like compressing a heavy suitcase so it fits in a small backpack. This allowed them to run the AI on a standard computer (using only 12GB of memory) instead of needing a massive data center.

4. The Result: The "Specialized Tutor" vs. The "General Know-It-All"

They tested their new AI, which they named Shabdabot, against famous, general-purpose AIs (like GPT-4 and Gemini). They asked it to act as a language teacher for students at different levels, from beginners to experts.

The General AIs: These were like brilliant scholars who knew a little bit about everything. They were good at understanding the meaning of words (Semantic Accuracy), but when it came to teaching a child or a beginner, they sometimes got too complicated or inconsistent.
Shabdabot (The Specialized Tutor): Because it was built directly from the structured "family tree" of words, it was a much better teacher.
- Pedagogical Score (Teaching Quality): Shabdabot scored 91.0, while the best general AI scored around 79.4.
- Consistency: This is the biggest win. The general AIs were like a mood ring; sometimes they gave a great answer, sometimes a confusing one. Shabdabot was 86% more consistent. It gave reliable, safe, and age-appropriate answers every single time.

5. The Big Takeaway

The paper argues that you don't always need a "big data" library to build a great AI. If you have a structured, expert-curated map of knowledge (like a WordNet), you can build a specialized AI that outperforms massive, general models in specific tasks (like education).

In short: They proved that for low-resource languages, a well-organized map of knowledge is often a better teacher than a messy pile of internet text. This opens the door to creating specialized AI tutors for hundreds of languages that currently have no digital presence, using only the linguistic maps that linguists have already built.

Technical Summary: From Lexicon to AI

Problem Statement

Low-resource languages face a critical barrier in developing specialized conversational AI systems: the lack of massive, high-quality training corpora. While large language models (LLMs) excel in high-resource languages, they often fail in specialized domains (such as education) for languages with limited digital content. Existing fine-tuning paradigms rely on unstructured web data, which is scarce for over 2,500 languages. Furthermore, even languages with substantial web presence, like Hindi, lack pedagogically structured examples necessary for specialized applications. Current approaches relying on cross-lingual transfer or prompting often suffer from hallucinations and factual inconsistencies in specialized domains because they depend on the model's latent, pre-trained knowledge rather than explicit, structured expert knowledge.

Methodology

The authors propose a systematic pipeline to transform structured linguistic resources (specifically WordNets) into specialized conversational AI systems. The methodology consists of four integrated stages:

1. Structured Knowledge Processing and Dataset Creation

The system converts the Hindi WordNet (containing 105,460 words and 40,466 synsets) into a diverse instruction-response dataset.

Data Generation: The pipeline generates four types of pairs to preserve semantic relationships:
- Basic Instructional Pairs: Fundamental Q&A for core concepts.
- Complex Multi-Aspect Pairs: Integrating definitions, synonyms, examples, and grammatical categories.
- Ontological Hierarchy Pairs: Teaching categorical relationships (hypernymy/hyponymy).
- Disambiguation Pairs: Addressing polysemy with contextual differentiation.
Quality Assurance: A dynamic chunking algorithm handles dense semantic relationships, splitting entries with >10 related terms into segments with 33% overlap to ensure continuity. The pipeline enforces coverage checks and removes duplicates via hashing, resulting in 1.25 million unique instruction-response pairs.

2. Resource-Efficient Model Specialization

Base Model: The authors utilize Gemma-3-12B-IT for its multilingual capabilities.
Optimization: To enable deployment in resource-constrained environments, the model is quantized to 4-bit (NF4) using double quantization, reducing memory requirements from 48GB to ~12GB.
Fine-Tuning: The model is specialized using Low-Rank Adaptation (LoRA) with a rank of 32 and alpha of 64. This approach tunes only 0.2% of the total parameters (67 million out of 12 billion), preserving pre-trained multilingual knowledge while adapting to the domain. Training was conducted on 2× NVIDIA A100 GPUs for approximately 40 hours.

3. Domain-Adaptive Response Generation

The system implements level-adaptive generation aligned with educational curricula, defining five proficiency levels (Beginner to Expert).

Prompt Engineering: Structured templates control vocabulary complexity, sentence length, and explanation depth for each level.
Safety: Safety constraints are embedded directly into the system prompt, restricting responses to family, school, and nature domains to ensure suitability for educational contexts.
System Name: The resulting fine-tuned model is referred to as Shabdabot.

4. Evaluation Framework

The system was evaluated against five models (Shabdabot, GPT-4.1, Claude-Sonnet-4, Gemini-2.5Pro, and the base Gemma-3-12B-IT) using 40 Hindi questions across five proficiency levels.

Semantic Answer Similarity (SAS): Measures semantic fidelity against expert golden answers using multilingual sentence transformers.
Level Adaptation Quality (LAQ): An automated expert judge (Claude-Sonnet-4) evaluates pedagogical effectiveness based on clarity, factual accuracy, relevance, language appropriateness, and educational value. Human linguists verified the reliability of this automated scoring.

Key Results

Pedagogical Effectiveness: Shabdabot achieved a Level Adaptation Quality (LAQ) score of 91.0, significantly outperforming all general-purpose models (which ranged from 79.4 to 83.6). This represents an 11.6-point advantage over the second-best model and a 12.6% improvement over its base model.
Consistency and Reliability: Shabdabot demonstrated exceptional stability with a standard deviation ( $\sigma$ ) of 1.0, compared to 7.4 for GPT-4.1. This indicates an 86% improvement in predictability, addressing a critical issue in educational AI where inconsistent responses can confuse learners.
Semantic Competence: While GPT-4.1 achieved the highest raw semantic similarity (0.762), Shabdabot maintained competitive semantic performance (0.731), showing that domain specialization did not degrade general linguistic competence.
Performance Patterns: Shabdabot uniquely peaked in performance at the "Advanced" level, though it showed a decline at the "Expert" level, attributed to the training data's emphasis on educational clarity over verbose academic discourse.

Significance and Claims

The paper claims to provide a proof-of-concept methodology for developing specialized AI systems in low-resource languages without relying on massive, unstructured corpora.

Paradigm Shift: The work challenges the assumption that general-purpose data produces the best specialized systems. It demonstrates that 1.25 million structured examples derived from expert-curated lexical databases can outperform models trained on orders of magnitude more unstructured text in specific domains.
Accessibility: By reducing computational requirements to ~12GB RAM and utilizing existing resources like WordNets (available for over 200 languages), the pipeline offers a practical, reproducible pathway for democratizing AI in resource-constrained settings.
Reliability: The study establishes that grounding generation in structured, expert knowledge fundamentally improves system reliability and consistency, making it a viable alternative to corpus-intensive approaches for educational and specialized applications.

Limitations

The authors acknowledge several constraints:

Expert-Level Performance: The system underperforms at the "Expert" level compared to "Advanced," likely due to the training data's focus on educational clarity rather than dense academic discourse.
Resource Dependence: The methodology relies on the availability and quality of structured resources like WordNets, which vary significantly across languages.
Scope: Evaluation was limited to Hindi and educational contexts; applicability to other language families (e.g., agglutinative or isolating languages) and long-term learning outcomes remains untested.