🤖 machine learning

TharuChat: Bootstrapping Large Language Models for a Low-Resource Language via Synthetic Data and Human Validation

This paper introduces TharuChat, a synthetic dataset created through an LLM-to-human bootstrapping pipeline to train Tharu-LLaMA (3B), a specialized instruction-following model that effectively addresses data scarcity and dialectal fragmentation for the low-resource Tharu language using consumer-grade hardware.

Original authors: Prajwal Panth, Agniva Maiti

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Prajwal Panth, Agniva Maiti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of Artificial Intelligence (AI) as a massive, high-tech library. For years, this library has been stocked with millions of books in English, Spanish, and Hindi. But for many indigenous languages, like Tharu (spoken by 1.7 million people in Nepal and India), the shelves are completely empty.

Because the AI "brain" (called a Large Language Model or LLM) has never read enough Tharu, when you ask it a question in Tharu, it gets confused. It starts speaking Tharu but then accidentally switches to Hindi or Nepali, effectively erasing the user's identity. This is the "digital divide."

This paper, "TharuChat," is a story about how two researchers built a new, specialized AI brain specifically for Tharu speakers, using a clever, low-budget method. Here is the breakdown in simple terms:

1. The Problem: The "Cold Start" Dilemma

To teach an AI a language, you usually need millions of examples (like books, websites, and tweets). But Tharu has very few written examples online.

The Catch-22: You can't train an AI without data, but you can't generate good data without a smart AI. It's like trying to bake a cake without flour, but you can't buy flour because the store is closed.

2. The Solution: The "Teacher-Student" Bootstrapping

The researchers used a clever trick called "LLM-to-Human" bootstrapping. Think of it like a master chef teaching an apprentice, who then helps the chef cook a meal.

Step 1: The Expert Teacher (Gemini): They took a very smart, existing AI (Google's Gemini) and gave it a "cheat sheet" of Tharu grammar rules, folk stories, and cultural facts. They told it, "Don't just translate; become a Tharu speaker."
Step 2: The Apprentice (Synthetic Data): This AI generated thousands of fake conversations (questions and answers) about farming, government forms, and local festivals.
Step 3: The Human Editor: Real Tharu speakers reviewed this "fake" data. They fixed mistakes where the AI accidentally slipped into Hindi grammar. They didn't aim for perfection; they aimed for "good enough" to be useful.

3. The Result: TharuChat and Tharu-LLaMA

The Dataset (TharuChat): They created a library of about 3,000 conversation pairs. It's a bit of a "melting pot" (mixing different Tharu dialects), which makes it a bit messy, but it covers the whole region.
The Model (Tharu-LLaMA): They took a small, efficient AI brain (3 billion parameters) and taught it using this new library.
- Why small? Big AI brains need supercomputers that cost millions. This small one can run on a standard laptop or a cheap cloud computer, making it accessible to people in Nepal and India.

4. The Magic of "Small Data"

The researchers tested a big question: Do you need millions of examples to teach an AI, or just a few good ones?

They ran an experiment like a video game level-up:

Level 1 (0% data): The AI was clueless (Perplexity > 88). It sounded like gibberish or just spoke Hindi.
Level 2 (25% data): Just 779 examples were enough to "unlock" the language. The AI suddenly understood the structure of Tharu.
Level 3 (100% data): With all 3,000 examples, the AI became fluent (Perplexity dropped to 2.88).

The Analogy: Imagine learning a new sport. You don't need to watch a million games to understand the rules. If you watch one perfect game and practice a few drills, you can start playing. The researchers proved that for low-resource languages, quality and specific examples matter more than massive volume.

5. Real-World Tests

They tested the new AI with real-life scenarios:

Banking: Asking how an ATM works. The AI explained it in Tharu, correctly using local grammar for "taking money out."
Tech: Explaining "Machine Learning." The AI simplified it to "teaching the computer to learn from new data."
Government: Asking where to get a citizenship card. The AI gave the correct local office name and advice.

6. The Bigger Picture: Why This Matters

This paper is a victory for democratization.

No More Gatekeepers: You don't need a billion-dollar company to build AI for your language. A small team with a standard computer can do it.
Preserving Culture: Instead of forcing Tharu speakers to speak Hindi or English to use technology, this tool lets them use AI in their own voice.
Embracing Imperfection: The researchers admit their data isn't "perfect" (it mixes dialects). But they argue that a "flawed" tool that works is infinitely better than a "perfect" tool that doesn't exist.

In a nutshell: The researchers built a bridge over the "digital cliff" for Tharu speakers. They used a smart AI to draft the bridge, humans to reinforce it, and proved that you don't need a massive construction crew to build it—you just need the right blueprint and a little bit of local knowledge.

1. Problem Statement

The paper addresses the "digital divide" in Artificial Intelligence, specifically the exclusion of low-resource indigenous languages like Tharu (spoken by ~1.7 million people in Nepal and India).

Data Scarcity: Tharu lacks the massive digitized corpora required for standard self-supervised learning.
Model Failure: State-of-the-art multilingual models (e.g., LLaMA-3, GPT-4) treat Tharu as a noisy variation of Hindi or Nepali due to pre-training contamination. This leads to "catastrophic code-switching," where models start in Tharu but revert to Hindi grammar mid-sentence, eroding linguistic identity.
The "Cold Start" Dilemma: There is no data to train a model, and no model exists to generate data. Traditional web scraping is ineffective due to the lack of online Tharu content.

2. Methodology: The "LLM-to-Human" Bootstrapping Pipeline

The authors propose a novel pipeline to create a training corpus from scratch, combining synthetic generation with rigorous human validation.

A. Dataset Construction (TharuChat)

Source Model: They utilized Gemini 2.5 Pro (prompt-engineered) as a "teacher" model.
Context Loading: To overcome the teacher model's lack of native Tharu knowledge, they injected:
- Explicit grammar rules (SOV structure, gender agreement, tense markers).
- Digitized folklore and oral traditions (e.g., "Alha" and "Sorathi" stories) to establish authentic diction.
- Dialect markers to distinguish between Rana (Western), Dangaura, and Kochila dialects.
Domain Elicitation: The model was prompted to generate instruction-response pairs in specific domains relevant to rural life: Agriculture, Civic Life (citizenship, land registry), and Cultural Knowledge.
Human Validation: Native speakers filtered the synthetic output to correct "Hindi-fication" (Hindi grammar with Tharu words) and dialect confusion.
Dataset Characteristics:
- Size: ~3,955 total pairs (3,116 used for training).
- Composition: ~70% Rana Tharu, ~20% Dangaura, ~10% Kochila/Others.
- Quality: Defined as "Silver Standard." It is not a prescriptive, single-dialect corpus but a "Pan-Tharu" representation that embraces natural code-mixing and dialectal heterogeneity.

B. Model Architecture & Training

Base Model: Meta's LLaMA-3.2-3B-Instruct. The 3B parameter size was chosen for accessibility on consumer-grade hardware (e.g., NVIDIA T4 GPUs).
Fine-Tuning Technique: Low-Rank Adaptation (LoRA).
- Configuration: Applied to all linear layers ($q, k, v, o, gate, up, down$).
- Hyperparameters: Rank $r=16$ , Alpha $\alpha=32$ (scaling factor $\alpha/r = 2$ ). This high scaling was crucial to overwrite the strong Hindi priors in the base model.
- Precision: Native FP16 (to ensure compatibility with older T4 architectures).
Hardware Constraints: The entire training pipeline was designed to run on a single 16GB VRAM NVIDIA T4 GPU, making it feasible for researchers in the Global South without access to H100/A100 clusters.

3. Key Contributions

TharuChat Dataset: The release of a curated, mixed-dialect corpus of ~3,955 instruction-response pairs, addressing the total lack of digital Tharu resources.
Tharu-LLaMA (3B): A publicly available, instruction-tuned model capable of fluent Tharu generation, deployable on consumer hardware.
Empirical Validation of Data Scaling: A rigorous ablation study proving that small, high-quality synthetic datasets can overcome the "data wall" for low-resource languages.

4. Experimental Results

The authors conducted an ablation study varying the training data size from 0% to 100% to measure Perplexity (PPL).

Data Volume	Samples	Validation Loss	Perplexity (PPL)
Base Model (Zero-Shot)	0	-	> 88.0 (Random/Hindi)
25%	779	1.8592	6.42
50%	1,558	1.4658	4.33
75%	2,337	1.2241	3.40
100%	3,116	1.0571	2.88

Key Finding: There is a linear reduction in perplexity as data volume increases.
Significance: A PPL of 2.88 is comparable to base models on high-resource languages, indicating high fluency. The model successfully "unlocked" the Tharu token space using only ~3,100 verified pairs, overriding the pre-trained bias toward Hindi.
Overfitting: No signs of overfitting were observed; validation loss decreased alongside training loss, attributed to aggressive dropout and LoRA regularization.

5. Qualitative Analysis & Limitations

Successes: The model demonstrated strong performance in procedural logic (e.g., explaining ATM usage), abstract concepts (e.g., defining Machine Learning), and civic processes (e.g., citizenship application), correctly handling code-mixing and specific morphological markers (e.g., honorifics).
Limitations:
- Dialectal Noise: The model may mix Rana and Dangaura morphologies, which might be jarring for users seeking a "pure" dialect.
- Residual Influence: Some "Hindi-fication" persists in complex sentence structures due to the synthetic generation source.
- Domain Specificity: The model excels in agriculture and local civics but may hallucinate or revert to English/Hindi on global topics (history, advanced physics) outside its training distribution.

6. Significance and Impact

Democratization of AI: The work proves that high-quality LLMs for indigenous languages can be built on consumer-grade hardware (single T4 GPU) with minimal data (~3k examples), challenging the narrative that AI requires billions of tokens and millions of dollars in compute.
Linguistic Preservation: By creating a "Pan-Tharu" model, the authors argue that aggregating dialects into a single "Koine" representation is a viable and necessary strategy for low-resource languages, prioritizing functional communication over prescriptive purity.
Reproducibility: The paper provides a transparent blueprint (dataset, code, weights) for researchers to replicate this "bootstrapping" approach for other endangered Himalayan languages.