💬 NLP

Konkani LLM: Multi-Script Instruction Tuning and Evaluation for a Low-Resource Indian Language

This paper addresses the performance deficit of Large Language Models in the low-resource, multi-script Konkani language by introducing the synthetic Konkani-Instruct-100k dataset and the resulting Konkani LLM, which demonstrates competitive or superior performance against proprietary baselines in instruction tuning and machine translation tasks.

Original authors: Reuben Chagas Fernandes, Gaurang S. Patkar

Published 2026-03-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Reuben Chagas Fernandes, Gaurang S. Patkar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, all-knowing librarian (a Large Language Model) who can speak thousands of languages. However, if you ask this librarian to tell a story in Konkani—a beautiful language spoken in the coastal regions of India—they might stumble, sound robotic, or even mix it up with Hindi or Marathi.

Why? Because Konkani is like a rare, multi-faceted gem that hasn't been polished enough in the digital world. It's "low-resource," meaning there aren't enough books or digital notes for the AI to study. To make things harder, Konkani is written in three different scripts (like wearing three different pairs of glasses):

Devanagari (the standard script used for Hindi).
Romi (Latin letters, used by Goan Catholics).
Kannada (used by Konkani speakers in Karnataka).

Most AI models only know one pair of glasses, or they get confused trying to wear all three at once.

The Problem: The "Language Contamination"

The authors of this paper noticed that even the smartest AIs (like GPT-4 or Claude) struggle with Konkani. It's not that they don't know the words; it's that they haven't been trained to speak it correctly. They often "contaminate" the language, accidentally slipping in Hindi or Marathi words, or they get the grammar wrong because they lack a proper "textbook" to learn from.

The Solution: Building a Custom "Konkani Tutor"

The researchers decided to build their own specialized training school for AI. Here is how they did it, step-by-step:

1. Creating the "Konkani-Instruct-100k" Textbook

Since there weren't enough real human-written examples, they used a powerful AI (Gemini) to write a new textbook from scratch.

The Analogy: Imagine you want to teach a student a language, but there are no textbooks. So, you hire a master teacher to write 100,000 practice questions and answers.
The Twist: They didn't just ask for random sentences. They used a "Tutor-Style" approach. The AI didn't just say, "Here is the answer." It had to show its work, like a math student: "First, I identified the gender of the noun, then I applied the verb rule, and finally, here is the sentence."
The Result: A massive dataset of 100,000+ examples, perfectly balanced across all three scripts (Devanagari, Romi, and Kannada).

2. The "Fine-Tuning" Surgery

They took existing, smart AI models (like Llama 3 and Qwen) and gave them this new textbook to study.

The Analogy: Think of these AI models as general-purpose doctors. They know a lot about medicine. The researchers didn't replace the doctors; they sent them to a specialized residency to become experts in "Konkani Medicine."
The Method: They used a technique called LoRA (Low-Rank Adaptation).
- Metaphor: Instead of rebuilding the entire doctor's brain (which is expensive and slow), they added a specialized "stethoscope" adapter. This small add-on teaches the AI how to listen specifically to Konkani nuances without forgetting everything else it knows.

3. The "Konkani-Bench" Exam

To prove their new AI models actually worked, they created a rigorous exam called Konkani-Bench.

The Exam: It tests the AI on translation (Konkani to English) and transliteration (switching between the three scripts).
The Judges: They used human experts and another AI to grade the answers, looking for things like: Did it use the right script? Did it accidentally use Hindi words? Is the grammar correct?

The Results: A New Champion

The results were impressive.

Before: The best general AI models scored poorly on Konkani, often making basic grammar mistakes or mixing scripts.
After: The new "Konkani LLM" models (specifically the konkani-Qwen2.5-14B) became the new champions.
- They outperformed the massive, expensive "proprietary" models (like the ones you pay for from big tech companies) in many tests.
- They could switch between scripts seamlessly, like a polyglot who can write a letter in English, then switch to Hindi, then to French, without losing their train of thought.

Why This Matters

This paper is a blueprint for saving and empowering low-resource languages.

The Big Picture: It shows that you don't need billions of dollars or millions of human hours to teach an AI a rare language. You just need a smart strategy: Synthetic Data (AI writing AI textbooks) + Smart Fine-Tuning (specialized adapters).
The Future: Now, anyone can use these free, open-source models to chat, translate, or write in Konkani, preserving the culture and making it accessible to the next generation, regardless of which script they prefer to read.

In short: The researchers took a struggling AI, gave it a custom-made, multi-script tutor, and turned it into a fluent, culturally aware Konkani speaker.

1. Problem Statement

The paper addresses the significant performance deficit of Large Language Models (LLMs) in Konkani, a low-resource Indo-Aryan language spoken in Goa and the Konkan region. The primary challenges identified are:

Data Scarcity: There is a lack of high-quality, large-scale instruction-tuning datasets for Konkani.
Script Diversity: Konkani is officially written in three distinct scripts: Devanagari, Romi (Latin), and Kannada. Existing datasets are often fragmented across these scripts or rely on automated transliteration, which introduces phonological errors and fails to capture dialectal nuances.
Language Contamination: General-purpose LLMs often struggle to distinguish Konkani from closely related languages like Marathi or Hindi, leading to "contamination" where models generate mixed or incorrect outputs.
Lack of Unified Models: Current efforts focus on high-resource Indian languages (Hindi, Tamil) or single-script models, leaving a gap for unified, multi-script Konkani models.

2. Methodology

The authors propose a pipeline centered on Synthetic Data Generation and Parameter-Efficient Fine-Tuning (PEFT) to overcome data scarcity.

A. Dataset Creation: Konkani-Instruct-100k

Synthesis Strategy: Due to the absence of native instruction data, the authors used Gemini 3 (specifically 2.5 Pro and 3 Flash) as "teacher models" to generate a synthetic dataset.
Pedagogical Framework: Instead of standard QA pairs, they employed a "Tutor-Style" framework. The models were prompted to teach 18 foundational linguistic topics (e.g., gender, oblique cases, tenses) using a "Grammar Decomposition Table." This forces the model to explicitly analyze morphology (gender, number, politeness) before generating the final sentence, reducing hallucinations.
Script Balance: The dataset (Konkani-Instruct-100k) contains 105,982 samples balanced across Devanagari, Romi, and Kannada scripts to prevent orthographic bias.
Structure: Every sample includes the instruction, the response, and a structural breakdown (Part-of-Speech tags, base-modifier morphology, and grammatical rules).

B. Model Training

Base Models: The authors fine-tuned several open-weight architectures, including Llama 3.1 (8B), Gemma 3 (4B, 12B, 27B), and Qwen2.5 (1.5B, 14B).
Technique: They used Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning.
- Configuration: Rank ( $r$ ) = 64, Scaling ( $\alpha$ ) = 128.
- Target Modules: Applied to attention projections ( $q, k, v, o$ ) and MLP projections ($gate, up, down$).
- Hyperparameters: 2 epochs, 4096 sequence length, 8-bit AdamW optimizer, bfloat16 precision.
Goal: To demonstrate that compact models trained on linguistically controlled synthetic data can outperform larger general-purpose models in script fidelity and grammatical accuracy.

C. Evaluation Benchmark: Konkani-Bench

A new human-annotated benchmark consisting of 200 items.
Tasks:
1. Translation: Konkani (all scripts) $\to$ English.
2. Transliteration: Pairwise conversion between Romi, Devanagari, and Kannada (6 directions).
Metrics:
- Automatic: BLEU, chrF++, and COMET (for translation).
- LLM-as-a-Judge: Used Gemini 3 Flash to score helpfulness, script fidelity, and the absence of Marathi/Hindi contamination.

3. Key Contributions

Konkani-Instruct-100k: The first large-scale, multi-script instruction-tuning dataset for Konkani, generated via distillation with a focus on morphological correctness.
Konkani LLM Suite: A series of fine-tuned models optimized for regional nuances across all three scripts.
Konkani-Bench: A rigorous evaluation benchmark designed to stress-test cross-script robustness and translation quality.
Demonstration of Synthetic Efficacy: Proving that synthetic data generated with "Tutor-Style" prompting can effectively train low-resource language models.

4. Experimental Results

The fine-tuned models showed substantial improvements over their base counterparts and competitive performance against proprietary models.

Translation Performance (Table 4):
- The best-performing fine-tuned model, konkani-Qwen2.5-14B-Instruct, achieved a COMET score of 83.58 and BLEU of 50.26.
- This outperformed the base Qwen2.5-14B (BLEU 5.59) significantly and surpassed several proprietary baselines (e.g., GPT-5-mini, Grok-4) in specific metrics.
- konkani-gemma-3-27b-it also performed strongly with a COMET of 82.91.
Transliteration Performance (Table 5):
- Fine-tuned models showed massive gains in cross-script transliteration. For example, konkani-Qwen2.5-14B achieved BLEU scores between 37.51 and 56.03 across different script directions, whereas base models often scored below 10.
- The models demonstrated strong robustness in converting between Devanagari, Romi, and Kannada.
LLM-as-a-Judge (Tables 6–8):
- Llama 8B and Qwen2.5 14B generally received the highest scores (4.0–4.6/5) for "Simple Chat" and "Creative Writing" across Romi and Devanagari scripts.
- The fine-tuned models successfully reduced language contamination (Marathi/Hindi leakage) compared to base models.

5. Significance and Impact

Bridging the Low-Resource Gap: This work provides a blueprint for developing LLMs for other low-resource Indian languages that suffer from script fragmentation.
Multi-Script Capability: It is one of the few efforts to successfully unify three distinct scripts (Devanagari, Romi, Kannada) into a single model family, addressing the unique sociolinguistic reality of Konkani speakers.
Synthetic Data Validation: The study validates that high-quality synthetic data, when generated with strict morphological constraints (Tutor-Style), can serve as a viable substitute for scarce human-annotated data.
Open Access: The authors have released the models on Hugging Face (huggingface.co/konkani) and a public demo at konkani.app, fostering community adoption and further research.

6. Limitations

Synthetic Bias: The dataset is entirely synthetic and may inherit biases or stylistic artifacts from the teacher models (Gemini).
Dialectal Diversity: While script-balanced, the dataset may not fully capture the full spectrum of dialectal variations across different Konkani-speaking communities.
Training Constraints: The study relied on LoRA (rank 64) due to serverless deployment constraints; full fine-tuning or continued pre-training might yield further improvements.

In conclusion, the paper successfully demonstrates that targeted synthetic instruction tuning can create robust, multi-script LLMs for low-resource languages, significantly outperforming general-purpose models in grammatical accuracy and script fidelity.