💬 NLP

Conversational Domain Adaptation of IndicTrans2 across 21 Indic Languages via Experience Replay and Model Soups

This paper presents an honest, end-to-end study demonstrating that combining experience replay and model souping allows IndicTrans2 to adapt to conversational registers across 21 Indic languages, significantly improving reference-matching metrics without degrading general-domain performance, while acknowledging that these gains reflect better register alignment rather than confirmed human-perceived quality improvements.

Original authors: Aditya Pratap Singh

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aditya Pratap Singh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, world-class translator named IndicTrans2. This translator is incredibly smart and knows how to translate formal news articles, official documents, and encyclopedia entries perfectly. However, if you ask them to translate a casual text message, a movie subtitle, or a quick voice note, they sound stiff and robotic, like a robot reading a legal contract.

The author of this paper asked: "Can we teach this translator to sound more natural and conversational without making them forget how to translate formal texts?"

Here is the story of how they tried to do that, using a recipe that worked on paper but had some surprising caveats.

The Problem: The "Robot" Translator

The translator was trained on "general" data (news, books). When you try to teach it "conversational" data (chat, subtitles) by just letting it practice on those new examples, it suffers from Catastrophic Forgetting.

Think of it like a student who studies hard for a math test but then spends a month only practicing poetry. When they take the math test again, they forget the formulas. In the paper's terms, the translator got better at chatting but got worse at formal translation.

The Solution: A Two-Step Recipe

The author used two existing techniques, mixing them together like a cooking recipe to fix the problem:

Experience Replay (The "Review Session"):
Instead of letting the translator only practice on casual chat, the author made it practice on casual chat and sneak in some of its old formal training data.
- Analogy: Imagine a chef learning to make street food. Instead of only cooking street food, they keep cooking a few classic dishes every day to keep their skills sharp. This stops them from forgetting the basics.
Model Soups (The "Blending"):
After training, the author didn't just pick the new "street food" version. Instead, they took the original "formal" translator and the new "street food" translator and averaged their brains together.
- Analogy: Imagine taking a cup of strong black coffee (the formal model) and a cup of sweet milk tea (the conversational model) and mixing them. The result is a drink that has the caffeine of the coffee but the smoothness of the tea. It's a "soup" of the two models.

The Results: A Metric Win, But a Reality Check

The author tested this new "Soup" model on 21 different Indian languages.

The Good News (The Numbers):

Conversational Score: The new model scored significantly higher on tests designed to measure how well it matches casual, human-like text. It improved by an average of 6.2 points across all 21 languages.
Formal Score: Crucially, it did not lose its ability to translate formal text. The score on formal tests stayed almost exactly the same (a tiny drop of 0.17 points, which is statistically negligible).
Conclusion: By the numbers, the recipe worked perfectly. The model became conversational without losing its formal skills.

The Bad News (The Reality Check):
The author was very honest about what these numbers actually mean.

The "Style" Trap: The test scores went up because the new model started sounding more like the test examples (which were casual subtitles). It used more informal words and sentence structures.
The Human Test: The author asked humans and other AI models to judge the quality. They did not agree that the translation was "better." They just noticed it sounded more casual.
The Verdict: The model didn't necessarily become a smarter translator; it just became a better mimic of the casual style found in the test data. It matched the "vibe" of the reference text, which boosted the score, but didn't necessarily improve the actual meaning or quality.

The Limits

The paper also points out where this recipe hits a wall:

Low-Resource Languages: For languages with very little data (like Santali or Sanskrit), the model couldn't improve much. It's like trying to teach a student a new skill when they don't have enough textbooks to begin with. The "floor" was set by the lack of data, not the method.
Test Bias: Some languages had "easy" tests that looked exactly like the training data, leading to huge score jumps. The author warns that these big jumps might be misleading because the test was too easy.

Summary

The author successfully created a "conversational" version of a top-tier translator for 21 languages using a clever mix of reviewing old data and blending model brains.

Did it work? Yes, by the computer's score (chrF), it became much more conversational without losing formal skills.
Is it better? The author says: "We don't know for sure." The computer thinks it's better because it sounds more casual, but human judges didn't feel a clear improvement in quality.

The paper is a study in honesty: showing that while you can game the metrics to make a model look better at a specific style, proving that it actually helps humans requires more than just a high score.

Technical Summary: Conversational Domain Adaptation of IndicTrans2

Problem Statement
While IndicTrans2 is currently the strongest open-source English-to-Indic translation system, it is trained on general-domain text (news, Wikipedia) and tends to produce stiff, formal output when applied to casual, conversational inputs (chat, subtitles, voice commands). The core challenge addressed is adapting this strong multilingual model to a conversational register across 21 Indic languages without sacrificing its general-domain performance. A standard approach—plain fine-tuning on conversational data—leads to catastrophic forgetting, where the model loses its ability to translate general text effectively (e.g., a 3.9 chrF drop on FLORES for Hindi).

Methodology
The authors propose a two-stage adaptation recipe using only public data (OpenSubtitles, BPCC-H-Daily, Tatoeba) and established techniques:

Experience Replay (Mixed Fine-Tuning): To prevent forgetting, the model is fine-tuned on a mix of conversational data and a "general anchor" dataset (BPCC-H-Wiki) sampled at a roughly 1:1 ratio. This maintains the model's general capabilities while learning the conversational style.
Model Soups: The weights of the fine-tuned model are averaged with the original base model weights ( $soup = (1 - \alpha) \times base + \alpha \times fine-tuned$ ). The mixing weight $\alpha$ is swept (0.3 to 0.7) to find the optimal trade-off between conversational fluency and general accuracy. The authors use a streaming approach to average tensors, keeping memory usage bounded.

The study covers 21 Indic languages using the IndicTrans2-1B base model. Evaluation is performed on held-out conversational test sets and the FLORES-200 general-domain benchmark using chrF2 (character n-gram F-score).

Key Results

Conversational Gains: The "mixed soup" approach improves conversational chrF scores across all 21 languages, with a mean gain of +6.2 chrF. The largest gains are observed in languages with larger conversational datasets (e.g., Gujarati +12.2, Telugu +10.1).
Preservation of General Quality: Crucially, the method eliminates the forgetting trade-off. The resulting models match the base model's performance on FLORES, with a mean change of -0.17 chrF (all languages within a 0.7 chrF band).
Statistical Significance: Paired bootstrap tests on languages with rigorous subtitle-based test sets (Hindi, Tamil, Malayalam) confirm that conversational gains are statistically significant ( $p \le 0.004$ ) and that general quality degradation is not significant.
The "Metric vs. Quality" Gap: While metrics show improvement, the authors note that the gains largely reflect a register match to the references (e.g., using informal pronouns or spoken word order) rather than a definitive improvement in translation meaning. A blind human evaluation and multi-model LLM check on Hindi did not confirm a perceived quality improvement, suggesting the metric overstates the actual utility gain.

Contributions and Significance
The paper's primary contribution is not the novelty of the techniques (experience replay and model soups are established), but the honest, end-to-end study of their application in the Indic conversational setting.

Reproducible Recipe: It provides a verified workflow for adapting multilingual models to specific registers across 21 languages using public data, complete with significance testing.
Honest Evaluation of Limits: The authors explicitly delineate the boundaries of their success:
- The method works by the metric (chrF) but the metric may overstate human-perceived quality.
- Gains are inflated on "easy" in-domain test sets (like BPCC-H-Daily) compared to harder subtitle-based sets.
- A "floor" exists for low-resource languages (e.g., Santali, Sanskrit, Kashmiri) where the base model's lack of general data limits the ceiling for adaptation.
Scope Definition: The authors treat the conversational gain as a successful register match rather than proof of superior translation quality, avoiding overclaiming based on automatic metrics alone.

Conclusion
The study demonstrates that it is possible to adapt a general-domain Indic translation model to a conversational register across 21 languages without degrading general performance, provided one uses experience replay and model souping. However, the authors conclude that while the model is measurably closer to human conversational references, whether this translates to a tangible quality improvement for end-users remains an open question requiring further controlled human evaluation.