💻 computer science

Evaluating Cross-lingual Knowledge Consistency in Code-Mixed vis-a-vis Indian Languages using IndicKLAR

The paper introduces IndicKLAR, a benchmark for 18 Indian languages and their code-mixed variants, revealing that code-mixed inputs significantly bridge the cross-lingual knowledge consistency gap between English and native languages, often matching English performance without model intervention.

Original authors: Debajyoti Mazumder, Divyansh Pathak, Prashant Kodali, Aditya Joshi, Akshay Agarwal, Jasabanta Patro

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Debajyoti Mazumder, Divyansh Pathak, Prashant Kodali, Aditya Joshi, Akshay Agarwal, Jasabanta Patro

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Language Wall"

Imagine you have a very smart librarian (a Large Language Model) who knows everything about the world. If you ask this librarian a question in English, they answer perfectly. But if you ask the exact same question in a local Indian language (like Hindi, Tamil, or Bengali), they often get it wrong or make things up.

This is a "knowledge gap." The librarian knows the facts, but the language barrier blocks them from accessing that knowledge when you speak their native tongue.

The Discovery: The "Code-Mixed" Shortcut

The researchers noticed something interesting about how people in India actually talk. They often mix English words into Indian sentences. This is called Code-Mixing.

Example: Instead of saying "What is the capital of India?" in pure Hindi, someone might say, "India ki capital kya hai?" (Mixing the English word "capital" into the Hindi sentence).

The paper found that when you ask the AI this "mixed" version, it suddenly gets much smarter. It's as if the English words act as a key that unlocks the librarian's knowledge, even though the rest of the sentence is in the local language.

The New Tool: INDIKLAR

To study this, the team built a new testing ground called INDIKLAR.

The Analogy: Imagine a gym with three different tracks:
1. The Native Track: Questions in pure local languages.
2. The English Track: Questions in pure English.
3. The Mixed Track: Questions that mix English words into local sentences.
They created this test for 18 different Indian languages and checked the answers with real native speakers to make sure the questions were natural.

The Results: The "Flip Point"

When they ran the tests, they found a clear pattern they call the "Flip Point."

Native Language: The AI often fails (like a student who forgot their textbook).
Code-Mixed: The AI suddenly starts getting it right (like the student remembering the textbook because they saw a familiar English word).
English: The AI gets it right (the textbook is fully open).

The "Flip Point" is the moment the AI switches from "wrong" to "right." The paper shows this switch happens right between the Native and Code-Mixed settings. Once you add those English keywords, the AI's performance jumps up to almost match its English performance.

The Solution: "Thinking in Thought" (TinT)

The researchers wanted to see if they could get the AI to do this "mixing" trick without you having to type the mixed words yourself. They invented a strategy called Translate-in-Thought (TinT).

The Analogy: Imagine asking the librarian, "Tell me the answer, but think about it in English first, then just give me the final answer in Hindi."
How it works: You don't see the English translation. The AI does the translation inside its "brain" (internally) and only spits out the final answer.
The Result: This worked! The AI got much better at answering in local languages just by being told to "think" in English or a mixed format internally. It was also faster than asking the AI to translate the question out loud first.

Key Takeaways

The Gap is Real: AI is much worse at answering questions in pure Indian languages than in English.
The Shortcut Works: Mixing English words into Indian sentences (Code-Mixing) fixes most of the problem immediately.
The Magic Trick: You can get similar results by telling the AI to "think" in English internally (TinT) without needing to change the question you type.
Size Matters: Bigger AI models are better at this "internal thinking" trick, but even smaller models can benefit from it.

What the paper does NOT say:

It does not claim this works for medical advice or legal decisions.
It does not say this will fix all AI problems in India forever.
It focuses strictly on testing the AI's ability to recall facts, not on creative writing or complex reasoning tasks.

In short: If you want an AI to know facts about India, speaking to it in a mix of English and local words (or telling it to "think" that way) is the secret sauce to getting the right answer.

Technical Summary: Evaluating Cross-lingual Knowledge Consistency in Code-Mixed Indian Languages using IndicKLAR

Problem Statement
Large Language Models (LLMs) exhibit a significant "cross-lingual consistency gap," where they reliably recall factual knowledge in high-resource languages like English but frequently fail when the same query is posed in low-resource languages. This disparity poses a critical barrier to equitable AI deployment in linguistically diverse regions like India, which recognizes 22 scheduled languages. Existing benchmarks predominantly focus on clean monolingual inputs or high-resource language pairs, largely ignoring the reality of code-mixed communication (interleaving English and Indian languages) common in everyday digital interactions. Furthermore, prior evaluations often rely on translating model outputs back to English via external systems, which can obscure whether the low-resource output was inherently faithful.

Methodology
To address these gaps, the authors introduce INDIKLAR, an Indic extension of the KLAR-CLC benchmark. The methodology involves three core components:

Benchmark Construction (INDIKLAR):
- Scope: Covers 18 of the 22 scheduled Indian languages.
- Variants: For 11 of these languages (where linguist coverage was available), the authors constructed code-mixed variants by interleaving English content words into the grammatical structure of the native language.
- Alignment: The dataset provides a three-way alignment for 11 languages: (1) Native Indian (monolingual), (2) Code-Mixed, and (3) English. The remaining 7 languages have two-way alignment (Native and English).
- Verification: Both monolingual and code-mixed variants underwent human quality verification by native or near-native speakers to ensure fluency and naturalness.
- Scale: Each language setting contains 2,619 instances spanning 20 fact categories.
Experimental Setup:
- Models: Evaluation was conducted on nine open-weight models across three families (Llama 3.1/3.2, Gemma 3, Qwen 2.5), ranging from 1B to 14B parameters.
- Metrics: The study measures Accuracy (correctness of predictions) and Cross-Lingual Consistency (CLC) (overlap of correct predictions across language pairs).
- Prompting Strategies: The authors systematically compared several strategies to bridge the performance gap:
  - Baseline: Direct prompting in the source language.
  - 2-Step Strategies: Separate inference calls for conversion (to code-mixed, English, or transliterated Roman script) followed by answering.
  - 1-Step Strategies: Joint generation of conversion and answer.
  - Translate-in-Thought (TinT): A novel single-step strategy where the model is instructed to internally convert the input (either to English or code-mixed) and emit only the final answer, without generating intermediate text.
Analysis:
- The study analyzes performance trajectories from Native $\to$ Code-Mixed $\to$ English.
- It includes an interpretability case study using layer-wise rank analysis to observe how the model's internal representation of the correct answer token evolves across layers.

Key Contributions

INDIKLAR Benchmark: The first benchmark to pair monolingual Indic factual queries with code-mixed counterparts under native-speaker verification at this scale, enabling aligned evaluation across English, native, and code-mixed inputs.
Translate-in-Thought (TinT): Introduction of a single-step implicit prompting strategy that elicits internal translation without explicit intermediate outputs, offering performance gains with lower inference latency compared to multi-step methods.
Identification of the "Flip Point": The discovery that the boundary between incorrect and correct prediction (the "flip point") consistently lies between the native low-resource setting and the code-mixed setting, rather than between code-mixed and English.

Results

The Native-English Gap: Native-language accuracy gaps relative to English can reach approximately 0.50, particularly for smaller models and languages like Assamese and Telugu.
Code-Mixed Efficacy: Code-mixed inputs significantly close this gap, bringing performance within ~0.05 of English-level accuracy without any model-level intervention (retraining).
Prompting Performance:
- TinT shows statistically significant gains over the baseline across languages and model families.
- Efficiency: TinT variants achieve these gains with significantly lower inference time (e.g., ~0.09s per sample for Llama-3.1-8B) compared to multi-step strategies.
- Scaling: The effectiveness of TinT scales with model size; larger models (e.g., 8B–14B) demonstrate stronger latent translation abilities. Smaller models (<1.5B) sometimes struggle with reformulation, though Gemma-3-1B showed consistent improvement.
Language Families: Indo-Aryan languages generally saw larger gains from TinT (+18% accuracy) compared to Dravidian languages (+14%), though TinT improved performance across most categories.
Flip Point Location: Whether induced by input surface form (Native vs. Code-Mixed) or internal conversion (Native vs. TinT), the transition from incorrect to correct prediction occurs between the native and code-mixed settings.
Ablation: Code-mixed variants outperformed pure romanization (transliteration) variants, suggesting that meaningful bilingual lexical cues (English anchors) are more critical than script uniformity alone.

Significance and Claims
The paper claims that code-mixing acts as a powerful bridge for cross-lingual knowledge consistency, allowing models to leverage stronger English reasoning capabilities while preserving low-resource semantics. The introduction of Translate-in-Thought (TinT) demonstrates that models possess a latent translation capability that can be elicited through prompting alone, without the need for expensive retraining or external translation systems.

The authors emphasize that the "flip point" finding suggests that the primary failure mode for low-resource knowledge recall is not a lack of knowledge, but rather the difficulty of accessing that knowledge through the native script. By shifting the input to a code-mixed or internally translated state, models can access the same knowledge they hold for English.

Limitations
The authors note several limitations:

Synthetic Generation: Code-mixed variants were generated via GPT APIs and verified by humans, but may not fully capture the dialectal diversity of real-world code-mixing.
Coverage: Four scheduled languages (Bodo, Kashmiri, Manipuri/Meitei, Santali) were excluded due to limited translation system quality and annotator access.
Generalization: Findings are based on a specific factual recall benchmark (KLAR-CLC) and may not generalize to other NLP tasks like reasoning or open-ended generation.
Model Scale: Experiments were limited to models up to 14B parameters; trends for larger models remain an open question.
Interpretability: The mechanistic analysis (layer-wise rank) was conducted on a single model (Llama-3.1-8B-Instruct) and should be viewed as illustrative rather than exhaustive.