Indic-TunedLens: Interpreting Multilingual Models in Indian Languages

1. Problem Statement

Multilingual Large Language Models (LLMs) are increasingly deployed in linguistically diverse regions like India. However, existing interpretability tools (e.g., Logit Lens, Tuned Lens) are primarily designed for English and fail to generalize effectively to Indian languages due to several critical factors:

• English-Centric Latent Spaces: Research indicates that even when generating non-English outputs, multilingual LLMs often operate in an English-centric representation space, activating English word representations before translating them.
• Linguistic Divergence: Indian languages exhibit distinct structural properties compared to English, including rich morphology (complex inflectional systems), flexible word order, and unique scripts (e.g., Devanagari, Gurmukhi).
• Projection Mismatch: Standard interpretability methods apply a single affine transformation (trained on English) to decode intermediate hidden states. This fails to align the morphologically rich and syntactically diverse representations of Indian languages with the output vocabulary space, leading to high uncertainty (entropy) and fragmented semantic analysis in early and middle layers.

2. Methodology: Indic-TunedLens

The authors propose Indic-TunedLens, a novel interpretability framework specifically designed to bridge the gap between intermediate hidden states and final output distributions for Indian languages.

• Core Mechanism: Unlike the standard Logit Lens (which uses the pre-trained unembedding matrix directly) or the standard Tuned Lens (which learns a single affine transformation), Indic-TunedLens learns language-specific affine transformations.
• Mathematical Formulation:
  For a transformer layer $n$ with hidden state $h_n$, the framework applies a learned translator consisting of a matrix $M_n \in \mathbb{R}^{d \times d}$ and a bias vector $b_n \in \mathbb{R}^d$:
  $$\tilde{h}_n = M_n h_n + b_n$$
  The transformed state is then passed through the model's final layer head to produce token logits:
  $$\text{Indic-TunedLens}_n(h_n) = \text{LogitHead}(\tilde{h}_n)$$
• Training Objective: The framework is trained to reproduce the next-token probability distribution of the base model's final layer ($p_{final}$). It minimizes the Kullback-Leibler (KL) divergence between the final layer's distribution and the distribution predicted by the lens at intermediate layers:
  $$\min_{M_n, b_n} \mathbb{E}_{h_n} [ D_{KL} ( p_{final}(x) \parallel \text{Indic-TunedLens}_n(h_n) ) ]$$
• Data & Model:
  • Base Model: Sarvam-1 (a multilingual LLM).
  • Languages: 10 Indian languages (Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Panjabi, Nepali, Tamil, Telugu).
  • Datasets: Trained on the Sangraha dataset and evaluated on a multilingual subset of the MMLU benchmark.

3. Key Contributions

1. Identification of the Projection Problem: The paper establishes that interpretability failure in Indian languages is fundamentally a projection problem, not an internal processing failure. The models process semantic content correctly, but English-centric lenses fail to decode it.
2. Indic-TunedLens Framework: The first layer-wise interpretability tool specifically tailored for multilingual LLMs in Indian languages, utilizing shared affine transformations adapted to Indic scripts.
3. Comprehensive Evaluation: A rigorous evaluation across 10 languages using the MMLU benchmark, analyzing layer-wise accuracy, entropy, and token position dynamics.
4. Linguistic Insights: Discovery of language-specific computational trajectories, revealing that morphological analysis and semantic composition occur at different depths depending on the language's typological structure (e.g., agglutinative vs. analytic).

4. Key Results

The evaluation demonstrates that Indic-TunedLens significantly outperforms standard English-centric methods (Logit Lens and standard Tuned Lens):

• Layer-Wise Accuracy & Early Interpretability:
  • Standard Tuned Lens: Shows minimal accuracy in early layers (0–15) for Indian languages (often <0.02), with significant improvements only appearing after layer 20.
  • Indic-TunedLens: Achieves immediate interpretability from Layer 1. For example, it achieves 0.13 accuracy for English and 0.11 for Telugu in the first layer, maintaining robust accuracy (0.03–0.06) throughout middle layers.
• Entropy Analysis (Uncertainty):
  • Standard lenses produce high, irregular entropy across layers for Indian languages, indicating unstable representations.
  • Indic-TunedLens exhibits a smooth, systematic reduction in entropy across layers, indicating progressive information consolidation and alignment with Indic linguistic structures.
• Ranking Performance:
  • Indic-TunedLens consistently places correct tokens in higher ranks (lower average rank numbers) across all layers compared to the baseline. For instance, Malayalam shows substantial improvement, moving from poor baseline ranks to consistent top-tier candidates earlier in the decoding process.
• Morphological Sensitivity:
  • The framework captures distinct patterns based on morphology. Languages with rich inflectional systems (e.g., Bengali, Hindi, Nepali) show the strongest improvements in the earliest layers (1–4), corresponding to morphological analysis.
  • Agglutinative languages (e.g., Tamil, Telugu) show processing trajectories that shift the spatial distribution of reasoning steps due to tokenization complexity.

5. Significance and Implications

• Beyond English-Centricity: The work proves that interpretability methods cannot be language-agnostic. Effective analysis of multilingual models requires "language-family-aware" lenses that respect typological differences.
• Democratizing AI Transparency: By enabling faithful decoding of intermediate states in low-resource, morphologically rich languages, this tool supports the development of more equitable, reliable, and trustworthy AI systems for the Indian subcontinent.
• Methodological Template: The approach provides a blueprint for extending interpretability to other non-English, morphologically complex language families globally.
• Insight into Model Internals: The results suggest that the "computational bottleneck" for interpreting multilingual models lies in the projection mechanism (the lens), not the internal processing of the model itself. Specialized training allows researchers to access semantically meaningful states much earlier in the network.

Availability: The model and code are publicly available on Hugging Face and GitHub, facilitating further research in multilingual NLP interpretability.