Neither Here Nor There: Cross-Lingual Representation Dynamics of Code-Mixed Text in Multilingual Encoders

1. Problem Statement

Multilingual Language Models (MLMs) like mBERT and XLM-R are widely used for code-mixed text analysis (e.g., Hinglish: Hindi-English mixed text). However, there is a significant gap in understanding how these models internally represent code-mixed inputs relative to their constituent monolingual languages (English and Hindi).

• The Core Question: Do code-mixed representations form a coherent bridge between the two languages, or do they occupy a disjointed, peripheral space?
• The Gap: Existing research focuses on monolingual cross-lingual alignment or downstream task performance, leaving the mechanistic relationship between code-mixed representations and their source languages underexplored.
• The Trade-off: It is unclear if adapting models on code-mixed data improves code-mixed understanding at the cost of degrading the original monolingual alignment (e.g., English-Hindi).

2. Methodology

A. Dataset Construction

The authors constructed a unified trilingual parallel corpus containing 21,139 aligned sentence triples:

1. English (EN): Monolingual English.
2. Hindi (HI): Monolingual Hindi in Devanagari script.
3. Code-Mixed (CM): Hindi-English mixed text in Roman script (Hinglish).

• Sources: Merged from CM-En Parallel, PHINC, and LinCE datasets. Missing Hindi translations were generated via Google Translate and verified with IndicTrans2.

B. Models Evaluated

The study evaluated standard multilingual encoders and their code-mixed adapted variants:

• Base: mBERT, XLM-R.
• Adapted: Hing-mBERT, Hing-RoBERTa (trained on L3Cube-HingCorpus).
• Mixed-Script Adapted: Variants trained on data preserving Devanagari Hindi alongside Romanized CM.

C. Analysis Techniques

The authors employed mechanistic interpretability tools to probe internal representations:

1. Cross-Lingual Retrieval: Measured alignment accuracy using dot-product similarity between parallel sentences (EN$\leftrightarrow$HI, EN$\leftrightarrow$CM, HI$\leftrightarrow$CM).
2. Centered Kernel Alignment (CKA) & SVCCA: Analyzed layer-wise similarity to track how alignment evolves through transformer layers.
3. Token-Level Saliency: Used Relative Importance (RI) scores to determine which language tokens (English vs. Hindi) drive the model's decision.
4. Entropy-Based Uncertainty: Measured how much conditioning on English or Hindi reduces the entropy (uncertainty) of the code-mixed representation.

D. Proposed Solution: Trilingual Post-Training Alignment

To address observed misalignments, the authors introduced a Trilingual Post-Training Alignment Stage:

• Objective: A joint loss function ($L_{align}$) that minimizes the cosine distance between the [CLS] embeddings of parallel English, Hindi, and Code-Mixed sentences.
• Formula: $L = L_{base} + \lambda \cdot L_{align}$, where $L_{align}$ encourages all three languages to occupy nearby regions in the embedding space simultaneously.
• Metric: Introduced Cross-Lingual Alignment Score (CLAS) to measure balanced alignment, accounting for average accuracy, directional bias, and variance across pairs.

3. Key Findings & Results

A. Representation Dynamics in Standard Models

• Weak Anchoring: In vanilla models, code-mixed (CM) representations are "loosely connected" to both English and Hindi, occupying a peripheral subspace.
• The Adaptation Trade-off: Continued pre-training on code-mixed data significantly improves English–CM alignment but degrades English–Hindi alignment. The model pulls English representations toward the code-mixed manifold, breaking the original monolingual bridge.
• Script Dependency: Preserving Hindi in Devanagari script during adaptation improves HI–CM alignment significantly compared to Romanized-only training.

B. Interpretability Insights

• English Dominance: Models process code-mixed text primarily through an English-dominant semantic subspace. English tokens receive significantly higher saliency scores than Hindi tokens.
• Complementary Signals: While English drives the representation, native-script Hindi provides complementary signals that reduce representational uncertainty. Conditioning on both languages yields the lowest entropy, indicating that native script is crucial for grounding CM meaning.
• Layer-wise Shift: In adapted models, the peak alignment shifts from the middle layers (typical in vanilla models) to earlier layers for EN–CM, while EN–HI alignment weakens.

C. Effectiveness of Trilingual Alignment

• Balanced Alignment: The proposed trilingual post-training objective successfully restores the degraded English–Hindi alignment while maintaining strong English–CM alignment.
• CLAS Improvement: The method yielded substantial gains in the Cross-Lingual Alignment Score (e.g., mBERT CLAS increased from 14.20 to 50.97).
• Downstream Gains: Models with trilingual alignment showed improved cross-lingual consistency (robustness across CM, EN, and HI test sets) on downstream tasks:
  • Sentiment Analysis: Improved consistency scores by ~36% for mBERT.
  • Hate Speech Detection: Significant gains in consistency, particularly when fine-tuning on Hindi data.

4. Key Contributions

1. Trilingual Corpus: Creation of a systematic parallel corpus (EN, HI, CM) enabling the first mechanistic study of code-mixed representation dynamics.
2. Mechanistic Analysis: Discovery that code-mixed adaptation creates a trade-off between CM alignment and monolingual alignment, and that models rely heavily on English semantics while using native script for uncertainty reduction.
3. Trilingual Alignment Objective: A novel post-training method that forces simultaneous alignment of all three languages, preventing the degradation of monolingual bridges.
4. New Metric: Introduction of CLAS to quantify balanced cross-lingual alignment.
5. Empirical Validation: Demonstration that grounding code-mixed representations in their constituent languages improves robustness in sentiment and hate speech detection.

5. Significance

This paper fundamentally shifts the understanding of how multilingual models handle code-mixed text. It challenges the assumption that code-mixed adaptation is purely beneficial, revealing that it can "unlearn" monolingual alignments.

• Theoretical Impact: It proves that code-mixed text is not a distinct third language but relies on a "twin foundation" of its constituents, with English acting as the primary anchor and native script providing necessary disambiguation.
• Practical Impact: The proposed trilingual alignment strategy offers a lightweight, effective method to improve multilingual models without requiring massive new pre-training corpora. It ensures that models remain robust across all language variants (monolingual and mixed), which is critical for real-world applications in multilingual societies.

Limitations: The study focuses on Hindi-English; future work should generalize to other language pairs. Additionally, the corpus relies on machine translation, which may introduce subtle semantic shifts.