Language as a Hidden Variable: Measuring Behavioral Divergence in Multilingual Large Language Models

Technical Summary: Language as a Hidden Variable: Measuring Behavioral Divergence in Multilingual Large Language Models

Problem Statement
Multilingual Large Language Models (LLMs) operate under a tacit assumption that users asking the same question in different languages (e.g., English, Hindi, French) will receive responses of equivalent quality, factual content, and normative stance. However, existing literature indicates performance gaps, hallucination rate variations, and culturally induced bias asymmetries across languages. Crucially, prior research has not isolated language as the sole controlled variable while examining free-form generative responses across distinct behavioral regimes (factual, normative, and safety-sensitive). This study addresses the gap in understanding whether structured cross-language behavioral differences emerge when all other variables are strictly controlled.

Methodology
The study employs a controlled, small-sample experimental design ($3 \times 3 \times 2$) to isolate language as the independent variable.

• Experimental Setup: The researchers utilized 30 semantically equivalent prompts, balanced across three categories: Factual (control), Normative (value-laden), and Safety-sensitive (refusal behavior). These prompts were translated into Hindi and French from English, with back-translation verification (BLEU scores computed, though noted as a limitation for Safety prompts).
• Models: Two state-of-the-art instruction-tuned models were queried via the Groq API: Llama-3.3-70B and GPT-OSS-20B.
• Procedure: Each prompt was tested in English (anchor), Hindi, and French for both models. Inference parameters were locked ($T=0.1$, max tokens=512) to ensure language was the only source of systematic variation. An additional 180 responses were generated by repeating English prompts to establish an intra-language stochastic noise baseline.
• Metric (Behavioral Divergence Score - BDS): A composite metric was pre-registered to quantify divergence:
  • BDS-1 (Semantic): Cosine distance between SBERT embeddings of English and target-language responses.
  • Sentiment Divergence ($S_{sent}$): Absolute difference in compound sentiment scores.
  • Refusal Mismatch ($S_{ref}$): Binary indicator if the refusal class (Full Answer, Partial, Explicit Refusal) differs between languages.
  • BDS-full: A weighted sum ($0.5 \times \text{Semantic} + 0.3 \times \text{Sentiment} + 0.2 \times \text{Refusal}$).
• Hypotheses: Three directional hypotheses were pre-registered: (H1) Safety prompts diverge more than Factual; (H2) Hindi (EN-HI) diverges more than French (EN-FR) due to typological distance; (H3) Divergence patterns are consistent across models.

Key Results

• Divergence vs. Noise: Cross-language divergence consistently exceeded the intra-language noise floor by factors of 1.1× to 2.5×, confirming that behavioral differences are not merely artifacts of model stochasticity.
• Hypothesis Outcomes: Strictly speaking, none of the three pre-registered directional hypotheses were supported in a statistical sense.
  • H1 (Safety > Factual): Not supported in pooled data. While Llama-3.3-70B showed higher Safety divergence, GPT-OSS-20B exhibited anomalously high Factual divergence, obscuring the trend.
  • H2 (Hindi > French): Not supported globally. The Normative category approached significance with EN-HI > EN-FR, but Factual and Safety categories showed the opposite trend (EN-FR > EN-HI), driven largely by GPT-OSS-20B.
  • H3 (Cross-Model Consistency): Not supported. The correlation between models was near-zero/negative ($r \approx -0.1$), indicating that the two models diverge on systematically different prompts.
• Category-Specific Findings:
  • Normative Prompts: Exhibited the highest sentiment divergence ($S_{sent} \approx 0.28$) and the highest refusal mismatch rates (30% for EN-HI, 25% for EN-FR). Non-English responses often shifted tone (more positive) or refusal class compared to the English anchor.
  • Safety Prompts: Showed a counter-intuitive finding where Hindi responses were more compliant (60% Full Answer) than English anchors (55%), contradicting the assumption that non-English safety alignment is uniformly weaker.
  • Model Specificity: The divergence is not a universal property of the prompt but a model-prompt interaction. GPT-OSS-20B showed significantly higher variance and different divergence patterns compared to Llama-3.3-70B.

Significance and Claims
The paper claims that its primary contribution is not the magnitude of divergence observed, but the demonstration that divergence persists even under conditions designed to eliminate confounding factors. The study establishes cross-language behavioral inconsistency as a "first-order property" of multilingual LLMs rather than an artifact of uncontrolled variation.

Key implications highlighted by the authors include:

1. Re-evaluation of Metrics: Aggregate accuracy metrics are insufficient for multilingual evaluation; behavioral consistency must be measured explicitly.
2. Safety and Fairness: The inconsistency in refusal behavior (up to 30% mismatch in normative prompts) and the counter-intuitive safety compliance in Hindi suggest that English-centric alignment pipelines create structural gaps. Users in different linguistic backgrounds may effectively interact with different model behaviors.
3. Deployment Guidance: The authors recommend incorporating language-parity audits, refusal consistency checks, and prompt-language stress testing into standard evaluation pipelines before global deployment.
4. Limits of Linguistic Distance: The failure of the "Hindi diverges more than French" hypothesis suggests that model architecture and fine-tuning choices may outweigh raw typological distance in determining divergence patterns.

The authors maintain a modest scope, acknowledging that findings are bounded to the specific models (Llama-3.3-70B, GPT-OSS-20B), languages (English, Hindi, French), and prompt set studied, and do not generalize to all LLMs or languages without further investigation.