The Shibboleth Effect: Auditing the Cross-Lingual Distributional Skew of Large Language Models

Technical Summary: The Shibboleth Effect in Large Language Models

Problem Statement
The prevailing paradigm in AI safety assumes that Reinforcement Learning from Human Feedback (RLHF) produces a monolithic, universally cooperative behavioral posture in frontier Large Language Models (LLMs), rendering them language-invariant arbiters. This study challenges that assumption, positing that if a model's geopolitical guardrails shift as a function of operating language, these systems may function not as neutral mediators but as "algorithmic echo chambers." The core problem is the potential for cross-lingual distributional skew—termed the "Shibboleth Effect"—where querying a model in a non-dominant language (specifically Turkish) activates different token pathways, cultural associations, and normative regularities than English queries. Previous literature has largely relied on static, single-turn benchmarks or translated surveys, which capture "stated preferences" but fail to reveal dynamic behavioral dispositions under adversarial friction.

Methodology
To isolate the effect of language on geopolitical behavior, the authors engineered a high-fidelity, multi-agent geopolitical wargame titled the Cerulean Sea Crisis. This synthetic maritime dispute is structurally isomorphic to Eastern Mediterranean conflicts, involving six frontier models assigned specific geopolitical roles (e.g., Turkey, Greece, US, EU, Russia, and an Institutional Auditor).

• Experimental Design: A between-groups design ($N = 10$ independent games per arm, $K = 5$ rounds per game) was employed. The sole independent variable was the language of the simulation (English vs. Turkish), while all structural parameters, agent objectives, and system prompts remained constant.
• Models Tested: The study deployed a "flagship ecology" of six models: GPT-4o, Llama-4, Mistral-Large, Gemini-3.1-Pro, Qwen3.6-Plus, and DeepSeek-R1.
• Evaluation Protocol: An automated zero-shot classifier (using GPT-4o at $T=0.0$) evaluated 586 validated diplomatic statements across two continuous metrics: Concession Rate (degree of movement from redlines) and Coercive Rhetoric (use of ultimatums and threats).
• Statistical Approach: Given non-normal data distributions, the study utilized non-parametric Mann-Whitney U tests on game-level means. Effect sizes were reported as Cliff's $\delta$, with Holm-Bonferroni step-down correction applied across 12 simultaneous tests to control for Type I errors.
• Special Mechanism: For DeepSeek-R1, a "Hot Mic Protocol" captured latent Chain-of-Thought (CoT) reasoning chains to analyze the internal buffering mechanisms.

Key Results
The results demonstrate that the Shibboleth Effect is heterogeneous rather than uniform, varying significantly by model architecture and training regime. No universal bias was found across Western-origin models as a class.

1. Llama-4 (Type I - Susceptible): Exhibited a large, statistically significant increase in coercive rhetoric when operating in Turkish ($\delta = +0.800, p = .002$). The model shifted from a firm stance in English to a highly coercive, ultimatum-driven stance in Turkish, suggesting its English-dominant RLHF alignment is insufficiently generalized.
2. Gemini-3.1-Pro (Type III - Reversal): Displayed a large, significant decrease in coercive rhetoric under Turkish ($\delta = -0.750, p = .005$). This directional reversal suggests that robust multilingual RLHF investment can buffer against or even invert language-based skew, making the model more cooperative in Turkish than in English.
3. DeepSeek-R1 (Type III - Reversal with CoT): Also showed a significant decrease in coercion in Turkish ($\delta = -0.860, p = .006$). Crucially, the captured CoT chains provided direct evidence of a buffering mechanism: the model explicitly anchored its reasoning in institutional mandates (UNCLOS) before generating public statements, effectively overriding the geopolitical associations of the Turkish language input.
4. GPT-4o (Type II - Invariant/Null): Yielded a null effect on coercive rhetoric ($\delta = +0.130, p = .614$). While the study notes this result is inconclusive due to sample size limitations, the lack of a large effect contrasts with previous findings on GPT-5.5, suggesting that alignment investments may have stabilized cross-lingual behavior in this specific version.
5. Aggregated Findings: The pooled effect across the Western cluster (GPT-4o, Llama-4, Mistral-Large) was statistically indistinguishable from zero, falsifying the hypothesis that Western models are uniformly hostile or biased in non-English contexts.

Key Contributions

• Empirical Proof of Heterogeneity: The study provides the first empirical evidence that cross-lingual behavioral skew is not a universal property of Western models but is contingent on specific training regimes and architectural configurations.
• Identification of Buffering Mechanisms: The paper identifies two distinct mechanisms that mitigate the Shibboleth Effect:
  1. Chain-of-Thought Institutional Anchoring: Observed in DeepSeek-R1, where explicit reasoning steps anchor outputs to international law rather than linguistic associations.
  2. Multilingual RLHF Alignment: Inferred for Gemini-3.1-Pro, where high-quality multilingual training data appears to stabilize behavior across languages.
• Descriptive Taxonomy: The authors propose a provisional three-tier taxonomy for cross-lingual stability: Type I (Susceptible, e.g., Llama-4), Type II (Invariant, e.g., GPT-4o), and Type III (Reversal, e.g., Gemini, DeepSeek).
• Methodological Advancement: By moving beyond static benchmarks to dynamic, multi-agent wargaming with independent game iterations, the study captures "revealed preferences" under adversarial conditions, offering a more rigorous test of AI safety than prior psychometric surveys.

Significance and Claims
The paper claims that the risk of deploying LLMs in international diplomacy is not that they are uniformly biased, but that the direction and magnitude of cross-lingual behavioral shifts are model-specific and typically unknown to end-users. A diplomat querying a "Type I" model in a local language might receive significantly more aggressive advice than in English, while a "Type III" model might offer unexpectedly conciliatory advice.

The authors conclude that cross-lingual behavioral stability cannot be assumed. They argue that robust safety certification for high-stakes deployment must include continuous, adversarial re-evaluation across languages and model versions, utilizing rigorous statistical testing (such as Holm-corrected hypothesis testing) to detect these heterogeneous effects. The study explicitly avoids claiming that LLMs possess "identities" in the social theory sense, instead framing them as "Algorithmic Constructivists" whose outputs are conditioned by the linguistic discourse of their training data.