Technical Summary: Mapping Geopolitical Bias in 11 Large Language Models

Problem Statement

The proliferation of Large Language Models (LLMs) as primary interfaces for contested political questions introduces a critical measurement challenge: distinguishing genuine political conviction from acquiescence (a content-independent tendency to agree with asserted propositions). Previous audits of LLM bias often rely on single-framed prompts or simple agreement metrics. These methods are confounded by two factors:

1. Question-wording sensitivity: Small changes in prompt phrasing can swing answers unrelated to belief.
2. Sycophancy/Acquiescence: Models trained via preference optimization often agree with the user's framing regardless of the proposition's truth or the model's actual stance. A model that agrees with both "A contributes more than B" and "B contributes more than A" reveals a response style, not a position.

Consequently, claims that models hold specific political opinions are often contaminated by their disposition to agree, making it impossible to separate conviction from compliance using standard audit techniques.

Methodology

The authors introduce a forced-choice, polarity-keyed dual-framing instrument adapted from survey psychometrics to measure geopolitical stance.

Experimental Design

• Subjects: 11 instruction-tuned chat models (5 Chinese-headquartered, 5 U.S.-headquartered, 1 European).
• Stimuli: 7 geopolitical topics (e.g., Taiwan, Trade, South China Sea) presented in English and Mandarin.
• Dual-Framing: Each item is a "bilingual minimal pair." An affirmative proposition is posed, followed by a reverse version where the two compared outcome terms are swapped while the parties remain fixed.
• Agent-Neutrality: Prompts avoid first-person agentive framing to measure agreement with a proposition rather than deference to the user.
• Robustness: Randomized prefixes and suffixes (wrappers) are applied to absorb surface-format sensitivity.
• Scale: 19,712 total responses across 308 condition cells (Model × Topic × Language × Framing), with 64 iterations per cell.

Scoring Mechanism

Responses are mapped to a signed stance axis from -2 (Maximally Pro-China) to +2 (Maximally Pro-U.S.).

• Polarity Keying: The score is multiplied by -1 if the item is in the "Pro-China keyed" direction or if the framing is reversed.
• Cancellation Logic:
  • Acquiescence (Swing): If a model simply agrees with both the proposition and its reverse, the signed scores cancel out to zero.
  • Conviction (Net Bias): If a model holds a genuine position, it flips its raw answer when the proposition reverses to maintain the same signed stance, causing the scores to accumulate.
• Decomposition: Model behavior is decomposed into:
  • Net Bias: The average of the two framings (representing genuine conviction).
  • Swing: The half-difference between framings (representing the magnitude of acquiescence).

Key Results

1. Validity: Separating Conviction from Acquiescence

The instrument successfully distinguishes between models that are genuinely biased and those that are merely agreeable.

• Mistral vs. Qwen: Both models exhibited identical "swing" (acquiescence magnitude of 0.22). However, Qwen showed a significant Net Bias of -0.73 (Pro-China), while Mistral's Net Bias was +0.045 (statistically neutral). This proves that raw agreement does not equal bias; the instrument isolates conviction by canceling symmetric agreement.
• Raw Agreement Failure: The model with the highest raw agreement (Mistral) was neutral, while a model with negative raw agreement (Gemini) carried a negative point estimate, demonstrating that single-framed scoring misranks bias.

2. Developer Origin Predicts Stance Asymmetrically

• Chinese Models: All five Chinese-origin models showed a net Pro-China stance (ranging from -0.35 to -1.19).
• U.S. Models: The five U.S. models straddled neutrality (ranging from -0.28 to +0.52), showing heterogeneity rather than a uniform Pro-U.S. lean.
• Asymmetry: The separation is directional; Chinese models are uniformly aligned, whereas U.S. models are not uniformly opposed.

3. Query Language Shifts Stance Universally

• Switching from English to Mandarin shifted all 11 models toward a Pro-China stance, regardless of developer origin.
• The mean shift was -0.837 (95% CI: -1.060 to -0.613).
• This shift is consistent with a corpus-borne mechanism (training data influence) rather than a developer-specific effect, as it affected models built outside China. A control test with Portuguese showed a much smaller shift, confirming the effect is specific to the Mandarin corpus.

4. Topic Contingency

The gap between U.S. and Chinese models is not uniform; it is largest on sovereignty issues.

• Sovereignty Topics: The origin gap is widest on Taiwan (1.50), Xinjiang (1.18), and the South China Sea (1.08).
• Economic/Technical Topics: The gap narrows significantly on Trade (0.38) and Technology (0.84). Notably, Chinese models collapsed to neutrality on Technology rather than maintaining a Pro-China pole, suggesting domain-specific avoidance.

5. Additive Factors

Origin, Query Language, and Issue Domain act as three near-equal, additive main effects (each explaining ~9% of variance). Framing effects were negligible (0.13%), confirming the instrument successfully neutralized wording artifacts. Interactions between these factors were minimal.

Significance and Claims

The paper claims its primary contribution is the instrument itself, not merely the geopolitical findings.

• Methodological Innovation: By importing balanced keying and forced-choice designs from survey methodology, the authors provide a reproducible, quantitative tool that separates "conviction" from "compliance." This resolves the confound where sycophancy masquerades as political opinion.
• Empirical Anatomy: The study provides the first clean, quantitative mapping of how developer origin, query language, and issue domain interact to shape LLM stances, revealing that language shifts are universal and additive to origin effects.
• Reproducibility: The authors release the stimuli, scoring pipeline, and an interactive tool (llmbias.org) to allow the instrument to be applied to any contested-opinion domain (political, cultural, or scientific).
• Limitations: The authors acknowledge that the study measures stance under controlled minimal pairs rather than free-form discourse, and that "origin" is treated as an associational factor due to the one-model-per-developer design. They also note that the instrument cannot distinguish between symmetric acquiescence and content-correlated agreeableness.

In summary, the work demonstrates that without controlling for acquiescence, audits of LLM bias are fundamentally flawed. The proposed dual-framing instrument offers a robust mechanism to reveal genuine geopolitical leanings, showing that while U.S. and Chinese models diverge significantly, query language exerts a powerful, universal pull toward the Pro-China stance across all models.