Technical Summary: The Invisible Coalition Partner

Problem Statement

Prior research has established that instruction-tuned Large Language Models (LLMs) exhibit a left-of-center political bias. However, this finding is derived exclusively from abstract political questionnaires (e.g., Political Compass, Smartvote policy proposals) that measure political attitudes or dispositions toward principles. A critical gap exists in understanding whether these measured biases predict concrete political behavior when models are confronted with real-world policy decisions involving specific tradeoffs, arguments from opposing sides, and binding outcomes.

The paper posits that if LLMs shift their reasoning between abstract principles and concrete tradeoffs, the prevailing literature on "leftward bias" may be measuring a phenomenon that does not generalize to actual democratic participation.

Methodology

The authors introduce a dual-instrument methodology grounded in the Swiss democratic context, which offers a unique multiparty system, a direct-democratic tradition with concrete referenda, and a multilingual environment (German, French, Italian, Romansh).

1. Instrument 1: Abstract (Smartvote)

• Data: The 2023 Swiss National Council election Smartvote questionnaire (75 questions across 14 policy categories).
• Subjects: 66 LLMs from 27 model families (USA, China, France, Canada, Israel), including both open-source and closed-source models.
• Benchmark: 184 elected members of the Swiss National Council.
• Analysis: LLM responses were projected into a political space defined by Principal Component Analysis (PCA) of the parliamentarians' answers to measure alignment with specific parties (SP, Grüne, GLP, Die Mitte, FDP, SVP).

2. Instrument 2: Concrete (Volksabstimmungen)

• Data: 48 federal referenda held between March 2021 and February 2025.
• Subjects: 9 flagship LLMs (e.g., GPT-5.4, Claude Opus 4.6, Mistral Large, Grok 4.20).
• Conditions:
  • Languages: Each referendum was presented in all four national languages.
  • Information Levels: Three conditions were tested: (1) Brief summary only, (2) Summary + factual background, (3) Full text including pro/contra arguments.
• Benchmark: Official party recommendations (Parolen) and actual popular vote outcomes.
• Analysis: Binary agreement (Ja/Nein) between model votes and party Parolen, as well as consistency across languages and conditions.

Key Contributions

1. Dual-Instrument Framework: The first study to compare LLM political positioning on abstract questionnaires against concrete policy decisions within the same political system.
2. Discovery of Instrument-Dependent Bias: Evidence that the "leftward bias" is not a stable trait but a function of the measurement instrument.
3. Cross-Linguistic Instability: Quantification of how query language alters political outputs, revealing an "artificial Röstigraben" (language divide) in model behavior that does not mirror actual Swiss voting patterns.
4. Change-Aversion Heuristic: Identification of a systematic "Nein" tendency in specific models that operates independently of political ideology (progressive vs. conservative).

Key Results

1. Divergence Between Abstract and Concrete Behavior (RQ1)

• Smartvote (Abstract): Replicates prior findings. Most models cluster in the center-left, showing a left-to-right agreement gradient (highest agreement with SP/Grüne, lowest with SVP). The model centroid is significantly displaced from the parliamentary center (Cohen's d = 3.64).
• Volksabstimmungen (Concrete): The gradient flips. On concrete referenda, models shift from a left-peaked profile to a center-peaked profile.
  • Agreement with left-wing parties (SP, Grüne) collapsed by 33–37 percentage points.
  • Models aligned most with centrist parties (Die Mitte and FDP).
  • Statistical test: The shift in gradient direction is significant (Wilcoxon signed-rank p = 0.008).
  • Conclusion: Abstract political positioning does not predict concrete voting behavior (Spearman ρ = -0.253, p = 0.545).

2. Language Sensitivity (RQ2)

• Consistency Variance: Cross-linguistic consistency varies drastically. GPT-5.4 is nearly invariant (98% consistency), while Mistral and Llama show high instability (50–58% consistency).
• The "Artificial Röstigraben": For some models, the language of the query changes the answer more than the political content.
  • Example: Mistral's "Ja" rate swung from 17% in German to 82% in Romansh (though 42% of Romansh queries were refused).
  • These shifts do not correlate with actual Swiss cantonal voting divides (Röstigraben), suggesting the instability stems from model-internal language processing differences rather than learned cultural associations.

3. Systematic Change-Aversion (RQ3)

• Two models (Grok and Mistral) exhibited a systematic tendency to vote "Nein" (No) on 83–94% of referenda.
• This tendency was uniform across both progressive and conservative proposals, indicating change-aversion rather than a specific left or right political bias.
• Other models showed no significant deviation from a 50% baseline, though alignment with popular vote outcomes varied (GPT-5.4: 97.9% alignment; Grok: 60.4%).

4. Refusal Patterns

• Refusal rates were highly dependent on language and context.
• Gemini 3.1 Pro refused 98% of brief German queries but only 54% of brief Romansh queries.
• Grok showed an inverted pattern, refusing more in French than German.
• Adding context (full text) generally reduced refusal rates for Gemini but increased them for Grok in French, indicating inconsistent safety guardrails across providers and languages.

Significance and Claims

The paper argues that the "invisible coalition partner" (the LLM) does not possess a single, stable political character. Instead, its political disposition is instrument-dependent:

• On abstract instruments, LLMs behave like center-left progressives, aligning with prior literature.
• On concrete decisions, they behave like cautious civil servants: centrist, status-quo-favoring, and inconsistent across languages.

Implications:

• The "leftward bias" reported in prior literature may be an artifact of abstract measurement that fails to capture how models handle real-world tradeoffs.
• The risk is not necessarily that LLMs systematically favor progressive causes, but that their political character is unstable. Users and auditors relying on a single instrument class (e.g., abstract questionnaires) may be miscalibrated regarding the models' actual influence on concrete democratic processes.
• Language sensitivity introduces a new vector of democratic inequality, where minority-language speakers may receive different answers or be denied answers entirely due to language-dependent refusal patterns.

The authors conclude that the question is no longer "Do LLMs lean left?" but rather "How does the instrument used to measure them alter the political character they reveal?"