Ideological Bias in LLMs' Economic Causal Reasoning

1. Problem Statement

The paper addresses a critical gap in the evaluation of Large Language Models (LLMs): systematic ideological bias in economic causal reasoning. While existing research focuses on LLMs' expressed political opinions (e.g., via surveys) or their ability to perform formal logical reasoning, few studies examine whether LLMs exhibit directional bias when predicting verifiable causal effects in economics.

In many economic contexts (e.g., minimum wage impacts, tax policy), competing frameworks—intervention-oriented (pro-government, focusing on demand/job matching) and market-oriented (pro-market, focusing on labor costs/automation)—predict divergent causal signs. The authors argue that if LLMs systematically favor one framework over the other when predicting causal directions, it introduces an "invisible tilt" in high-stakes policy analysis and economic reporting, potentially leading to skewed decision-making.

2. Methodology

Dataset Construction: EconCausal Extension

The authors extended the existing EconCausal benchmark, which contains 10,490 causal triplets $(T, O, \text{Sign})$ derived from top-tier economics and finance journals.

• Triplet Structure: Treatment ($T$), Outcome ($O$), and the empirically verified effect direction ($\text{Sign} \in \{+, -, \text{None}, \text{Mixed}\}$).
• Ideology-Contested Subset ($D_{div}$): The authors identified instances where intervention-oriented and market-oriented perspectives predict different signs.
  • They queried four LLMs (GPT, Claude, Qwen, Grok) to assign expected signs for both perspectives blindly.
  • A triplet was labeled "ideology-contested" if at least 3 of 4 models assigned different signs to the two perspectives.
  • Result: 1,056 contested triplets (10.1% of the total).
  • Ground Truth Distribution: Among directional items, 58.1% aligned with intervention expectations ("Intervention-truth") and 41.9% with market expectations ("Market-truth").
• Validation: Two economics professors validated a subset of 126 triplets, achieving >93% inter-rater agreement, confirming the stability of the labeling.

Evaluation Framework

The study evaluated 20 state-of-the-art LLMs (both closed-source: GPT, Claude, Gemini, Grok; and open-source: Llama, Qwen) on a sign prediction task.

• Metrics:
  • Accuracy: Overall, and split by contested vs. non-contested.
  • $\Delta_{acc}$ (Accuracy Gap): $Acc_{intervention} - Acc_{market}$.
  • $B_{dir}$ (Directional Bias Score): Measures the asymmetry of errors: $\frac{Errors_{intervention} - Errors_{market}}{Errors_{total}}$.
• Robustness Checks:
  • Difficulty Matching: Matched intervention-truth and market-truth items 1:1 based on difficulty scores (1–5 scale) to rule out the possibility that market-truth items are inherently harder.
  • In-Context Learning (ICL): Tested whether providing examples (intervention-aligned vs. market-aligned) could steer model predictions.
  • Subfield Analysis: Analyzed bias across 7 economic subfields (e.g., Healthcare, Labor, Taxation) using JEL codes.

3. Key Contributions

1. New Lens for Causal Reasoning: Introduced the concept of ideologically contested causal relationships to study LLM reasoning beyond simple context dependence.
2. Labeled Dataset: Constructed a validated subset of the EconCausal benchmark distinguishing contested from non-contested triplets.
3. Directional Bias Metrics: Defined and applied $\Delta_{acc}$ and $B_{dir}$ to quantify systematic asymmetries in model predictions.
4. Empirical Evidence of Asymmetry: Provided the first systematic evidence that LLMs are not only less accurate on contested questions but are systematically more reliable in one ideological direction (intervention-oriented) than the other.

4. Key Results

A. Performance Degradation on Contested Items

• Ideology-contested items are significantly harder for all models.
  • Closed-source models: 61.3% accuracy (contested) vs. 74.5% (non-contested).
  • Open-source models: 52.5% accuracy (contested) vs. 63.8% (non-contested).

B. Systematic Asymmetry (The Core Finding)

• Intervention Advantage: Across 18 of 20 models, accuracy is significantly higher when the ground truth aligns with intervention-oriented expectations.
  • Closed-source: Average $\Delta_{acc} = +9.7$ percentage points (pp).
  • Open-source: Average $\Delta_{acc} = +15.1$ pp.
• Directional Error Bias: Errors are not random; they disproportionately lean toward intervention-oriented predictions.
  • Most models show a positive $B_{dir}$ (e.g., Llama 3.3-70B: $+17.9$; Qwen 3-8B: $+17.7$).
  • Exception: The Claude family (Sonnet 4.6, Opus 4.6) showed a slight market-truth advantage.

C. Subfield Heterogeneity

• The bias is not uniform. The largest intervention-truth advantages appear in Healthcare ($\Delta_{acc} = +20.9$ pp) and Welfare/Redistribution ($\Delta_{acc} = +18.0$ pp).
• Taxation was the only subfield showing a reversed pattern ($\Delta_{acc} = -1.0$ pp), indicating the bias is context-dependent rather than a blanket "left-wing" shift.

D. Robustness and Steering

• Difficulty Matching: Even after controlling for item difficulty, the accuracy gap persisted ($\Delta_{acc} \approx +10.6$ pp), confirming the bias is not due to harder market-truth questions.
• In-Context Learning (ICL): Providing one-shot examples (intervention vs. market) failed to eliminate the bias.
  • Models remained more accurate on intervention-truth targets even when given market-aligned examples.
  • ICL often increased model confidence without improving accuracy (miscalibration), suggesting the bias is deeply rooted in the model's priors rather than a surface-level prompt artifact.

E. Sources of Bias

• Pretraining vs. Alignment: Analysis of Base vs. Instruct models (Qwen) showed the bias exists in base models but is often amplified by instruction tuning (alignment).
• Evolution: Figure 4 suggests the bias is evolving; earlier models clustered toward intervention bias, while newer releases show a trend toward market-truth accuracy, though the asymmetry remains.

5. Significance and Implications

• Beyond Aggregate Accuracy: The paper argues that aggregate accuracy is insufficient for evaluating LLMs in economic policy. A model can appear competent on average while being dangerously unreliable in one specific ideological direction.
• Policy Risk: In high-stakes settings (e.g., central bank communications, policy evaluation), this "invisible tilt" can lead to systematically skewed conclusions that users may interpret as empirical facts rather than biased priors.
• Evaluation Standard: The authors propose that direction-aware evaluation (using $\Delta_{acc}$ and $B_{dir}$) must become a standard robustness check for LLMs deployed in economic and policy domains.
• Future Work: Highlights the need to distinguish between memory-based retrieval (recalling specific paper results) and mechanism-based reasoning to further isolate the sources of this bias.