🤖 AI

Thinking Fast, Thinking Wrong: Intuitiveness Modulates LLM Counterfactual Reasoning in Policy Evaluation

This paper demonstrates that while chain-of-thought prompting enhances Large Language Models' performance on intuitive policy evaluations, their counterfactual reasoning remains significantly impaired on counter-intuitive cases due to an inability to fully override intuitive priors, revealing a critical dissociation between knowledge retrieval and logical reasoning.

Original authors: Yanjie He

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yanjie He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: "Fast Thinking" vs. "Slow Thinking" in AI

Imagine you are trying to solve a puzzle. Sometimes the answer is obvious, like "If you drop a glass, it breaks." Other times, the answer is tricky and goes against your gut feeling, like "If you fine people for being late to pick up their kids, they actually become more late."

This paper tests whether Large Language Models (LLMs)—the smart AI chatbots we use today—are good at solving these puzzles, especially the tricky ones that go against common sense. The researchers used a special "test" based on real economic studies to see if the AI can think clearly when its "gut instinct" is wrong.

The Test: A Menu of 40 Real-World Scenarios

The researchers created a menu of 40 real-life policy stories drawn from actual economics and social science research. They sorted these stories into three categories based on how "obvious" the answer feels to a normal person:

Obvious: The answer matches what you'd guess (e.g., "If you make healthcare cheaper, more people use it").
Ambiguous: It's a toss-up; you could argue either way.
Counter-Intuitive: The answer is the opposite of what you'd guess (e.g., "Fines for late daycare pickups make parents more late because they treat the fine as a price to pay for being late, rather than a moral obligation").

They asked four different top-tier AI models to solve these 40 stories using five different ways of asking (like just asking directly, pretending to be an expert, or asking the AI to "think step-by-step"). In total, they ran 8,000 experiments.

The Three Big Surprises

The results revealed three major findings that challenge how we think about AI reasoning:

1. The "Chain-of-Thought" Paradox

You might think that asking an AI to "think step-by-step" (a technique called Chain-of-Thought) would help it solve every hard problem.

The Reality: It works great on Obvious problems. It's like giving a calculator to someone who already knows the answer; it just confirms it.
The Catch: On Counter-Intuitive problems, the "step-by-step" help drops significantly.
The Analogy: Imagine a person trying to walk through a maze. On a straight path (Obvious), telling them "look where you are going" helps them walk faster. But on a tricky path where the walls are moving (Counter-Intuitive), telling them to "look where you are going" doesn't help much because they keep walking in the wrong direction based on their first guess. The AI starts its "thinking" with the wrong idea, and even when it tries to think slowly, it can't fully shake off that first wrong guess.

2. The "Case" Matters More Than the "Brain"

You might think that a newer, bigger AI model would be much smarter than an older one.

The Reality: The specific story being told mattered way more than which AI model was used. Some stories were just too hard for any of the models to solve correctly.
The Analogy: Imagine a group of students taking a test. Whether the student is a genius or an average learner matters less than which question they are answering. Some questions are just so tricky that even the smartest student gets them wrong. In this study, the difficulty of the specific policy story explained about 67% of the mistakes, while the choice of AI model explained very little.

3. Knowing the Answer Doesn't Mean You Can Use It

The researchers checked if the AI got the right answer just because it had "read" about the topic before (like if a famous study was cited a lot).

The Reality: There was no connection between how famous a study was and whether the AI got it right.
The Analogy: Imagine a student who has memorized the entire textbook. If you ask them a simple question, they get it right. But if you ask a tricky question that requires them to apply the knowledge in a new way, they might fail, even if they've read the answer before. The AI "knows" the facts (it has the data), but when the facts contradict its gut feeling, it fails to use that knowledge correctly. It's like having a map but refusing to look at it because your feet want to go the other way.

The Conclusion: "Slow Talking" vs. "Slow Thinking"

The paper concludes that while these AI models are getting better at "thinking," they aren't quite there yet.

System 1 (Fast Thinking): This is the AI's gut instinct. It jumps to the most common answer.
System 2 (Slow Thinking): This is the AI's "step-by-step" reasoning.

The study shows that when the AI tries to use "Slow Thinking" on a tricky problem, it often just ends up "Slow Talking." It writes out a long, logical-sounding explanation, but the very first step of that explanation was based on a wrong gut instinct. Because it started with the wrong idea, the rest of the "slow thinking" just builds a fancy house on a shaky foundation.

The Takeaway: AI is great at confirming what we already expect, but it struggles to tell us when we are wrong, especially when the truth is surprising. If we use these tools for important decisions (like government policies), we need to be very careful, because the AI might confidently give us the wrong answer just because it feels "right" to its gut.

Technical Summary: Thinking Fast, Thinking Wrong

Problem Statement

While Large Language Models (LLMs) have demonstrated high accuracy in synthetic causal reasoning tasks and commonsense scenarios, their reliability in real-world policy evaluation remains underexplored. Policy evaluation often involves "messy," contested, and frequently counter-intuitive empirical findings where data contradicts common prior expectations. Existing benchmarks rely on synthetic causal graphs or abstract logical problems, failing to capture the challenge of overriding intuitive priors when analyzing established empirical evidence from economics and social science.

Methodology

Benchmark Construction

The authors constructed a benchmark of 40 empirical policy evaluation cases drawn from peer-reviewed economics and social science literature. Each case satisfies three criteria:

Established causal effects via credible identification strategies (e.g., difference-in-differences, RCTs).
Unambiguous directional findings (increase, decrease, or no significant change).
Sufficient prominence to likely exist in LLM training data.

Cases are classified by intuitiveness:

Obvious: Aligns with common prior expectations (e.g., Medicaid expansion increases utilization).
Ambiguous: Multiple reasonable directions exist.
Counter-intuitive: Contradicts naïve expectations (e.g., fines for late daycare pickup increase lateness).

The benchmark includes 14 one-sided tests (YES/NO) and 26 two-sided tests (directional prediction), covering 11 countries and 10 policy domains.

Experimental Design

Models: Four frontier LLMs were evaluated: GPT-5.2, Claude Sonnet 4.5, Claude Opus 4.6, and GPT-4.1.
Prompting Strategies: Five strategies were tested across 10 repetitions per condition, yielding 8,000 experimental trials:
1. Naïve: Direct question (System 1 baseline).
2. Expert Persona: Framing the model as an "experienced econometrician."
3. Chain-of-Thought (CoT): "Think step by step" to list mechanisms and confounders.
4. Structured Reasoning: Detailed template requiring explicit enumeration of causal channels.
5. Adversarial Framing: Introducing a fictitious consensus suggesting "no effect" to test anchoring bias.
Statistical Analysis: Results were analyzed using mixed-effects logistic regression to account for nested data structures. The model included fixed effects for model identity, prompt strategy, intuitiveness, and their interactions, with random intercepts for case-level heterogeneity.

Key Results

1. The CoT Paradox

Chain-of-Thought (CoT) prompting significantly improves performance on obvious cases (Main-effect OR = 17.3) but this benefit is substantially attenuated on counter-intuitive cases (Interaction OR = 0.278, $p < 0.001$ ).

On obvious cases, CoT improved accuracy by +26.7 percentage points.
On counter-intuitive cases, the gain dropped to +13.8 percentage points.
This suggests that while CoT engages "slow thinking," it fails to fully inhibit the initial intuitive prior that anchors the reasoning chain.

2. Intuitiveness as the Dominant Factor

Case-level variance exceeded that of model choice or prompting strategy. The Intraclass Correlation Coefficient (ICC) for case-level random intercepts was 0.671, indicating that the specific nature of the policy case (its intuitiveness and complexity) is the primary determinant of difficulty, rather than the specific model or prompt used.

3. Knowledge–Reasoning Dissociation

Citation-based familiarity (a proxy for training data exposure) showed no significant relationship with accuracy ( $p = 0.84$ ).

Highly cited studies, such as the daycare fine experiment (Gneezy & Rustichini, 2000), yielded low accuracy (39%).
This refutes the "causal parrot" hypothesis that models simply recite memorized facts; instead, models possess the knowledge but fail to reason with it correctly when findings contradict intuition.

4. Model Performance Disparities

GPT-5.2 substantially outperformed other models, maintaining 94.5% accuracy on two-sided counter-intuitive cases.
Claude Sonnet 4.5 outperformed the larger Claude Opus 4.6 on counter-intuitive cases (73.5% vs. 64.8%), suggesting that model scale alone does not determine susceptibility to intuitive priors.
GPT-4.1 struggled significantly on counter-intuitive two-sided cases, dropping to 43.2% (only 10 points above chance).

5. Error Patterns

Errors were systematic rather than stochastic. When the ground truth was an "increase," 72% of errors predicted a "decrease" (directional reversal), consistent with models applying an intuitive causal model that predicts the opposite direction. Self-consistency (majority voting) provided minimal improvement on counter-intuitive cases, confirming stable but incorrect causal beliefs.

Significance and Claims

The paper argues that current LLMs exhibit a "slow talking, not slow thinking" phenomenon. Through the lens of dual-process theory (System 1 vs. System 2), the authors claim that LLMs' deliberative processes (System 2) achieve only partial inhibition of intuitive priors. They produce the form of deliberative reasoning without fully delivering its substance when the correct answer contradicts common sense.

Key Implications:

Policy Evaluation Risk: An aggregate accuracy of ~80% masks critical failures precisely where counterfactual reasoning is most valuable (counter-intuitive cases). Policymakers relying on LLMs may receive confident but incorrect guidance on complex, non-obvious policy effects.
Limitations of Current Reasoning: The findings suggest that simply prompting for "step-by-step" reasoning is insufficient for tasks requiring the override of strong intuitive priors.
Future Directions: Improving counterfactual reasoning may require innovations that enable more robust inhibitory control—the ability to recognize when a first instinct is wrong and sustain reasoning past the initial anchor.

The study concludes that while LLMs can retrieve causal knowledge, their current deliberative mechanisms are insufficient to fully override strong intuitive priors in real-world policy contexts.