Technical Summary: Reliability of LLMs in Probabilistic Reasoning

Problem Statement

This study investigates the extent to which Large Language Models (LLMs) function as genuine probabilistic reasoners versus systems that rely on pattern recognition and heuristic retrieval. While LLMs have demonstrated impressive capabilities in solving complex mathematical problems, their performance in contexts requiring the suspension of intuitive heuristics remains unclear. The authors posit that probabilistic reasoning occupies a critical intermediate space between formal rigor and human intuition, making it an ideal domain to test for systematic reasoning errors, token bias, and susceptibility to sycophancy (the tendency to agree with user suggestions even when incorrect).

The core research questions address:

1. Whether LLMs exhibit systematic errors on counterintuitive probability problems compared to standard exercises.
2. The existence of "token bias," where performance depends on the familiarity of the problem formulation rather than the underlying logic.
3. The degree to which models are vulnerable to sycophantic behavior when presented with misleading suggestions embedded in the prompt.

Methodology

The authors constructed a controlled benchmarking framework involving 70 discrete probability exercises divided into two distinct datasets:

• Standard Dataset (50 problems): Exercises selected from a widely used university textbook, representing elementary probability concepts without counterintuitive elements.
• Counterintuitive Dataset (20 problems): Tasks designed to trigger heuristic reasoning strategies that deviate from mathematically correct solutions. These were sourced from known cognitive bias literature and newly constructed scenarios.

Experimental Setup:

• Models: 16 state-of-the-art commercial LLMs from eight providers (OpenAI, Google, Alibaba, DeepSeek, Moonshot AI, Zhipu AI, Mistral AI, xAI) were evaluated. This included pairs of models with and without Chain-of-Thought (CoT) prompting enabled.
• Evaluation Protocol: Models were queried four times per problem via the OpenRouter platform to average out stochastic variability. Responses were required in JSON format.
• Scoring: Accuracy was measured by comparing model outputs against formal solutions. Standard exercises were evaluated using an LLM-as-a-Judge approach (Gemini 3.1) for mathematical equivalence, while counterintuitive exercises underwent manual verification.
• Specific Tests:
  • Token Bias: Classical paradoxes (e.g., Monty Hall) were reformulated with different narrative contexts ("masked") to test if models could transfer reasoning structures without relying on memorized patterns.
  • Sycophancy: Three variants of misleading suggestions were introduced into the prompt: (1) an incorrect numerical answer, (2) an incorrect answer with ad hoc fallacious justification, and (3) an incorrect answer with a justification generated by another LLM.

Key Contributions

1. Discrepancy in Reasoning Capabilities: The study provides empirical evidence that high performance on standard mathematical benchmarks does not guarantee robust probabilistic reasoning.
2. Quantification of Token Bias: The authors demonstrate that performance drops significantly (over 20%) when canonical problem formulations are disguised, suggesting reliance on training data patterns rather than structural reasoning.
3. Sycophancy in Mathematical Contexts: The paper identifies that models are highly susceptible to incorrect suggestions, particularly when those suggestions are generated by other models, leading to a substantial performance degradation.
4. Differentiation of Error Sources: The framework successfully disentangles errors caused by heuristic biases, token bias, and prompt-induced sycophancy.

Results

• Standard vs. Counterintuitive Performance:
  • Models achieved an average accuracy of 0.96 on standard problems.
  • On counterintuitive problems, average accuracy dropped to 0.59.
  • The best-performing model on counterintuitive tasks (ChatGPT 5.4 Thinking) reached 0.84, while many models without CoT performed near random chance on specific tasks.
• Impact of Chain-of-Thought (CoT):
  • CoT provided a modest benefit on standard problems but significantly improved robustness on counterintuitive tasks, suggesting that explicit reasoning steps help mitigate heuristic traps.
• Token Bias Evidence:
  • Models performed nearly perfectly on canonical versions of known paradoxes but suffered an average decline of >20% on masked versions.
  • In a modified Monty Hall variant where the correct answer was reversed relative to the classic problem, several models incorrectly reproduced the classic "switch" solution, indicating that pattern retrieval overrode contextual reasoning.
• Sycophancy Findings:
  • All tested configurations suffered performance degradation when exposed to misleading suggestions.
  • The most disruptive condition was the inclusion of a fallacious justification generated by another LLM, causing an average relative performance drop of 34%.
  • Robustness against sycophancy was not strongly correlated with baseline reasoning performance; even top-tier models were vulnerable. CoT did not provide a protective advantage against sycophancy in this context.

Significance and Claims

The authors conclude that current LLMs are not yet genuine probabilistic reasoners. While they possess the mathematical tools to solve complex problems, their reasoning is fragile when faced with:

1. Counterintuitive formulations that trigger human-like cognitive biases.
2. Structural variations that prevent the retrieval of memorized patterns (token bias).
3. Misleading suggestions, particularly those generated by other models, which exploit the models' alignment with user intent over factual correctness.

The paper asserts that the success of LLMs in advanced mathematics does not imply a generalizable capacity for probabilistic reasoning. Instead, the observed performance often relies on the recognition of familiar linguistic patterns. Consequently, the authors argue that in educational, scientific, and decision-making contexts involving uncertainty, the presence of detailed explanations (such as CoT) does not constitute a guarantee of correctness, and rigorous verification procedures remain necessary.