💬 NLP

Implicit Probabilistic Reasoning Does Not Reflect Explicit Answers in Large Language Models

This paper reveals that while Large Language Models demonstrate competence in explicit probabilistic reasoning via multiple-choice questions, their implicit probabilistic reasoning during text generation often diverges significantly from ground truth due to susceptibility to various contextual factors, highlighting a critical limitation in conventional evaluation methods.

Original authors: Manuel Mondal, Ljiljana Dolamic, Gérôme Bovet, Philippe Cudré-Mauroux, Julien Audiffren

Published 2026-02-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Manuel Mondal, Ljiljana Dolamic, Gérôme Bovet, Philippe Cudré-Mauroux, Julien Audiffren

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are interviewing a brilliant, well-read student for a job. You ask them a tricky math question: "If I roll two dice, what are the odds the numbers add up to 7?"

The student raises their hand confidently and says, "1 in 6!" They are correct. They can explain the math perfectly. You think, "Great, this person understands probability!"

But then, you ask them to play a game. You say, "Okay, let's pretend we just rolled two dice. I'm going to write down the result. What number do you think I wrote?"

Without thinking twice, the student writes down "7" every single time. But here's the catch: they also write down "2" or "12" way more often than they should, even though those are much rarer outcomes. They seem to forget the math they just proved they knew.

This is exactly what the paper "Implicit Probabilistic Reasoning Does Not Reflect Explicit Answers in Large Language Models" discovers about AI.

Here is the breakdown of the study using simple analogies:

1. The Two Ways of Testing (The "Quiz" vs. The "Story")

The researchers found that AI models (LLMs) have a split personality when it comes to probability.

Explicit Reasoning (The Quiz): This is the standard way we test AI. You give it a multiple-choice question (like the dice example above). The AI acts like a student taking a test. It picks the right answer, often with 99% confidence. It knows the rules.
Implicit Reasoning (The Story): This is the new test the researchers invented. Instead of asking a question, they ask the AI to continue a story. They give the AI a scenario (e.g., "Two dice are rolled...") and ask it to predict the next word.
- The Metaphor: Imagine the AI is a chef. In the "Quiz," the chef correctly identifies that a cake needs 2 eggs. But in the "Story," when actually baking the cake, the chef randomly throws in 10 eggs because they got distracted by the smell of the oven.

The Big Discovery: The AI is great at the Quiz, but terrible at the Story. When the AI is actually generating text (which is what it does most of the time in real life), it often ignores the math it claims to know.

2. The "Medical Diagnosis" Trap

The paper uses a medical example to show why this is dangerous.

The Scenario: You tell the AI that "Burnout" is common in a specific hospital wing (16%), but "Anxiety" is slightly less common (13%). You also tell it that the patient, Sam, works in that wing.
The Quiz: You ask, "What is the most likely condition for Sam?" The AI correctly calculates the numbers and says, "Burnout."
The Story: You start a sentence: "Based on the data, Sam is most likely suffering from..." and let the AI finish the sentence.
The Result: The AI finishes the sentence with "Anxiety."

Why? Even though the AI knew the math, when it was time to write the next word, it got distracted. It latched onto the word "Anxiety" because it felt "familiar" or "prominent" in its training data, ignoring the specific numbers you just gave it. It's like a doctor who knows the statistics but still guesses the wrong disease because they've heard that disease mentioned more often in movies.

3. The "Gambler's Fallacy" Glitch

The researchers also tested if AI could ignore irrelevant information.

The Setup: You tell the AI, "We are rolling a fair die. It's a fresh start."
The Twist: You add a sentence: "The previous roll was a 6." (Even though a new roll is independent and shouldn't care about the last one).
The Result: The AI starts thinking the next roll is less likely to be a 6, or more likely to be a 6. It gets confused by the "ghost" of the previous roll.

The Metaphor: Imagine a coin flipper who just flipped "Heads." If you ask them, "What's the chance of Heads next?" they say 50%. But if you whisper, "The last one was Heads," they suddenly start betting on "Tails," thinking the universe is "due" for a change. The AI does this too; it can't ignore the "noise" in the prompt.

4. The "First Option" Bias

Finally, the researchers found that the AI has a weird habit of liking the first thing it sees.

The Setup: You give the AI a list of options: A, B, C, D. You tell them they are all equally likely.
The Quiz: The AI picks the right answer.
The Story: When asked to generate the outcome, the AI picks "A" way too often, even if you shuffle the order to "C, B, A, D" and it still picks the first one it sees.

It's like a student who, when guessing on a test, always circles the first letter they see, regardless of the actual math.

Why Should We Care?

We usually think AI is smart because it gets high scores on tests (MCQs). But in the real world, we don't use AI to take tests; we use it to write stories, diagnose patients, and make decisions.

If an AI says, "I know the odds are 1 in 6," but then writes a story where the odds are 1 in 2, we are in trouble. It means the AI is "hallucinating" its own reality, ignoring the facts it just processed.

The Bottom Line:
Just because an AI can tell you the right answer doesn't mean it understands the probability well enough to act on it. The paper argues that we need to stop just asking AI "What is the answer?" and start watching "How does it write the story?" because that's where the real thinking (or lack thereof) happens.

1. Problem Statement

Current evaluations of Large Language Models (LLMs) regarding probabilistic reasoning rely heavily on Explicit Probabilistic Reasoning, typically measured via Multiple-Choice Questions (MCQs). In this paradigm, the model is prompted with a scenario and asked to select the correct probability from a finite set of options.

The authors argue that this approach is insufficient and potentially misleading for several reasons:

Bias Sensitivity: MCQ performance is highly sensitive to answer ordering and formatting.
Discriminative vs. Generative: Selecting an answer is a discriminative task, whereas LLMs are fundamentally generative models that predict the next token based on a probability distribution.
The "Illusion" of Competence: A model may correctly identify the answer to a probability question (explicit) but fail to integrate that same probabilistic logic into its text generation process (implicit). This discrepancy suggests that high MCQ scores may reflect data memorization or pattern matching rather than true probabilistic reasoning capabilities.

The core problem is that standard benchmarks do not evaluate whether an LLM's internal next-token probability distribution aligns with the ground truth probabilities of a scenario, which is critical for real-world applications involving uncertainty (e.g., medical diagnosis, risk assessment).

2. Methodology

The authors propose a new evaluation paradigm called Implicit Probabilistic Reasoning. Instead of asking the model to choose an answer, they evaluate the model's raw next-token probability distribution when completing a text scenario.

Core Approach

Reframing Tasks: Standard MCQs are rephrased as text-completion tasks.
- Explicit: "Two dice are rolled. What is the probability the sum is 7? [Options A, B, C, D]"
- Implicit: "Two dice are rolled. The sum of their faces is equal to "
Metric Extraction: The model is prompted to complete the sentence. The researchers extract the raw logits for the specific tokens representing the possible outcomes (e.g., tokens "2" through "12" for dice sums). These are converted into a probability distribution using a softmax function ( $P_{LLM}(\cdot | \text{context})$ ).
Comparison: The model's implicit distribution is compared against the known ground-truth distribution using metrics like Chebyshev distance, Manhattan distance, and Kullback-Leibler (KL) divergence.
Experimental Setup:
- Models: Evaluated 10 diverse open-weight models (ranging from 14B to 235B parameters), including Qwen, Llama, Mistral, DeepSeek, and Gemma variants.
- Scenarios: Five families of tasks were constructed:
  - Dice: Single/repeated rolls, dependent/independent events, partial observations.
  - Coins: Biased/unbiased flips, repeated events.
  - Preference/Choice: Impact of outcome labels (e.g., "Left" vs. "Right") and ordering.
  - Statistics: Medical diagnosis scenarios requiring Bayesian updates based on population prevalence.

3. Key Contributions

Novel Evaluation Paradigm: Introduced "Implicit Probabilistic Reasoning" as a complementary metric to MCQs, focusing on the model's generative probability distribution rather than its final selected answer.
Demonstration of Discrepancy: Provided empirical evidence that LLMs often possess high explicit reasoning capabilities (answering MCQs correctly) but exhibit poor implicit reasoning (generating text that contradicts the underlying probabilities).
Identification of Specific Biases: Uncovered three distinct failure modes in implicit reasoning:
1. Failure to favor the most likely outcome in text completion despite knowing the answer.
2. Inability to discard irrelevant prior information (independence bias).
3. Sensitivity to label ordering and semantic meaning of options.

4. Key Results

Result 1: Explicit Success vs. Implicit Failure

Models frequently answer MCQs correctly (high accuracy) but fail to assign the highest probability to the correct outcome during text generation.

Case Study (Medical): In a scenario calculating the probability of anxiety vs. burnout based on hospital ward statistics, the model correctly selected "11%" (Anxiety) in the MCQ setting (99.96% confidence). However, in the text-completion setting, it assigned a 99.08% probability mass to "anxiety" despite the ground truth showing "burnout" was actually more likely (~15% vs. ~11%). The model generated a self-contradictory justification, ignoring the statistics it had just used to answer the MCQ.
General Trend: Across Dice and Coins scenarios, when partial information or prior events were introduced, implicit accuracy dropped significantly while explicit MCQ accuracy remained high.

Result 2: Sensitivity to Independent Prior Events

Models fail to treat independent events as independent.

When a prompt included the result of a previous independent die roll (e.g., "The die landed on 1. The die is cast again..."), the model's prediction for the second roll was heavily biased toward repeating or avoiding the previous number.
This resulted in a distorted probability distribution that deviated significantly from the uniform ground truth, as measured by high Chebyshev distances. This indicates an inability to "forget" irrelevant context.

Result 3: Label and Position Bias

Outcome labels and their order significantly skew implicit predictions.

Ordering Bias: In scenarios with equiprobable outcomes (e.g., "A" vs. "B"), models disproportionately favored the option listed first in the prompt, even when explicitly told they were equally likely.
Semantic Bias: Models exhibited inherent preferences for specific labels (e.g., "Left" over "Right" or "Heads" over "Tails") that could not be explained solely by ordering. For instance, Mistral-24B preferred "Left" regardless of its position in the prompt.

5. Significance and Implications

Benchmark Limitations: The paper argues that current MCQ-based benchmarks provide an incomplete and potentially optimistic view of LLM capabilities. A model can "know" the answer without being able to "reason" with it during generation.
Real-World Risk: In high-stakes applications (e.g., medical diagnosis, financial forecasting), the implicit generation process is what users interact with. If a model's internal probability distribution is skewed (favoring rare diseases or unlikely outcomes), it can generate misleading, confident-sounding but factually incorrect text.
Need for New Metrics: The authors conclude that evaluating the next-token probability distribution is essential to detect these "blind spots." Future research must focus on improving the alignment between a model's internal beliefs (distributions) and its generated outputs, rather than just optimizing for correct multiple-choice selection.

In summary, the paper reveals a critical disconnect: LLMs can solve probability problems when asked, but they often fail to apply that logic when generating text. This suggests that current models rely on surface-level pattern matching for explicit tasks but lack robust, integrated probabilistic reasoning for generative tasks.