📊 statistics

When are likely answers right? On Sequence Probability and Correctness in LLMs

This paper investigates the relationship between sequence probability and correctness in large language models, finding that while higher probability often predicts correctness within a fixed dataset, it is not a reliable indicator for improving accuracy through decoding method changes or for distinguishing correct responses to the same prompt.

Original authors: Johannes Zenn, Jonas Geiping

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Johannes Zenn, Jonas Geiping

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a super-imaginative storyteller who has read almost every book in the library. When you give this storyteller a prompt (a story starter), they don't just spit out one ending; they can generate thousands of different possible endings.

The big question this paper asks is: "If a story ending feels very 'likely' or 'natural' to the storyteller, does that mean it's actually the correct answer?"

The authors treat the model's "probability" (how likely the model thinks a sentence is) as a confidence meter. They wanted to know if turning up the volume on this confidence meter actually leads to better answers.

Here is what they found, broken down into simple analogies:

1. The "Library" Analogy (Within a Dataset)

The Finding: If you look at a whole pile of answers generated for a specific type of question (like a math test), the ones the model thinks are most "likely" are usually the correct ones.

The Metaphor: Imagine a librarian who has organized a shelf of books. If you ask, "Which of these books is the right guide for fixing a car?", the librarian points to the book they are most confident is the right one. In a pile of different questions, the librarian is usually right.
The Catch: This only works if you are comparing different questions against each other. It doesn't mean the librarian is perfect at picking the right book for one specific question just because they feel confident.

2. The "Tuning Knob" Analogy (Changing Settings)

The Finding: If you try to make the model "more confident" by changing its settings (like turning a dial to make it less random), you don't necessarily get better answers. In fact, you often get worse answers that just sound more confident.

The Metaphor: Imagine a radio. You can turn the "signal strength" knob up to make the music sound louder and clearer. But if you turn it up too high, you might just get static that sounds very loud and clear, even though it's not the right song.
The Reality: The paper found that tweaking the model's settings to force it to pick the "most likely" sequences often makes the model sound more sure of itself, but it doesn't make it smarter. Sometimes, the "most likely" path is actually a trap.

3. The "Crowd Vote" Analogy (Same Question, Different Answers)

The Finding: If you ask the model the exact same question 32 times, the "most likely" answer isn't always the right one. The model might give you 32 different answers, and the one it thinks is most probable might be wrong.

The Metaphor: Imagine asking a room of 32 people the same riddle. You ask, "Who is the smartest person here?" The person who raises their hand the loudest (highest probability) isn't necessarily the one with the right answer. Sometimes the room is split, and the "loud" answer is just a popular guess, not the truth.
The Exception: The paper found one specific type of puzzle (Math problems) where the "loudest" voice was usually the smartest. But for other types of questions (like following instructions or medical advice), the loudest voice was often just as likely to be wrong as the quiet ones.

4. The "Self-Improvement" Trap

The Finding: Some people try to make AI smarter by having it pick its own "best" answers and learn from them. This paper warns that this is risky.

The Metaphor: Imagine a student trying to get better at math by only studying the answers they think are right. If the student isn't already pretty good at math, they will just keep reinforcing their own mistakes, thinking, "Oh, this wrong answer feels very familiar, so it must be right!"
The Lesson: You can only use "confidence" to improve the AI if the AI is already good at the task. If it's struggling, trusting its own confidence will just make it more confidently wrong.

Summary of the "Rules of Thumb"

The paper gives us three main takeaways for using these models:

Don't trust the "Volume": Just because the model says an answer is "highly probable" doesn't mean it's correct.
Context Matters: The relationship between "feeling sure" and "being right" changes depending on the type of question (Math is different from Code, which is different from Instructions).
Don't Over-Optimize: Trying to force the model to be "more likely" by changing its settings usually doesn't help and can hurt performance.

In short: The model's "gut feeling" (probability) is a useful clue when looking at a big picture, but it is a terrible compass for navigating a single, specific problem. You can't just turn up the "confidence dial" and expect the answers to get better.

Technical Summary: When are likely answers right? On Sequence Probability and Correctness in LLMs

Problem Statement

Large Language Models (LLMs) are autoregressive systems that assign probabilities to token sequences. Decoding methods (e.g., beam search, top-k sampling, Best-of-N) are often designed to shift probability mass toward outputs that are more likely under the model, either locally at the token level or globally at the sequence level. A fundamental assumption driving these methods is that higher sequence probability correlates with correctness. However, empirical observations suggest this relationship is not universal; for instance, beam search, which approximates the most likely sequence, often degrades performance compared to sampling methods.

This paper addresses the core question: When does sequence probability (the conditional probability of a continuation given a prompt) actually align with correctness? The authors aim to quantify this relationship across different decoding methods, models, and benchmarks to determine when maximizing probability leads to better answers and when it does not.

Methodology

The authors conduct a large-scale empirical study quantifying the correlation between sequence log-probability and correctness across four distinct levels of granularity:

Across-Method Correlation: Comparing different decoding methods (and their optimal hyperparameters) on a fixed model and dataset.
Within-Method Correlation: Analyzing how changing hyperparameters within a single decoding method affects both probability and correctness.
Within-Dataset Correlation: Examining the relationship between log-probability and correctness across all prompt-answer pairs within a specific dataset, model, and method.
Within-Sample Correlation: Investigating the correlation among multiple responses generated for the same prompt.

Experimental Setup:

Models: 14 models from the Qwen2.5, Qwen3, and Olmo3 families (ranging from 0.6B to 8B parameters), including both base and post-trained variants.
Datasets: 6 benchmark datasets covering mathematical reasoning (MATH500), code generation (HumanEval), instruction following (IFEval), and general knowledge/medical reasoning (GPQA, MedQA, MMLU).
Decoding Methods: 8 methods were evaluated:
- Local methods: Low-temperature sampling (LTS), Top-k, Top-p, $\epsilon$ -sampling.
- Global methods: Beam search, Best-of-N (BoN), Scalable Power Sampling (SPS), and Power-SMC.
Metrics: The study utilizes Pearson correlation coefficients (on binned data) and Spearman rank correlation coefficients to measure the relationship between log-probability and binary correctness labels.

Key Contributions and Findings

1. Framing Decoding as Probability Maximization

The paper formally frames decoding methods as approaches to maximize sequence probability. It distinguishes between local methods (which modify the next-token distribution at each step, often failing to maximize global sequence probability due to local normalization constants) and global methods (which target the mode or power distribution of the full sequence). The authors provide a variational objective showing that the power distribution acts as a soft maximizer of sequence probability, trading off expected log-probability and entropy.

2. Granularity-Dependent Correlation

The central finding is that the relationship between probability and correctness is not uniform; it depends heavily on the level of analysis:

Within-Dataset (Finding 1): There is a consistent, often positive correlation between log-probability and correctness within a fixed dataset, model, and method. This correlation is largely determined by the dataset and model variant rather than the decoding method. For example, MATH500 shows a near-perfect positive correlation, while IFEval often shows negative correlation for base models. Post-trained models generally show positive correlations, whereas base models are more diverse.
Within-Method (Finding 2): Tuning the hyperparameters of a specific decoding method to increase sequence probability does not reliably improve correctness. While changing hyperparameters (e.g., temperature, beam width) often yields sequences with higher log-probability, these sequences are frequently less correct. There is no consistent way to tune a method to simultaneously maximize both probability and accuracy across datasets.
Across-Methods (Finding 3): No single decoding method consistently outperforms others in terms of correctness, even when choosing optimal hyperparameters. While many methods produce sequences that are both more probable and more correct than a low-temperature baseline (LTS), this is not a reliable rule. Specifically, increasing sequence probability by switching methods (e.g., to Power Sampling) does not guarantee higher accuracy.
Within-Sample (Finding 5): For a single prompt, the correlation between log-probability and correctness among repeated samples is generally weak and symmetrically distributed around zero. However, prompts where the model generates a higher fraction of correct answers tend to exhibit stronger positive within-sample correlations.

3. Implications for Self-Improvement and Self-Consistency

The paper connects these findings to verifier-free self-improvement and self-consistency (SC):

Self-Consistency (Finding 6): For low-temperature sampling, majority voting (uniform weighting) often outperforms probability-weighted voting. This is attributed to the weak or symmetric within-sample correlations, meaning high probability does not reliably indicate the correct answer for a specific prompt.
Self-Improvement (Finding 4): Probability-based recursive self-improvement loops may require a sufficient baseline accuracy to be effective. The correlation between probability and correctness scales with the model's overall accuracy; if the model is not already accurate, reinforcing high-probability samples may reinforce incorrect ones.

Significance and Claims

The paper claims to clarify the conditions under which decoding can be expected to improve correctness. Its primary contribution is a nuanced understanding that sequence probability is a useful signal for distinguishing correct from incorrect answers across a dataset, but it is a poor signal for selecting hyperparameters or decoding methods to improve performance on a specific task.

The authors emphasize that the intuition "higher probability equals higher correctness" holds at the dataset level (allowing models to distinguish good from bad answers in aggregate) but fails at the decision-making level (selecting the best answer for a specific prompt or tuning a method). Consequently, the paper advises against relying solely on sequence probability for:

Selecting the optimal decoding method or hyperparameters without empirical validation.
Probability-weighted aggregation in self-consistency (unless the model has very high accuracy).
Verifier-free self-improvement loops without ensuring a sufficient baseline of task accuracy.

The work suggests that future decoding strategies should focus on the granularity of the correlation: leveraging probability signals for dataset-level filtering while acknowledging their limitations for instance-level selection.