📊 statistics

Perplexity Cannot Always Tell Right from Wrong

This paper rigorously demonstrates that perplexity is an unsuitable metric for model selection because it can favor models that are highly confident yet inaccurate, proving that a model's increased confidence must be matched by commensurate accuracy to yield a lower perplexity score.

Original authors: Petar Veličković, Federico Barbero, Christos Perivolaropoulos, Simon Osindero, Razvan Pascanu

Published 2026-02-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Petar Veličković, Federico Barbero, Christos Perivolaropoulos, Simon Osindero, Razvan Pascanu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Confident Fool" Problem

Imagine you are hiring a tour guide for a city you've never visited. You have two candidates:

Guide A is very accurate but sometimes hesitates, saying, "I think it's this way, but I'm not 100% sure."
Guide B is often wrong, but they are incredibly confident. They point in the wrong direction and say, "I am absolutely certain this is the way!"

In the world of AI, we use a metric called Perplexity to decide which guide is better. Think of Perplexity as a "Surprise Meter." It measures how shocked the AI is when it sees the correct answer.

If the AI guesses correctly and is confident, the Surprise Meter is low (Good!).
If the AI guesses correctly but is unsure, the Surprise Meter is a bit higher.
If the AI guesses wrong, the Surprise Meter goes up.

The paper's main claim is this: The Surprise Meter (Perplexity) is broken. It can trick you into picking Guide B (the confident fool) over Guide A (the accurate but hesitant one). In fact, the paper proves that a model can be so confident in its wrong answers that the "Surprise Meter" actually reads lower than if it were being accurate but unsure.

Analogy 1: The Copycat and the "Smoothie" Effect

To prove this, the authors used a simple game: The Copycat Task.
Imagine you show a robot a long string of numbers, like 010101, and ask it to copy it.

The Setup: The robot learns to copy the string 010101 perfectly. It becomes a master copycat.
The Trap: Because the robot is so good at copying that specific pattern, it becomes "overconfident." It starts to believe that any long string of numbers must be 010101.
The Mistake: If you give it a slightly different string, like 010100, the robot still confidently outputs 010101. It is wrong, but it is very confident.

The "Smoothie" Problem:
Perplexity works like a smoothie. It takes the "surprise" of every single mistake and blends them all together into one average number.

If the robot makes one tiny mistake in a 1,000-character string, that one mistake gets diluted in the "smoothie."
The paper proves mathematically that if the robot is confident enough, the "smoothie" (the average surprise) can become so low that the robot looks perfect, even though it failed to copy the specific string correctly.

The Result: You end up selecting a robot that fails the test, simply because its "average surprise" score looks better than a robot that is actually getting the answers right but is a little more cautious.

Analogy 2: The "Confidence vs. Accuracy" Balance Scale

The authors also looked at a graph they call an "Iso-perplexity Curve." Imagine a balance scale.

On one side is Accuracy (getting the right answer).
On the other side is Confidence (how sure you are).

Usually, you want both. But the paper shows that Perplexity has a weird bias:

If you increase your confidence, the "Surprise Meter" drops very fast.
If you increase your accuracy, the "Surprise Meter" drops slowly.

The "Unfair Zone":
There is a zone on this scale where a model can become more confident but less accurate, and the "Surprise Meter" still goes down!

Imagine a student who stops studying (accuracy drops) but starts shouting answers with extreme confidence.
The teacher (the Perplexity metric) sees the low "surprise" and thinks, "Wow, this student is great!"
Meanwhile, a student who studies hard and gets 90% of the answers right but whispers them in uncertainty gets a "worse" score.

The paper calls this the "Unjustified Free Lunch": You can get a better score just by being louder and more confident, even if you are wrong more often.

The Real-World Test: The "Parity" Game

To see if this happens in real life, the authors trained an AI on a math game called Parity.

The Game: Look at a string of 0s and 1s and count how many 1s there are. If it's an even number, say "0". If it's odd, say "1".
The Twist: They trained the AI on short strings (easy) but tested it on very long strings (hard).

What happened?

On Easy Strings: The AI got better and better. As it got smarter, it also got more confident. The "Surprise Meter" went down, and the accuracy went up. Everything looked good.
On Hard Strings (The Trap): As the AI got more confident, it started making mistakes on the long strings. However, because it was so confident in its wrong answers, the "Surprise Meter" didn't go up.
The Conclusion: The AI that was actually best at solving the hard problem had a worse Perplexity score than a worse AI. The metric failed to pick the winner.

The Takeaway

The paper concludes that Perplexity is not a reliable judge.

It cannot always tell right from wrong.
It favors models that are confident over models that are accurate.
This is especially dangerous when the AI faces new, difficult situations (like the long strings in the experiment).

In short: If you use Perplexity to pick the best AI, you might accidentally pick the one that is the most sure of itself, even if it's the one getting the most things wrong. The paper warns us not to blindly trust this metric, especially when we can't easily check if the AI is actually right.

Technical Summary: Perplexity Cannot Always Tell Right from Wrong

Problem Statement

Perplexity is the standard metric for evaluating autoregressive language models, serving as both a loss function and a proxy for model quality. It measures a model's "surprise" when encountering ground-truth data. While widely used due to its computational simplicity, empirical observations have suggested that perplexity does not always correlate with sample quality or accuracy. Specifically, a model with lower accuracy but better-calibrated (or over-) confidence may achieve a lower perplexity score than a more accurate but hesitant model.

This paper addresses the theoretical underpinnings of this disconnect. It investigates whether perplexity is a reliable objective for model selection, particularly in the context of decoder-only Transformers. The core problem is that perplexity conflates confidence with correctness, potentially rewarding models that are confidently wrong, especially in long-context scenarios where error signals might be diluted.

Methodology

The authors employ a combination of rigorous theoretical proofs, analytical modeling, and empirical validation to demonstrate the limitations of perplexity.

Theoretical Framework (Continuity and Collapse):
- The paper leverages recent theoretical results regarding the continuity of Transformer representations (Pasten et al., 2025).
- It focuses on decoder-only Transformers with Compact Position Embeddings (CPE), a class including common embeddings like RoPE.
- The authors analyze the bitstring copy task, where a model must reproduce a sequence of bits exactly. This task is chosen because it has a clear ground truth and is known to be difficult for LLMs to learn robustly.
- They utilize greedy decoding (deterministic sampling) as a baseline to establish robust claims about accuracy and perplexity, later extending the analysis to stochastic sampling via temperature scaling.
Analytical Modeling (Iso-Perplexity Curves):
- The authors construct a simplified binary classification framework to derive iso-perplexity curves.
- They model the relationship between accuracy ( $a$ ) and confidence ( $1-\gamma$ ) as scalar variables.
- By solving for the "critical accuracy" required to justify an increase in confidence without increasing perplexity, they map out regions in the confidence/accuracy space where perplexity fails to select the more accurate model.
Empirical Validation:
- Copy Task Experiments: The authors trained a custom CPE Transformer on the bitstring copy task and evaluated a pre-trained Gemma 3 4B model. They compared the model's performance on a correct sequence ( $\alpha_N$ ) versus a perturbed sequence ( $\beta_N$ ) that the model fails to copy.
- Parity Task Experiments: To study model selection in a more complex setting, they trained a Transformer on the parity (XOR) task. They tracked checkpoints throughout training, evaluating them on both in-distribution (IID) and out-of-distribution (OOD) data. They analyzed the correlation between log-perplexity, micro-F1 score, and averaged Shannon entropy (as a proxy for confidence).

Key Contributions

Proof of Perplexity Failure via Continuity:
The authors prove that for a compact decoder-only Transformer, if the model predicts a sufficiently long sequence accurately and confidently, there must exist another sequence where the model fails, yet the log-perplexity of that failure approaches zero as the sequence length increases. This implies that high confidence on correct inputs necessitates the existence of "confounding" inputs where the model is confidently wrong, but the error is invisible to the perplexity metric due to averaging.
Vanishing Gradients on Incorrect Samples:
The paper demonstrates that as a model achieves perfect loss on a correct sequence, the gradient signal for the corresponding "confounding" incorrect sequence vanishes. Consequently, the training signal for these incorrect samples disappears, making it impossible for the model to jointly learn to correct them while maintaining high confidence on the correct ones.
Analytical Iso-Perplexity Curves:
The authors derive analytical expressions for iso-perplexity curves in the confidence-accuracy space. They identify "unfavourable regions" where:
- A model can become more confident without improving accuracy, yet still lower its perplexity (an "unjustified free lunch").
- Conversely, a model may improve its accuracy but fail to lower its perplexity if the confidence increase is not sufficiently matched by the accuracy gain.
- Specifically, for a perfectly confident model to be selected by perplexity, it must be perfectly accurate; otherwise, it will be rejected.
Empirical Evidence of Distribution Shift Sensitivity:
Through the parity task experiments, the authors show that while perplexity correlates well with accuracy on in-distribution data, this relationship breaks down on out-of-distribution (OOD) data. In OOD regimes, models with high confidence (low entropy) often achieve the best perplexity scores despite having lower accuracy, effectively landing on the "wrong side" of the iso-perplexity curve.

Results

Theoretical Limits: The proofs confirm that the continuity of Transformer representations creates a scenario where a model can be highly confident on a correct sequence $\alpha_N$ while failing on a nearby sequence $\beta_N$ , with the perplexity of $\beta_N$ converging to zero as $N \to \infty$ .
Empirical Trends:
- In copy task experiments, the probability of correctly copying the perturbed sequence $\beta_N$ collapsed to near zero, while the log-perplexity between the correct and incorrect sequences converged, validating the theoretical prediction.
- In the parity task, the Pearson correlation between log-perplexity and accuracy was strong and negative ( $r \approx -0.94$ ) for in-distribution data but became positive for out-of-distribution data.
- Checkpoints with the highest OOD accuracy often had the worst OOD perplexity, and these checkpoints were characterized by low entropy (high confidence).

Significance and Claims

The paper argues that perplexity is not merely an aggregation artifact (as suggested by some prior work focusing on long-context averaging) but is inherently skewed due to the mathematical relationship between confidence and the log-likelihood objective.

Model Selection Risk: The authors claim that relying solely on perplexity for model selection can lead to the selection of models that are "confident but wrong," particularly in regimes involving distribution shifts or long contexts.
Training Dynamics: The work suggests that the drive to minimize perplexity encourages models to trade accuracy for confidence, creating "unfavourable regions" where a model becomes more confident without becoming more accurate, yet still improves its metric score.
Cautionary Note: The paper does not propose a new metric to replace perplexity. Instead, it serves as a diagnostic warning. It suggests that practitioners should exercise appropriate care when using perplexity as a proxy for model quality, especially when accuracy cannot be directly measured, and highlights the need to consider model confidence and distribution shifts in model selection protocols.

In summary, the paper provides a rigorous theoretical and empirical foundation showing that perplexity cannot always distinguish between right and wrong, as it can be minimized by models that are confidently incorrect, thereby failing as a reliable selector for generalization performance.