🤖 machine learning

Reliability Scales Inversely: Bigger Models Compound Mistakes Faster via a Hidden Auto-Regressive Risk Regime

This paper reveals that as language models scale, they increasingly enter a self-perpetuating "auto-regressive risk regime" where low-probability token errors trigger confident, snowballing hallucinations that persist longer than the model's own uncertainty signals can detect, ultimately causing larger models to compound mistakes faster despite gaining general capability.

Original authors: Kushal Chakrabarti

Published 2026-07-22

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Kushal Chakrabarti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Great AI Paradox: Why Bigger Isn't Always Better

Imagine you are teaching a robot to tell stories. In the world of artificial intelligence, there is a long-held belief that if you just give the robot more brainpower, more books to read, and more time to study, it will eventually become perfect. This idea is called "scaling": the bigger the model, the smarter it gets. But there's a catch. When these robots write long stories, they don't just get better; they sometimes get worse at staying truthful as the story goes on. They might start with a perfect fact, but then they accidentally invent a lie, and because they are so confident in that lie, they build the rest of the story on top of it, creating a snowball of nonsense.

To understand why this happens, we need to look at how these robots "think." They don't know facts like we do; they predict the next word in a sentence based on patterns they've seen before. Scientists have found that when a robot makes a mistake, it's not always because it doesn't know the answer. Sometimes, it knows the answer but guesses wrong anyway. Once it makes that wrong guess, it treats it as a fact for the next sentence. This is called "auto-regression": the robot feeds its own output back into itself. The big question researchers are asking is: Why does this mistake-making get worse as the robots get bigger, and why can't the robots even see that they are lying?

The Paper's Big Discovery: The Invisible Snowball

This paper, written by Kushal Chakrabarti, flips the script on how we think about AI reliability. The main finding is a bit counterintuitive: As AI models get bigger, they don't just get smarter; they also get better at making mistakes that compound faster.

Think of an AI model like a student taking a long, multi-part test.

The Old Idea: The "Knowledge Gap" theory says that if a student gets a question wrong, it's because they didn't study enough. The solution? Give them a bigger library (more data) and a bigger brain (more parameters).
The New Discovery: This paper argues that for very smart students (big models), the problem isn't that they don't know the facts. It's that once they make a tiny, confident guess that happens to be wrong, they get stuck in a "risk regime." They commit to that wrong guess, and because they are so confident, they don't realize they are wrong. They then use that wrong guess to answer the next question, which leads to another wrong guess, and so on. The bigger the student, the faster this snowball of lies grows.

The "Risk" You Can't Feel

To explain this, the authors use a clever math trick to split the AI's "uncertainty" into two parts:

Felt Uncertainty: This is how nervous the AI feels. If it's guessing, it feels jittery. If it's sure, it feels calm.
Commitment Risk: This is the hidden danger. It's the gap between what the AI thinks is right and what a "super-smart oracle" (a perfect reference model) knows is right.

Here is the scary part: The AI can only feel its own nervousness. It cannot feel the "Commitment Risk."

Imagine you are walking on a bridge.

Felt Uncertainty is how shaky your knees feel.
Commitment Risk is how far the bridge is actually from the ground.

When the AI makes a mistake (a "fabrication"), its "knees" (felt uncertainty) stop shaking almost immediately. It feels calm and confident again. But the bridge is still far from the ground! The "Commitment Risk" stays high for a long time. Because the AI feels calm, it doesn't know it's still in danger. It keeps walking confidently right off the edge, creating a chain of lies.

The "Snowball" Effect

The paper shows that this isn't just a one-time mistake. Once the AI starts lying, it makes it 1.08 to 1.71 times more likely to lie again in the very next sentence.

Small Models: They make mistakes, but they don't always chain them together perfectly.
Big Models: They are so good at predicting the next word that once they pick a wrong path, they lock onto it with super-confidence. They don't just make one mistake; they make a chain of mistakes.

The authors found that as models scale up (get bigger), the "knowledge gap" (how much they don't know) gets smaller by about 6 times. But the "knowledge degradation" (how fast they fall apart once they start lying) gets 11 to 39 times worse. In other words, bigger models are better at starting a story correctly, but they are terrible at finishing it without making up nonsense.

Why Can't the AI Catch Itself?

You might ask, "Why doesn't the AI check its own work?" The paper reveals a blind spot. Most current tools that try to detect AI lies look at how "nervous" the AI is (its felt uncertainty).

The Problem: When the AI starts a lie, it feels nervous for a split second, then relaxes. The detection tools see the relaxation and think, "Oh, it's calm, so it must be telling the truth!"
The Reality: The AI is calm, but it's still in the "danger zone" (high risk). The tools miss the lie because they are looking for nervousness, not the hidden risk. The paper shows that these detectors fire 30% less often on the risky branch where the lies happen, even though that branch holds nearly 4 times more fabrications than the initial mistake.

The "Magic Fix" (And What It Proves)

To prove that this "hidden risk" is the real villain, the authors ran a simulation. They took a model that was about to make a mistake and artificially forced it to be less "risky" without changing what it thought was true.

The Result: When they squeezed out that hidden risk, the number of lies dropped by 35% to 74% across different types of models.
What This Means: This proves that the problem isn't that the AI doesn't know the facts. It's that the AI gets stuck in a state where it's confident but wrong, and that state causes the errors to pile up.

The Takeaway

This paper suggests that simply making AI models bigger won't fix their lying problem. In fact, bigger models might be more prone to these "snowball" lies because they get confident too quickly. The solution isn't just more data; it's finding a way to detect that hidden "Commitment Risk" before the AI locks itself into a lie. Until we can see the bridge is far from the ground, even the smartest AI might confidently walk right off the edge.

Technical Summary: Reliability Scales Inversely

Problem Statement

The prevailing view of hallucination in Large Language Models (LLMs) treats it as a "knowledge gap"—a deficit of information that scaling, retrieval, or additional data can fill. This paper challenges that account, arguing that while scaling improves initial knowledge (reducing the "knowledge gap"), it simultaneously erodes reliability by amplifying a distinct failure mode: auto-regressive risk.

The core problem is that as models scale, they do not merely become more accurate; they become more prone to a specific type of compounding error. When a model commits to a low-probability, unsupported token (a fabrication), it conditions subsequent generations on that error. This process converts transient sampling variance into persistent bias. Crucially, this failure mode is structurally invisible to the model's own self-monitoring mechanisms, which rely on the model's internal probability distribution ( $p_M$ ) rather than an external oracle.

Methodology

The authors introduce a theoretical framework based on a disagreement variable ( $\delta$ ) defined as the log-likelihood ratio between the model's distribution ( $p_M$ ) and a stronger, same-family oracle ( $p_O$ ):
$\delta_t(x) = \log p_M(x | y_{<t}) - \log p_O(x | y_{<t})$

The second moment of this variable, $E[\delta^2]$ , is decomposed into two distinct components:

Bias ( $Bias^2$ ): The squared mean, $KL(p_M \| p_O)^2$ , representing the systematic gap between the model and the oracle.
Risk ( $Var[\delta]$ ): The variance of the disagreement, representing the spread of how the oracle weighs candidates the model considers.

The paper utilizes three primary experimental approaches across multiple model families (Qwen3, OLMo-3, Llama-3) and scales (0.6B to 32B):

Scaling Analysis: Measuring the "knowledge gap" (initial hallucination rate) versus "knowledge degradation" (the rate of hallucination increase over the length of a response) using FActScore and LongFact++ benchmarks.
Onset Dynamics: Analyzing the temporal behavior of $\delta$ , its moments, and the model's felt uncertainty ( $H(p_M)$ ) immediately following a fabrication onset.
Causal Intervention (In Silico Knockout): Implementing a "risk contraction" mechanism. At steps where the model diverges from the oracle, the model's next-token distribution is modified to reduce the variance of $\delta$ while strictly preserving the mean (KL divergence). This isolates the causal effect of risk from bias.

Key Contributions

1. The Decomposition of Error

The paper establishes that hallucination is not solely a function of the knowledge gap (bias). It identifies Risk ( $Var[\delta]$ ) as a dominant, distinct error term. While scaling reduces the bias (the model gets closer to the oracle on average), it fails to reduce the risk (the spread of disagreement), which actually becomes a larger share of the total error as models grow.

2. The Persistence Asymmetry

A critical finding is the divergence between "felt uncertainty" and "oracle-referenced risk" after a fabrication:

Felt Uncertainty ( $H(p_M)$ ): The model's internal sense of uncertainty spikes at the moment of fabrication but relaxes rapidly (within ~0.3 tokens) as the generation becomes fluent.
Oracle-Referenced Risk ( $Var[\delta]$ ): The risk associated with the committed error persists for significantly longer (3.5–5.0 tokens).
This creates a "confident-but-precarious" regime where the model feels certain but is statistically divergent from the truth, a state that self-monitoring detectors cannot detect because they only read $H(p_M)$ .

3. Auto-Regressive Conversion and Snowballing

The paper demonstrates that autoregression acts as a machine that converts this persistent risk into fixed bias. Once a model samples an unsupported token, the risk collapses into a fixed gap that is inherited by all subsequent tokens. This leads to error compounding: the relative risk of a subsequent fabrication increases monotonically with scale (from 1.08× to 1.71×), meaning larger models are more likely to chain hallucinations together.

4. Structural Blindness of Self-Monitoring

Detectors based solely on the model's distribution ( $p_M$ ), such as Semantic Entropy or SelfCheckGPT, are structurally blind to this risk regime. The paper shows that these detectors fire significantly less (approx. 30% lower) on the "bridge" tokens between fabrications, despite these tokens carrying nearly 4× more hallucinations than the onset tokens. The detectors miss the "precarious" branch because the model's internal uncertainty has already relaxed.

Key Results

Inverse Scaling of Reliability: Across model families, scaling reduces the initial knowledge gap by 5.7–7.0×, but increases knowledge degradation (the rate of error accumulation) by 11–39×. Capability and reliability are shown to be distinct scaling axes.
Risk Dominance: As models scale, the share of error attributed to risk (variance) increases from 31% to 49% of the total squared error. The tail of the disagreement distribution also becomes heavier (excess kurtosis rises from 8.9 to 33.0).
Causal Reduction of Hallucination: By applying the risk contraction intervention (reducing variance while holding bias constant), the authors causally reduced web-verified hallucinations by 35–74% across three model families. This confirms that the compounding quantity is variance, not knowledge.
Detector Failure: Semantic entropy detectors were found to be significantly less sensitive to the "bridge" regime (the state between fabrications) compared to the onset, with statistical significance ( $p < 10^{-16}$ ).

Significance and Claims

The paper claims to redefine the understanding of LLM hallucination from a static knowledge deficit to a dynamic, self-perpetuating risk regime. Its primary significance lies in:

Mechanistic Causality: Providing the first on-policy, causal evidence that contracting the variance of the model-oracle gap (without changing the mean) directly reduces hallucination, proving that "risk" is the driver of compounding errors.
The Limits of Scaling: Demonstrating that simply scaling models does not solve reliability; in fact, larger models may be more susceptible to specific types of compounding errors due to the sharpening of the risk tail.
The Blind Spot: Highlighting that current self-monitoring techniques are fundamentally limited because they cannot "see" the risk term ( $Var[\delta]$ ), which is the very mechanism driving downstream fabrications.

The authors conclude that reliability is a separate scaling axis that requires new interventions targeting the variance of the model's commitment, rather than just improving the mean knowledge or relying on self-reported uncertainty. The dominant failure mode is described as "invisible to the model itself," necessitating external or white-box signals for effective mitigation.