💬 NLP

Beyond Exponential Decay: Rethinking Error Accumulation in Large Language Models

This paper challenges the prevailing assumption of exponential reliability decay in large language models by demonstrating that errors are concentrated at sparse "key tokens," proposing a new framework that prioritizes strategic preservation and dynamic computation at critical decision points to achieve sustained long-context coherence without proportionate scaling.

Original authors: Mikhail L. Arbuzov, Sisong Bei, Ziwei Dong, Dmitri Kalaev, Alexey A. Shvets

Published 2026-05-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mikhail L. Arbuzov, Sisong Bei, Ziwei Dong, Dmitri Kalaev, Alexey A. Shvets

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Old Story: The "Broken Chain" Theory

For a long time, experts believed that Large Language Models (LLMs) were doomed to fail as they wrote longer and longer texts. The logic was simple and scary:

Imagine you are building a chain out of paperclips. If there is even a tiny chance (say, 1%) that any single paperclip is weak, then as you add more and more clips, the whole chain becomes weaker and weaker. By the time you have 100 clips, the chain is almost guaranteed to snap.

In the world of AI, this meant that if a model made a small mistake on one word, that mistake would compound, making the next word more likely to be wrong, and the next even worse. The theory predicted that long stories or essays would eventually turn into nonsense because the errors would multiply exponentially.

The New Discovery: It's Not a Chain, It's a Road Trip

This paper argues that the "broken chain" theory is wrong. The authors say LLMs don't fail uniformly; they fail in a very specific, structured way. They propose a new model based on three main ideas:

1. Not All Words Are Created Equal (The "Steering Wheel" Analogy)

The old theory assumed every word in a sentence is equally important. The authors say this is false.

The Reality: In a long text, only a tiny fraction of words (about 5% to 10%) are the "critical" ones. These are the Key Tokens. They are like the steering wheel of a car, the traffic lights, or the major intersections on a road trip. If you mess these up, you go off the road.
The Rest: The other 90% of words are just "filler" or "scenery." They are like the trees passing by or the color of the road. They follow local rules (grammar, common phrases) and are actually easier to predict the more context you have.
The Result: You don't need a perfect car to drive 1,000 miles; you just need to make sure you don't crash at the few critical intersections. The model gets better at predicting the "scenery" words as it goes along, so the error rate for those drops to near zero.

2. The Model Lives in "Semantic Neighborhoods" (The "Mall" Analogy)

The paper suggests that the model's internal brain (its "embeddings") isn't a flat, messy space. It's organized like a giant shopping mall with distinct, low-dimensional "neighborhoods."

The Reality: Once the model decides to talk about "cooking," it enters the "Cooking Neighborhood." Even if it makes a small slip-up (like saying "salt" instead of "sugar"), it's still inside the Cooking Neighborhood. It hasn't left the building.
The Danger: The only time the model truly fails is if it makes a "Key Token" mistake that kicks it out of the Cooking Neighborhood and into the "Car Repair" neighborhood.
The Result: Small errors don't break the whole story because the model stays in the right "neighborhood." It's like walking through a mall; if you take a wrong turn down an aisle, you can still find your way back to the main hall. You don't fall off the edge of the world.

3. Mistakes Are Random, Not Systematic (The "Guessing Game" Analogy)

When the model does make a big mistake on a critical word, it's usually a unique, random fluke, not a consistent flaw.

The Reality: If you ask the model to solve a math problem 10 times, it might get the right answer 8 times. The 2 times it gets it wrong, it fails in different ways (one time it adds wrong, another time it subtracts wrong).
The Fix: Because the errors are random and scattered, if you take the "majority vote" (ask it 10 times and pick the most common answer), the random mistakes cancel each other out, and the correct answer rises to the top.
The Result: This is why techniques like "Self-Consistency" (asking the model to think again) work so well. They aren't fixing a broken engine; they are just filtering out the random noise.

The Big Picture: A New Formula

The authors combine these ideas into a new formula for how reliable a model is.

Old Formula: Reliability = (1 - Error) ^ (Total Words). This predicts a steep drop-off.
New Formula: Reliability = (1 - Error on Critical Words) ^ (Number of Critical Words) × (1 - Tiny Error on Normal Words) ^ (Rest of the Words).

Because the number of "Critical Words" (Key Tokens) doesn't grow as fast as the total text length (it might even stop growing after a certain point), the model doesn't crash. It stays reliable.

What This Means for Current Tech

The paper explains that recent "miracle" technologies aren't magic; they are just natural consequences of this structure:

Compression: We can throw away 99% of the text and keep only the "Key Words" (the steering wheel) without losing the meaning.
Long Context: Models can handle huge amounts of text because they only need to pay close attention to the few critical decision points, not every single word.
Self-Correction: When models fix their own mistakes, it's because they are staying within the right "neighborhood" and just correcting a random slip-up.

Summary

The pessimistic view said: "If you make one mistake, the whole long text is ruined."
This paper says: "No. You only make a few critical mistakes. The rest of the text is easy to predict, and the model stays on the right path. As long as you get the few 'steering wheel' moments right, the whole journey is safe."

Technical Summary: Beyond Exponential Decay

Problem Statement
A prevailing pessimistic view in autoregressive language modeling posits that correctness decays exponentially with output length. This argument assumes that if each token has an independent error probability $e$ , the probability of a sequence of length $n$ being correct follows $(1-e)^n$ , which approaches zero as $n$ grows. This model suggests a hard ceiling on the reliability of long-context generation. However, this "independent-error" hypothesis fails to account for empirical observations where models maintain coherence over long outputs, exhibit mid-generation self-correction, and show that attention weights are heavily concentrated on a small subset of the context.

Methodology and Theoretical Framework
The paper proposes a two-rate error model that rejects the assumption of uniform per-token error. Instead, it partitions tokens into two distinct categories:

Key Tokens ( $k$ ): A small subset (empirically estimated at 5%–10%) that carries long-range dependencies, global coherence, factual claims, and logical transitions. These tokens have a higher error rate ( $e_{key}$ ).
Non-Key Tokens ( $n-k$ ): The majority of tokens, governed by local syntax, frequent collocations, and accumulating context. Their error rate ( $e_{non}$ ) is significantly lower and approaches zero as context accumulates.

The model posits that the probability of a correct sequence is approximated by:
$P(\text{correct}) \approx (1 - e_{key})^k \cdot (1 - e_{non})^{n-k}$

Crucially, the number of key tokens $k$ scales sublinearly with sequence length $n$ (logarithmically, as a fractional power, or saturating at a task-specific maximum $k_{max}$ ). Furthermore, the model incorporates stratified manifold dynamics, suggesting that embeddings organize into low-dimensional semantic manifolds. Minor errors (non-key) often remain within the correct manifold, while only errors on key tokens cause a trajectory jump to a different manifold. Finally, the framework accounts for ensemble convergence, noting that reasoning errors are often idiosyncratic (varying across samples) rather than systematic, allowing majority-vote methods to recover accuracy.

Key Contributions
The paper makes three primary contributions:

A Refined Error Model: It introduces the two-rate model (Eq. 1), explicitly distinguishing between key and non-key tokens and modeling the dependence of reliability on the sublinear scaling of $k(n)$ .
Synthesis of Empirical Evidence: It unifies three distinct lines of recent research to support the model's components:
- Attention Sparsity: Evidence that only ~9% of tokens require long-range context (Fang et al., 2024) and that attention concentrates on a tiny fraction of the sequence (Liu et al., 2024).
- Embedding Geometry: Findings that LLM embeddings form stratified low-dimensional manifolds where perturbations in "noise" dimensions do not alter the semantic trajectory (Li and Sarwate, 2025; Robinson et al., 2025).
- Ensemble Convergence: Data showing that self-consistency and tree-search methods improve accuracy because correct reasoning paths converge on a manifold while incorrect paths diverge (Wang et al., 2023).
Reframing System-Level Results: It interprets recent engineering successes—such as anchor compression, retrieval-augmented attention, and self-consistency—not as independent wins, but as natural consequences of the structural fact that long-context reliability hinges on a handful of decision points.

Results and Predicted Decay Regimes
Based on the behavior of $k$ and $e_{non}$ , the paper predicts three decay regimes that are significantly gentler than exponential decay:

Logarithmic Growth ( $k \sim \log n$ ): Results in polynomial decay ( $n^{-c}$ ).
Fractional-Power Growth ( $k \sim \sqrt{n}$ ): Results in stretched-exponential decay.
Saturation ( $k \to k_{max}$ ): Reliability becomes constant with respect to $n$ once the critical number of key facts is committed.

The paper highlights that the third regime explains why systems can achieve 99% context reduction with minimal accuracy loss (Pang et al., 2024) and why 128K-token contexts are feasible on consumer GPUs (Liu et al., 2024).

Significance and Implications
The paper argues that the independent-error model is fundamentally flawed because it averages over a heterogeneous population of tokens and ignores error correlation. The significance of this work lies in shifting the focus from "uniform per-token accuracy" to "identifying and securing key decision points."

Practically, this framework suggests:

Sparse Attention: Dense attention is unnecessary; compute should focus on the sparse signal of key tokens.
Targeted Compute: Resources should be allocated to "manifold transitions" (high-uncertainty points) rather than uniformly distributed.
Strategic Ensembles: Majority voting is effective for reasoning tasks (where errors are idiosyncratic) but not for knowledge retrieval (where errors are systematic).
Evaluation: Standard perplexity is a poor predictor of downstream performance; metrics restricted to key tokens (e.g., LongPPL) correlate much more strongly with task success.

Limitations
The authors note that the framework is a synthesis of existing evidence rather than a derivation from first principles. The parameters $k$ , $e_{key}$ , and $e_{non}$ have not yet been jointly measured on a single benchmark. Additionally, the distinction between idiosyncratic and systematic errors currently relies on post-hoc observation rather than a quantitative pre-hoc criterion.

Conclusion
The paper concludes that modern LLMs maintain coherence over long outputs not because engineering has overcome an exponential decay barrier, but because the barrier itself was a mischaracterization of the problem. Reliability depends on a bounded or sublinearly growing number of key decisions, not on the uniform accuracy of every token.