💬 NLP

Information-Theoretic Limits of Reliability and Scaling in Language Models

This paper challenges the assumption that perfect reliability is achievable through scaling alone by establishing an information-theoretic framework that defines inherent reliability ceilings based on task ambiguity and inter-token dependencies, thereby deriving a unified scaling law that identifies the bottleneck between training data and model capacity while explaining phenomena like retrieval augmentation and catastrophic forgetting.

Original authors: Subhabrata Majumdar

Published 2026-07-17

📖 9 min read🧠 Deep dive

CC BY 4.0

Original authors: Subhabrata Majumdar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to tell a story, solve a math problem, or write a poem. You might think that if you just give the robot more brainpower (more computer chips) and feed it more books (more data), it will eventually become perfect at everything. This idea has been the driving force behind the rapid growth of Artificial Intelligence. But what if there's a hidden ceiling? What if some tasks are just impossible to get 100% right, no matter how smart the robot gets? This paper dives into that question using a branch of science called information theory. Think of information theory as the study of how much "surprise" or "uncertainty" is packed into a message. If you ask a robot "What is 2+2?", there is zero surprise; the answer is always 4. But if you ask "Write a poem about a sad cloud," there is huge surprise because there is no single right answer. The authors are asking: How much of that surprise can a robot actually solve, and where does it hit a wall?

The paper argues that the popular belief—that bigger models will eventually achieve perfect reliability on any task—is mathematically wrong. The authors show that every task has a "reliability ceiling," a hard limit on how accurate a model can ever be. This limit isn't about how big the model is; it's about the nature of the task itself. For a math problem, the ceiling is the sky (100% accuracy is possible). For creative writing, the ceiling is much lower because the "correct" answer depends on subjective feelings and cultural context that no robot can truly know. The paper proves that you can't scale your way past this ceiling.

Furthermore, the authors discovered that the way these models generate text—word by word, like a chain reaction—makes things worse. If the model makes a small mistake early on, that error can snowball, making the rest of the story or code go off the rails. They found that some tasks, like coding, are like a sturdy brick wall where one loose brick doesn't bring the whole thing down. Other tasks, like writing a poem, are like a house of cards; one wrong move early on can collapse the entire structure.

Finally, the paper offers a new rule for how to grow these models. Instead of just throwing more data or more computer power at a problem, the authors suggest that performance is bottlenecked by whichever resource is scarcer. If you have tons of data but a tiny brain, adding more data won't help. If you have a giant brain but no data, adding more brainpower won't help. You need to balance them perfectly. This new rule explains why some tasks get better with scaling while others hit a plateau, and it provides a mathematical map for when adding more resources actually works and when it's just a waste of time.

The Invisible Ceiling

Imagine you are playing a game of "Guess the Next Word." Sometimes, the game is easy. If the sentence is "The sun rises in the...", the next word is almost certainly "East." There is no mystery. But other times, the game is a guessing game with a twist. If the sentence is "The cat sat on the...", the next word could be "mat," "rug," "sofa," or "floor." All are correct, but they feel different.

The authors of this paper call the "East" scenario a fully verifiable task. In these cases, like solving a math equation or writing code that must run without errors, the answer is determined entirely by the question. There is no hidden information. The "reliability ceiling" for these tasks is 100%. If you have a perfect model, you get a perfect score.

But then there are unverifiable tasks, like creative writing or giving life advice. Here, the "correct" answer depends on things the model can't see, like the writer's mood, the reader's taste, or the cultural moment. The authors call this missing information "latent context." Because this context is hidden and subjective, no amount of data or computer power can ever make the model 100% reliable. The ceiling is lower, and it's permanent. You can't scale your way to perfection on a task where perfection is a matter of opinion.

The paper breaks this down into two types of gaps. The first is the resolvable gap. This is missing information that could be provided. For example, if a model is writing a story about a specific person but doesn't know their name, that's a resolvable gap. If you give the model the name (by adding it to the input), the gap closes, and the model gets better. The second is the subjective gap. This is missing information that cannot be provided because it doesn't have a fixed value. If the task is to write a "funny" joke, what is funny to one person might be boring to another. No amount of extra data can fix this. The paper proves that for these tasks, the model will always have a permanent floor of unreliability.

The Domino Effect

Now, imagine the model is building a tower, one block at a time. It places the first block, then the second, then the third. This is how Large Language Models work; they generate text token by token. The problem is that if the model places the first block slightly crooked, the second block has to be placed on top of a crooked foundation.

The authors call this autoregressive degradation. It's like a game of "Telephone" where the message gets distorted with every whisper. If the model makes a tiny mistake early in a sentence, that mistake changes the context for the next word, which changes the context for the word after that. The errors compound.

However, not all towers are built the same way. The paper introduces a concept called the dependency kernel, which is a fancy way of describing how much each word depends on the words before it.

Banded Dependencies (The Brick Wall): Think of coding or math. In a line of code, the next word usually depends on the one immediately before it (like a closing bracket matching an opening one). If you make a mistake, it's usually contained to that small area. The rest of the code can still make sense. The "dependency kernel" here is narrow, like a brick wall. A mistake in one brick doesn't bring down the whole wall.
Dense Dependencies (The House of Cards): Think of poetry or creative writing. In a poem, the word you choose at the very beginning might determine the rhyme scheme for the entire poem. If you pick the wrong word early on, the whole poem might lose its rhythm or meaning. The "dependency kernel" here is dense, like a house of cards. A mistake in the first card can ruin the whole structure.

The paper shows that for tasks with dense dependencies, the model's performance degrades much faster as the text gets longer. A single early error can cascade into a total failure. This explains why models are great at short, logical tasks but struggle with long, creative ones.

The Balancing Act

Finally, the paper tackles the question of scaling: "If we just make the model bigger and give it more data, will it get better?"

The answer is: It depends on what you have less of.

The authors derive a new "scaling law" that looks like a simple math equation. It says that the model's performance is limited by the scarcer resource.

If you have a massive amount of data but a tiny model, adding more data won't help much because the model is too small to learn it all.
If you have a massive model but very little data, adding more parameters won't help because the model has nothing to learn from.

The sweet spot, where you get the most improvement, is when you balance the two. This is similar to the "Chinchilla" law that researchers already knew, but this paper explains why it works and adds a crucial new detail: the max function. It means you don't get extra credit for having too much of one thing. If you have 100 times more data than your model can handle, that extra data is useless. You have to match your data to your model size.

What This Means for the Future

This paper doesn't just tell us how to build better models; it tells us what we can't do. It suggests that the idea of a "general AI" that is perfect at everything is a myth. There are tasks where the ceiling is low, and no amount of scaling will ever get us to 100% accuracy.

It also explains why certain tricks work. For example, Retrieval-Augmented Generation (RAG)—where the model looks up facts in a database before answering—works because it fills in the "resolvable gaps." It gives the model the missing context it needs to reach a higher ceiling. But for subjective tasks, even RAG can't fix the "subjective gap."

The paper also warns us about catastrophic forgetting. When you train a model on a new task, it might "forget" an old one. The authors explain this as a reallocation of resources. The model has a limited "budget" of brainpower. If you force it to focus on a new task that has a very different structure (a different dependency kernel), it has to drop the old structures to make room. It's like trying to learn a new language; if you focus too hard on French, you might start forgetting the grammar rules of Spanish.

In short, the paper tells us that AI is not a magic wand that will solve every problem. It is a tool with specific limits. Some problems are solvable with enough data and compute; others are fundamentally ambiguous. The key to the future isn't just "bigger," but "smarter" about what we ask the models to do and how we balance our resources. We need to stop expecting perfection where it's impossible and start engineering solutions that respect the natural ceilings of each task.

Technical Summary: Information-Theoretic Limits of Reliability and Scaling in Language Models

Problem Statement

Current narratives regarding Large Language Models (LLMs) often assume that perfect reliability is achievable for any task given sufficient scale. However, empirical evidence contradicts this, showing that performance on different tasks saturates at varying levels, with some tasks (e.g., creative writing) failing to converge toward perfect accuracy even as models scale, while others (e.g., code generation) approach near-perfect reliability. Existing scaling laws, such as the Chinchilla law, provide empirical power-law relationships between model size, training data, and loss but lack a rigorous theoretical interpretation for why these ceilings exist, why they differ across tasks, and how the interplay between data and capacity governs performance.

This paper addresses these gaps by introducing an information-theoretic framework that formalizes the fundamental limits of generative language models. It posits that reliability is not merely a function of scale but is bounded by intrinsic properties of the task distribution and the structural dependencies within the output.

Methodology and Theoretical Framework

The authors develop a framework grounded in information theory, specifically utilizing the Data Processing Inequality (DPI) and spectral analysis of task mixtures.

1. The Reliability Ceiling

The paper defines a reliability ceiling ( $R^*$ ) for any generative task, determined by the conditional entropy of the output given the input:
$R^* := \frac{I(X; Y)}{H(Y)}$
where $X$ is the observable context and $Y$ is the target output. The authors decompose the unreliability gap ( $1 - R^*$ ) into two components:

Resolvable Gap ( $\delta_r$ ): Arises from latent context ( $C_r$ ) that is relevant to the output but absent from the input (e.g., missing facts, user preferences). This gap can theoretically be closed by enriching the input (e.g., via retrieval or tool use).
Subjective Gap ( $\delta_u$ ): Arises from latent context ( $C_u$ ) that is inherently ambiguous or subjective (e.g., aesthetic preferences, cultural norms). This gap represents a permanent floor that no model can overcome, as the "correct" output is not uniquely determined by the input.

2. Autoregressive Degradation and Dependency Kernels

The framework extends beyond single-shot generation to account for the compounding errors inherent in autoregressive models. The authors introduce the Dependency Kernel ( $K_Y$ ), an $L \times L$ matrix where entries $K_Y(t, t')$ measure the mutual information between tokens $t$ and $t'$ conditioned on the rest of the sequence and input.

Banded Kernels: Tasks like code generation exhibit local dependencies (syntax, variable scoping). Errors are contained locally.
Dense Kernels: Tasks like creative writing exhibit global dependencies (tone, thematic arc). Errors propagate globally, causing rapid degradation of sequence-level reliability.

The paper derives that autoregressive generation degrades the reliability ceiling at a rate governed by the cumulative drift, which is a function of the task's dependency kernel structure.

3. Derivation of a First-Principles Scaling Law

By analyzing the spectral properties of the task mixture (specifically the decay of eigenvalues $\nu_T$ and the regularity of target projections $\bar{\mu}$ ), the authors derive a scaling law for cross-entropy loss $L(N, D)$ as a function of model parameters $N$ and training tokens $D$ :

$L(N, D) = \underbrace{H(Y | X)}_{\text{Irreducible Error}} + \max \left( \underbrace{\frac{A}{N^{\alpha}}}_{\text{Approximation Error}}, \underbrace{\frac{B}{D^{\beta}}}_{\text{Estimation Error}} \right) + \text{lower order}$

Where:

$\alpha = \frac{\bar{\mu}-1}{d(\nu_T + 1)}$ is the capacity exponent.
$\beta = \frac{\bar{\mu}-1}{\nu_T}$ is the data exponent.
The max form indicates that performance is bottlenecked by the scarcer resource (data or capacity).
The irreducible error term $H(Y|X)$ corresponds to the subjective gap and the conditional entropy of the task.

Key Contributions

Formalization of the Reliability Ceiling: The paper establishes that perfect reliability is information-theoretically impossible for tasks with non-zero conditional entropy given the input. It provides a taxonomy of tasks based on verifiability: Fully Verifiable (deterministic output), Ideally Verifiable (deterministic given full context), and Unverifiable (subjective output).
Dependency Kernel and Error Propagation: It introduces the dependency kernel to quantify how inter-token correlations dictate the rate of error propagation in autoregressive generation, explaining why code models are more robust to errors than creative writing models.
Max-Form Scaling Law: The authors derive a scaling law from first principles that recovers the Chinchilla law as a special case (along the compute-optimal frontier where data and capacity are balanced) but generalizes it to a max-form. This formulation predicts that scaling one resource (e.g., data) without the other yields diminishing returns once the bottleneck shifts.
Unified Explanation of Practical Phenomena: The framework structurally explains:
- Retrieval-Augmented Generation (RAG): Works by converting the resolvable gap ( $\delta_r$ ) into observable input, raising the effective ceiling.
- Catastrophic Forgetting: Occurs because fine-tuning reallocates the model's finite representational budget (eigenmodes) toward the target task, suppressing modes required for orthogonal tasks.
- Benchmark Instability: Leaderboard rankings on low-ceiling tasks are inherently noisy due to the subjective gap, making rank correlations unstable.

Results and Empirical Validation

Benchmark Saturation Analysis: The authors adapt a saturation index metric to analyze benchmarks across code, math, Q&A, and writing. Results show that writing benchmarks exhibit high saturation at scores well below perfection, supporting the existence of a subjective gap ( $\delta_u > 0$ ). Conversely, math and code benchmarks with verifiers show near-perfect top scores, consistent with a ceiling of $R^* \approx 1$ .
Dependency Kernel Visualization: Empirical plots of mutual information between tokens confirm the theoretical distinction: code and math tasks show banded/block-diagonal structures, while writing and summarization tasks show dense kernels.
Scaling Law Recovery: The derived exponents $\alpha$ and $\beta$ align with empirical estimates (e.g., $\alpha \approx 0.348, \beta \approx 0.366$ ), validating the spectral assumptions.

Significance and Claims

The paper claims to offer a unified theory of performance limits in generative language models. Its primary significance lies in shifting the discourse from "scaling will solve everything" to a nuanced understanding of resource-complexity tradeoffs.

Theoretical Rigor: It moves beyond empirical curve-fitting to provide a structural account of why scaling improves reliability only up to a task-specific ceiling determined by information availability and dependency structure.
Practical Guidance: It suggests that for unverifiable tasks, no amount of scaling will achieve perfect reliability; instead, efforts should focus on reducing the resolvable gap (via better context) or accepting the subjective floor.
Optimization Strategy: The max-form scaling law provides a theoretical justification for the "compute-optimal" strategy of balancing model size and data, warning against scaling one resource far beyond the other.

The authors conclude that the "jagged frontier" of AI capability—where models excel at some tasks but fail at others—is not a bug but a fundamental consequence of the information-theoretic properties of the tasks themselves.