🤖 machine learning

Language models suffer from a curse of ambiguity

This paper identifies and theoretically analyzes a "curse of ambiguity" in large language models, demonstrating that ambiguous next-token distributions are inherently harder to learn accurately due to increased capacity requirements, embedding sizes, training steps, and amplified sampling noise, a finding validated through both synthetic and real-world experiments.

Original authors: Nicolas Zucchet, Hyun Dong Lee, Scott Linderman

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Nicolas Zucchet, Hyun Dong Lee, Scott Linderman

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern computers that write and speak are built on a simple, powerful idea: they predict the next word in a sentence by looking at the words that came before. This process happens one word at a time, like a person thinking through a story, choosing the most likely next step based on what has already been said. For years, engineers have focused on making these machines better at finding the single most obvious answer. But as these systems grow more advanced, they are increasingly asked to handle situations where there is no single obvious answer. When a sentence could end in many different, equally reasonable ways, the machine must learn to spread its confidence across all those possibilities, not just pick the top one. This ability to understand uncertainty is becoming critical, especially as these machines begin to write the training data for their own future versions. If they fail to capture the full range of possibilities, they risk creating a narrow, repetitive loop that degrades the quality of their own learning.

A team of researchers at Stanford University has discovered a fundamental barrier that makes this task surprisingly difficult. They found that the more ambiguous a situation is—meaning the more plausible next words there are—the harder it is for the machine to learn the correct distribution of probabilities. They call this the "curse of ambiguity." It is not just a matter of the machine being slightly slower or needing more data; the difficulty is baked into the very architecture of the neural networks that power these models. The researchers showed that as the number of possible correct answers increases, the machine requires significantly more internal storage space, larger internal representations of words, and many more steps of training to get the math right. Even when the machine finally learns, the process of picking a single word from the many possibilities introduces a type of noise that prevents it from ever perfectly matching the true distribution of language.

To understand why this happens, the team broke the problem down into its smallest parts. They treated the machine's final decision-making layer as a simple classifier that maps a context to a set of possible outcomes. In their experiments, they created synthetic tasks where they knew the exact ground truth: a specific number of words were equally likely to follow a given phrase. They found that storing a pattern with ten possible outcomes required roughly ten times more capacity than storing a pattern with only one. It is as if the machine has to build a separate, distinct pathway for every possible option it must remember. Furthermore, they discovered that the internal "vocabulary" the machine uses to hold these contexts must grow in size to accommodate the ambiguity. If the machine tries to represent a situation with many valid endings using a representation that is too small, it simply cannot distinguish between the options, no matter how much it trains.

The difficulty extends beyond just storage. The researchers also analyzed how the machine learns over time. They found that when a situation has many possible outcomes, the learning process starts much more slowly. Because the machine sees each specific correct word less frequently when there are many of them, the signal it receives to adjust its internal settings is weaker. This means it takes longer to begin making progress. Even worse, the act of training on real data, where the machine only sees one random example of the correct next word at a time, introduces a persistent error. When the target is a single, certain word, the machine can eventually learn it perfectly. But when the target is a spread of many words, the randomness of seeing just one example at each step creates a floor of error that the machine cannot climb over. No matter how long it trains, it will always be slightly off, unable to perfectly replicate the true spread of probabilities.

The team confirmed these findings not just in their controlled, synthetic tests, but also in large language models trained on real-world text. By analyzing how these models performed on actual data, they observed the same signatures: as the ambiguity of a context increased, the model's predictions became less accurate, and it tended to leak probability mass to words that should not be there. This suggests that the curse of ambiguity is a universal property of these systems, affecting everything from small experimental models to the massive, trillion-parameter systems used today. The researchers argue that this limitation is not a bug that can be fixed with a simple software update, but a fundamental constraint of how these networks represent uncertainty.

This discovery changes how we should think about the capabilities of artificial intelligence. It suggests that while scaling up the size of these models helps, it does not solve the core problem of ambiguity. Even the largest models will always struggle to perfectly model situations with many plausible outcomes, leaving a residual fraction of the truth unlearned. This has practical implications for how we trust these systems. In fields where precision matters, such as medical diagnosis or legal reasoning, we must be aware that the model's confidence distribution may be inherently noisy when the answer is not clear-cut. The work provides a new framework for understanding when to trust a machine's output and when its uncertainty is a sign of a deeper structural limitation rather than a lack of data. Ultimately, the paper reveals that the very thing that makes human language rich and flexible—the ability to say many different things in many different ways—is the same thing that makes it the hardest thing for a machine to learn.

Technical Summary: The Curse of Ambiguity in Large Language Models

Problem Statement

Large language models (LLMs) increasingly rely on sampling to drive improvements in inference-time scaling, reinforcement learning, and synthetic data generation. These methods depend critically on the fidelity of the model's learned next-token distribution. However, not all distributions are equally learnable. The paper identifies a fundamental limitation termed the curse of ambiguity: in autoregressive language models (and broadly in neural networks producing discrete probability distributions), the more ambiguous a next-token distribution is (i.e., the more tokens share significant probability mass), the harder it is to learn accurately.

This phenomenon poses risks for model performance, particularly in ambiguous settings where multiple continuations are plausible. If models fail to capture the full distribution and instead collapse toward a single mode, it can lead to a lack of diversity in generated content, compounding errors in recursive training, and poor calibration in high-stakes applications.

Methodology

The authors employ a multi-faceted approach combining theoretical analysis, controlled synthetic experiments, and evaluation on real-world data.

1. Theoretical Framework

The authors formalize next-token prediction as a context-to-token classification problem. They isolate the linear head of the Transformer architecture, treating the backbone's context embeddings as given, to analyze the specific challenges of fitting target distributions $p_i$ defined over a vocabulary of size $d$ .

Ambiguity Metric: Ambiguity is quantified by the support size $k$ (the number of tokens with non-zero probability in the ground truth). For uniform distributions, $k$ is the exact support size; for non-uniform distributions, it is the effective support size (exponential of entropy).
Architectural Analysis (Sections 3): The authors analyze the capacity and representational requirements of the linear head.
- Capacity: Using a Hebbian-like construction and signal-noise decomposition, they derive that the number of contexts a model can store scales inversely with the support size $k$ .
- Embedding Dimension: Drawing on compressed sensing and the theory of convex polytopes (specifically $k$ -neighborliness), they prove that the embedding dimension $h$ must scale linearly with $k$ ( $h \ge 2k$ ) to exactly represent all possible size- $k$ distributions.
Learning Dynamics Analysis (Section 4): The authors study gradient flow and stochastic gradient descent (SGD) dynamics.
- Optimization Speed: They show that learning ambiguous distributions is slower initially because each token in the support is seen less frequently, providing a weaker signal. The convergence rate in the late phase is independent of $k$ , but the loss prefactor scales with $k$ .
- Sampling Noise: They analyze the effect of training on single sampled tokens rather than full distributions. They demonstrate that sampling introduces a "noise floor" in the KL divergence that scales with $k$ , preventing the model from reaching the same asymptotic loss as deterministic targets.

2. Empirical Validation

Synthetic Tasks (Section 5.1): The authors train Transformers on a controlled 3-gram task derived from the TinyStories corpus. Ground truth distributions are known, allowing precise measurement of prediction errors. They ablate model capacity, embedding dimension, training steps, and loss functions (cross-entropy vs. KL divergence).
Real-World Models (Section 5.2): They evaluate pretrained models (OLMo-2 1B and 13B) on natural data (Dolma v1.7). Since ground truth is unknown, they use the larger model as a proxy for the true distribution and estimate context frequency via embedding similarity.

Key Contributions

Formalization of the Curse of Ambiguity: The paper defines and formalizes the phenomenon where higher ambiguity (larger support size $k$ ) leads to increased difficulty in learning accurate distributions.
Identification of Architectural Roots:
- Capacity Constraint: Ambiguous distributions require $k$ times more parameter capacity to store than deterministic ones ( $Capacity \propto hd/k$ ).
- Representation Constraint: Ambiguous distributions require embedding dimensions that scale linearly with the support size ( $h \ge 2k$ ) to be represented without error.
Identification of Learning Roots:
- Optimization Slowdown: Ambiguous distributions learn slower initially (rate $\propto 1/k$ ) and converge to a higher loss plateau.
- Noise Amplification: Training on sampled tokens (standard practice) amplifies noise for ambiguous distributions, creating a $k$ -dependent noise floor in the KL divergence that prevents perfect calibration.
Empirical Verification: The study confirms these theoretical signatures in both synthetic settings and real-world language models, showing that performance degrades as effective support size increases, even when context frequency is held constant.

Results

Capacity and Embeddings: Experiments on synthetic data confirm that increasing the embedding dimension or model capacity improves performance uniformly across support sizes, but specifically, increasing the embedding dimension or optimizing the expected KL divergence (removing sampling noise) significantly improves the modeling of high-ambiguity contexts while leaving deterministic contexts largely unaffected.
Learning Dynamics: Simulations show that ambiguous distributions suffer from a "noise floor" in SGD that is absent in gradient flow. As training progresses, the loss for ambiguous distributions plateaus while the loss for deterministic distributions continues to decrease.
Real-World Behavior: In real-world models, contexts with higher effective support sizes exhibit higher KL divergence and systematically "leak" probability mass outside the true support, leading to over-entropy in the sequence-level distribution.

Significance and Claims

The paper claims to provide a new perspective on the statistical capabilities of large language models, offering a practical framework for understanding when to trust a model's output distribution.

Mechanistic Insight: The work moves beyond empirical scaling laws to provide a mechanistic explanation for why models struggle with uncertainty. It links the difficulty of learning ambiguity to fundamental geometric and optimization properties of neural networks.
Implications for Scaling: The authors argue that while scaling (increasing parameters and data) mitigates the curse, it does so only asymptotically. Even at large scales, a residual fraction of the distribution remains unmodeled, particularly for highly ambiguous contexts.
Broader Relevance: The findings extend beyond LLMs to any neural network producing discrete probability distributions. The paper draws a parallel to the "paradox of choice" in cognitive psychology, suggesting that representing a distribution spread over many options is fundamentally more demanding in capacity and samples than representing a single answer.
Practical Framework: The results suggest that for tasks requiring high-fidelity distribution modeling (e.g., reasoning, planning, or generating diverse content), simply increasing model size may be insufficient; specific attention to embedding dimensions and the handling of sampling noise (e.g., via KL divergence optimization or larger batch sizes) may be necessary.

The authors conclude that the curse of ambiguity is a fundamental constraint that must be accounted for when designing models for tasks involving uncertainty, and that current models likely underfit the true complexity of ambiguous natural language contexts.