💬 NLP

Learning is Forgetting: LLM Training As Lossy Compression

This paper proposes that large language models function as instances of lossy compression that approach the Information Bottleneck bound during pre-training, demonstrating that a model's compression optimality and information content can predict its downstream performance across diverse benchmarks.

Original authors: Henry C. Conklin, Tom Hosking, Tan Yi-Chern, Julian Gold, Jonathan D. Cohen, Thomas L. Griffiths, Max Bartolo, Seraphina Goldfarb-Tarrant

Published 2026-04-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Henry C. Conklin, Tom Hosking, Tan Yi-Chern, Julian Gold, Jonathan D. Cohen, Thomas L. Griffiths, Max Bartolo, Seraphina Goldfarb-Tarrant

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Learning is Actually Forgetting

Imagine you are trying to memorize a massive library of books to become the world's best storyteller. You have a brain the size of a city (a Large Language Model, or LLM).

If you tried to remember every single word, typo, and random fact in those books perfectly, your brain would explode. You wouldn't be able to tell a good story because you'd be drowning in useless details.

This paper argues that LLMs don't learn by remembering everything. Instead, they learn by forgetting.

Think of it like MP3 compression for music. When you turn a huge CD into an MP3, the computer throws away the sounds humans can't hear (like very high-pitched squeaks) to save space. The song still sounds great, but the file is much smaller.

The authors say LLMs do the same thing with language. They take the "noise" of the internet and compress it down to the "signal"—the parts that actually help them predict what word comes next.

The Two-Phase Training Dance

The paper describes how these models learn in two distinct phases, like a student studying for a final exam:

Phase 1: The "Cramming" Phase (Fitting)

What happens: The model reads the data and tries to memorize everything. It gets very excited, storing lots of information.
The Analogy: Imagine a student frantically writing down every single fact from a textbook, hoping nothing is missed. The "notes" (representations) are huge and messy.
The Result: The model gets good at predicting the next word, but it's holding onto too much junk.

Phase 2: The "Editing" Phase (Compression)

What happens: The model starts to realize, "Wait, I don't need to remember the color of the sky in every single sentence to know what comes next." It starts deleting the useless details.
The Analogy: The student goes back through their notes and highlights only the key concepts. They throw away the doodles and the random facts. They are now compressing their knowledge.
The Result: The model becomes smarter and more efficient. It keeps only the information that is actually useful for the goal (predicting the next word).

The Paper's Discovery: The best models are the ones that get really good at this "editing" phase. They approach a theoretical limit called the Information Bottleneck, which is the perfect balance between knowing enough to be smart but forgetting enough to be efficient.

Why Size Matters (The "Small Brain" Problem)

The paper found that bigger models are better at forgetting.

The Big Models (7B, 32B parameters): These are like geniuses. They can read the whole library, memorize the basics, and then successfully throw away the junk to find the perfect summary. They get very close to that "perfect compression" line.
The Small Models (1B parameters): These are like students with a smaller memory. They can memorize the facts (Phase 1), but when they try to edit and compress (Phase 2), they get confused. They keep too much junk or delete the wrong things. They struggle to reach that "perfect compression" state.

The Takeaway: To learn effectively, you need enough brainpower to first understand the complexity, and then enough power to simplify it.

The "Secret Sauce" of Performance

The authors tested many different models against real-world tests (like math, logic, and writing). They found a surprising link:

The models that are best at "forgetting" (compressing) are the ones that perform best.

Complexity vs. Expressivity: Imagine a suitcase.
- Complexity is how much stuff you packed.
- Expressivity is how well you can use that stuff to build a house.
- Optimality is the ratio of "useful house-building stuff" to "total stuff packed."
The winning models didn't necessarily pack the most stuff; they packed the most useful stuff. They had a high "Expressivity per bit of Complexity."

The "Preference" Twist

The paper also looked at how models learn to be "helpful" and "safe" (aligning with human preferences).

Pre-training (The Library): The model learns general facts by reading the internet. This is where the "compression" happens.
Post-training (The Coaching): Humans then teach the model, "Don't say that, say this instead."
The Finding: The "compression" of the internet data predicts how good the model is at math and facts. But the amount of "preference information" (how much it learned about what humans like) predicts how good it is at following instructions.

It's like a chef:

Pre-training is learning how to cook (compressing recipes).
Post-training is learning that the customer hates cilantro (preference).
A great chef needs to be good at both compressing the recipes and remembering the customer's specific dislikes.

Summary: The "Lossy" Lesson

This paper gives us a new way to look at AI. Instead of thinking of these models as giant databases that remember everything, we should see them as master editors.

Learning = Forgetting.
Success = Keeping only what matters.
The Best Models = The ones that forget the best.

By understanding that LLMs are essentially lossy compression algorithms, we can better predict how they will perform, how to train them, and why some are smarter than others. It turns out, in the world of AI, knowing what to ignore is just as important as knowing what to keep.

1. Problem Statement

Despite the prevalence of Large Language Models (LLMs), there is a limited understanding of how their internal representational spaces are structured and how this structure relates to learning and generalization. Existing interpretability methods often focus on specific circuits, neurons, or behavioral outputs, lacking a unified theoretical framework that connects the training dynamics of massive models to established theories of learning. The authors propose viewing LLMs not just as predictors, but as instances of lossy compression, where the model learns to retain only the information in the training data relevant to its objective (next-token prediction) while discarding irrelevant details.

2. Methodology

The paper operationalizes Rate Distortion Theory and the Information Bottleneck (IB) principle to analyze LLM training dynamics.

Core Theoretical Framework

Information Plane: The authors map model representations ( $Z$ $Z$ ) onto a 2D plane defined by:
- Complexity ( $I(X; Z)$ ): Mutual information between the input ( $X$ ) and the representation.
- Expressivity ( $I(Y; Z)$ ): Mutual information between the representation ( $Z$ ) and the target output ( $Y$ ).
The IB Bound: The optimal compression curve where a model preserves only the information about $X$ necessary to predict $Y$ . Models approaching this bound are considered "optimally compressed."
Training Trajectory: The IB theory predicts a two-phase trajectory:
1. Fitting Phase: $I(Y; Z)$ increases rapidly (learning the task).
2. Compression Phase: $I(X; Z)$ decreases (forgetting irrelevant input details) while maintaining $I(Y; Z)$ , moving the model toward the optimal bound.

Technical Implementation: Soft Entropy Estimation

Computing mutual information for high-dimensional continuous embeddings in LLMs is traditionally intractable due to memory constraints. The authors utilize a Soft-Entropy Estimator (based on Conklin, 2025) to overcome this:

Normalization: Embeddings are normalized to the unit sphere.
Soft Quantization: Instead of hard binning, embeddings are softly assigned to $n$ random anchor points sampled uniformly from the unit sphere using a softmax function over cosine similarities.
Entropy Calculation: Shannon entropy is computed on the resulting categorical distribution. This allows for efficient, differentiable estimation of mutual information across the entire model in a single forward pass.
Contextual Back-off: To analyze context, the authors compute mutual information conditioned on $n$ -grams (token, bigram, trigram, quadgram) using a "back-off" strategy to handle data sparsity.

Experimental Setup

Models: Analyzed the OLMo2 family (1B, 7B, 32B) with access to intermediate checkpoints to track pre-training time-courses. Also analyzed a diverse array of 75 open-weight models (Llama, Gemma, Qwen, Pythia, etc.) at their final training stages.
Datasets:
- C4: Used for estimating input/output mutual information (next-token prediction).
- Tulu: Used to estimate Preference Information ( $I(Z; \text{preferred})$ ) to measure alignment with human preferences.
Benchmarks: Performance evaluated against 6 benchmarks (MMLU Pro, BBH, Math LVL5, IFEval, GPQA, MuSR).

3. Key Contributions & Results

A. Validation of the Two-Phase IB Trajectory at Scale

Finding: Pre-training dynamics of LLMs (specifically OLMo2 7B) closely follow the theoretical IB prediction.
Evidence: Models first expand representations (increasing expressivity) and then enter a compression phase where input complexity decreases while expressivity is maintained.
Correlation: The transition to the compression phase coincides with the saturation of the next-token prediction loss.

B. The Role of Scale in Compression

Finding: Model size is a critical factor in achieving optimal compression.
Evidence:
- Large Models (7B, 32B): Successfully follow the two-phase trajectory and converge near the IB bound.
- Small Models (1B): Struggle to compress. They complete the expansion phase but fail to reduce input complexity, oscillating away from the optimal bound. This suggests a parameter threshold is necessary to compress complex data effectively.

C. Convergence of Diverse Models

Finding: Across 75 open-weight models from different families and training recipes, representations converge to a similar point near the IB bound at the end of training.
Significance: This suggests that optimal lossy compression is a fundamental property of deep learning on language data, independent of specific architectural or training idiosyncrasies.

D. Linking Representation Structure to Performance

The authors establish strong correlations between information-theoretic metrics and downstream performance:

Optimality Metric: The ratio of Expressivity to Complexity ( $I(Y; Z) / I(X; Z)$ ) is a significant predictor of performance ( $r = 0.52, p < 0.001$ ). Better models are more optimally compressed.
Context vs. Token Information:
- Models with lower token-level complexity and higher local contextual information (bigram/trigram) perform better.
- This indicates that high-performing models allocate representational capacity to understanding context rather than just memorizing individual tokens.
Preference Information:
- The amount of preference information ( $I(Z; \text{preferred})$ ) in a model is a strong predictor of performance ( $r = 0.76$ ).
- Differentiation: Optimal compression of general data (C4) predicts performance on reasoning and factual tasks (Math, Science) but not instruction following. Instruction following is predicted by preference information, suggesting post-training (RLHF/DPO) specifically edits the model to encode human preferences without drastically altering the core compression of the training data.

4. Significance and Implications

Unified Theory of Learning: The paper provides a unified Information-Theoretic framing for LLM learning, bridging the gap between theoretical learning curves (IB theory) and the practical behavior of massive models.
Actionable Insights:
- Training Stopping Criteria: The distance to the IB bound could serve as a stopping criterion for pre-training (stop when the model stops compressing).
- Model Selection: One can select checkpoints based on their "optimality" or "preference information" rather than running expensive benchmark suites.
- Interpretability: Offers a holistic view of the model as a whole system rather than focusing on individual neurons, explaining why models generalize (by discarding irrelevant noise).
Scalability: The proposed soft-entropy estimation method is the first to make information-theoretic analysis feasible at the scale of modern LLMs (e.g., 32B parameters), overcoming previous memory bottlenecks.

Conclusion

The paper argues that "learning is forgetting." LLMs achieve impressive performance by acting as lossy compressors that discard irrelevant details of the training data while retaining the minimal sufficient statistics required for prediction and alignment. The optimality of this compression, and the specific types of information (contextual vs. token, preference vs. factual) that survive the process, directly dictate the model's capabilities and generalization performance.