🤖 machine learning

Geometric and Information Compression of Representations in Deep Learning

This paper challenges the assumption that low mutual information between inputs and representations implies geometric compression, revealing instead a complex, nonlinear relationship where generalization acts as a confounding factor rather than a direct consequence.

Original authors: Linara Adilova, Henning Petzka, Asja Fischer, Bernhard C. Geiger

Published 2026-06-23✓ Author reviewed ⓘ

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Linara Adilova, Henning Petzka, Asja Fischer, Bernhard C. Geiger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to recognize different types of fruit. You show it thousands of pictures of apples, bananas, and oranges. The robot's job is to look at the picture (the input) and create a secret, internal summary (the latent representation) that helps it decide which fruit it is.

This paper asks a very specific question about how that robot "thinks": Does making the robot's internal summary "smaller" or "simpler" always mean it's grouping similar fruits together more tightly?

For a long time, researchers believed the answer was "Yes." They thought that if the robot threw away a lot of unnecessary details about the picture (low Information), it must be because it had neatly packed all the apples into one tight pile, all the bananas into another, and so on (high Geometric Compression).

The authors of this paper say: "Not so fast."

Here is a breakdown of their findings using simple analogies:

1. The Two Ways to Measure "Simplicity"

The researchers looked at two different ways to measure how "compressed" the robot's thinking is:

Information Compression (The "Noise" Test): This measures how much of the original picture the robot remembers. If the robot forgets almost everything about the specific photo (like the lighting or the background) and only remembers the fruit type, it has low information.
- Analogy: Imagine you are describing a friend to a police sketch artist. If you say, "It's a person," you have compressed the information heavily. You forgot the hair color, the height, and the clothes.
Geometric Compression (The "Packing" Test): This measures how tightly the robot groups similar items together. If all the "apple" pictures are clustered in one tiny, tight circle in the robot's mind, and all "banana" pictures are in another tight circle far away, that is high geometric compression.
- Analogy: Imagine a closet. If you throw all your shirts into one big, messy pile, that's low geometric compression. If you fold them perfectly and stack them in a neat, tight tower, that's high geometric compression.

2. The Big Surprise: They Don't Always Match

The paper's main discovery is that low information does not automatically mean tight packing.

The authors used a "toy model" (a simple simulation) to show why. They found two ways to get "low information" (forgetting details):

The "Tight Pile" Method: The robot learns perfectly. It groups all apples into a tiny, neat pile. It forgets the background noise.
- Result: Low Information + High Geometric Compression. (This is what people used to expect).
The "Static" Method: The robot is so confused or noisy that it can't tell the difference between an apple and a banana. It just outputs random static. Because it's outputting random noise, it isn't remembering any details about the input.
- Result: Low Information + Zero Geometric Compression (everything is a messy blur).

The Analogy:
Imagine you are trying to sort a deck of cards.

Scenario A: You sort them perfectly by suit and number. You have removed all the "mess" (low information) and the cards are in tight, neat piles (high geometric compression).
Scenario B: You throw the cards into a blender. The blender destroys the cards. Now, you have no information about the cards left (low information), but you definitely don't have neat piles of suits (no geometric compression).

The paper shows that in deep learning, the robot can end up in Scenario B (just being noisy) and still look like it has "compressed" the information, even though it hasn't organized the data at all.

3. The "Confusing" Factor: Generalization

The researchers found that the relationship between "forgetting details" and "packing things tight" is actually negative and non-linear.

Sometimes, when the robot gets better at generalizing (predicting new fruits it hasn't seen before), the two measures move in opposite directions.
They suspect that Generalization (the robot's ability to be smart about new data) is acting like a "confounder." It's a third factor that messes up the simple link between "forgetting" and "packing."

Think of it like a student taking a test:

If the student memorizes the answers perfectly, they might get a high score, but they haven't really "learned" the concept (bad generalization).
If the student understands the concept, they might forget the specific numbers but get the logic right (good generalization).
The paper suggests that trying to measure "learning" just by looking at how much data is "forgotten" is misleading because the student's actual understanding (generalization) changes the whole picture.

4. What They Did to Prove It

To prove this, they didn't just guess. They built many different robot brains (neural networks) and forced them to learn in different ways:

Some were forced to be very noisy (adding random static to their thoughts).
Some were forced to group things tightly (using a special rule to make them cluster).
They tested these robots on images (like cats and dogs) and text (like news articles).

The Result: Across almost all these experiments, they found that low information did not reliably mean tight packing. In fact, sometimes the tighter the packing got, the more information the robot seemed to hold, and vice versa.

The Bottom Line

For a long time, scientists thought: "If the robot forgets the details, it must be organizing the data perfectly."

This paper says: "Not necessarily."
The robot might be forgetting details because it's organizing perfectly, OR it might be forgetting details because it's just confused and noisy. You cannot use "how much it forgets" as a simple shortcut to measure "how well it organizes."

The paper concludes that we need to be much more careful when we try to explain why deep learning works. We can't just say "it compressed the data," because compression can happen in two very different ways: one that is smart (good clustering) and one that is just messy (noise).

Technical Summary: Geometric and Information Compression of Representations in Deep Learning

Problem Statement
Deep neural networks (DNNs) transform input data into latent representations that facilitate downstream tasks. These representations are often analyzed through two distinct lenses: information-theoretic compression, measured by the mutual information (MI) between inputs and representations, and geometric compression, characterized by the clustering of representations (e.g., neural collapse). While prior work has suggested a tight, positive correlation between these two forms of compression—implying that low MI necessitates tight geometric clustering—the precise relationship remains poorly understood. A central open question is whether low MI between inputs and representations necessarily implies geometrically compressed latent spaces, and vice versa. Furthermore, the role of generalization in mediating this relationship is unclear.

Methodology
The authors investigate this interplay using two distinct experimental frameworks designed to induce variation in both information-theoretic and geometric properties:

Conditional Entropy Bottleneck (CEB) Networks: These models utilize data-dependent additive Gaussian noise. The training objective minimizes the conditional entropy bottleneck $I(X; Z|Y)$ while maintaining classification performance. By sweeping the trade-off parameter $\beta$ , the authors generate models with varying degrees of MI compression and generalization capabilities. The MI is estimated directly via the variational bound embedded in the training loss.
Gaussian Dropout Networks: These models employ multiplicative fixed-variance noise (continuous dropout). To induce variation in geometric compression without altering the noise variance, the authors introduce a geometric regularizer based on the Neural Collapse (NC) statistic. This allows for the training of models with differing cluster tightness while keeping the noise injection mechanism constant. For these deterministic-like models (with stochasticity only via dropout), MI is estimated externally using the Difference of Entropies (DoE) estimator, which approximates unconditional and conditional entropies via parametric logistic distributions.

Theoretical Foundation
A critical prerequisite for the study of dropout-regularized networks is the guarantee that the MI between inputs and representations is finite. The paper proves Theorem 1, extending previous results from ReLU networks to networks with arbitrary real analytic activation functions (e.g., GELU, sigmoid). The proof relies on the Lojasiewicz Structure Theorem to show that the set of zeros for real analytic functions has measure zero and that the function behaves polynomially near these zeros, ensuring the finiteness of the expected log-determinant required for finite MI. This establishes a theoretically sound basis for applying information-theoretic analysis to modern architectures with continuous dropout.

Key Results
The study evaluates 8 architectures (including fully connected networks, CNNs, and transformers) across 5 datasets (vision and NLP tasks), training approximately 500 models. The analysis focuses on end-of-training representations rather than training dynamics.

Negative Non-Linear Correlation: Contrary to the intuition that low MI and tight geometric clustering (low NC) always coincide, the experiments reveal a consistent negative and non-linear correlation between MI and geometric compression. In many setups, models with lower MI exhibited looser geometric clustering, and vice versa.
Role of Generalization as a Confounder: The relationship between MI and geometric compression is not invariant. The sign and strength of the correlation change depending on the training regime and the model's generalization ability. For instance, in Gaussian dropout experiments, varying the regularization strength $\lambda$ produced regimes where the correlation was negative, near-zero, or even positive. The authors observe that the correlation tends to vanish or reverse in regimes of stable generalization, suggesting that generalization acts as a confounder rather than a direct consequence of either compression metric.
Decoupling of Compression Types: A toy model analysis (Fig. 1) demonstrates that low MI can arise from two distinct sources: (1) strong geometric clustering (small encoder spread) or (2) high noise variance (uninformative encoders). Geometric compression (neural collapse) only occurs in the former case. This explains why low MI does not reliably predict geometric compression.

Significance and Claims
The paper contributes to the debate on representation learning by challenging the assumption that information-theoretic and geometric compression are interchangeable proxies for representation quality or generalization.

Theoretical Contribution: The proof of finite MI for continuous dropout networks with analytic activations enables rigorous information-theoretic analysis of a broader class of state-of-the-art models.
Empirical Insight: The findings suggest that the connection between geometric and information compression is more nuanced than previously assumed. The authors posit that generalization ability may be the underlying factor confounding the observed relationships between MI and geometric clustering.
Caveats: The authors emphasize that their conclusions are conditioned on the specific MI estimators used (variational bounds for CEB and DoE for dropout) and that systematic comparisons across estimators are necessary to validate the robustness of the observed MI-NC relationship. They explicitly state that the causal structure among MI, geometric compression, and generalization requires further investigation beyond the correlational evidence presented.

The paper concludes that claims about generalization based solely on either MI or geometric compression should be treated with caution, as the two metrics do not necessarily align and can exhibit complex, non-linear interactions dependent on the training setup.

1. The Two Ways to Measure "Simplicity"

2. The Big Surprise: They Don't Always Match

3. The "Confusing" Factor: Generalization

4. What They Did to Prove It

The Bottom Line

More like this