The Information-Theoretic Imperative: Compression and the Epistemic Foundations of Intelligence

1. The Problem: The Convergence Mystery

The paper addresses a fundamental puzzle in neuroscience and artificial intelligence: Why do biological neural networks (brains) and engineered deep neural networks (DNNs) converge on similar representational structures despite having radically different substrates, optimization dynamics, and evolutionary histories?

• Observation: Performance-optimized DNNs quantitatively predict primate ventral stream responses (e.g., CNN features predict Inferior Temporal cortex activity). Both systems exhibit sparse coding, hierarchical abstraction, and invariance to nuisance transformations.
• The Gap: Existing explanations (shared natural image statistics, task structure, or architectural inductive biases like convolution) are insufficient. They fail to explain why systems with different constraints (metabolic limits/local plasticity in biology vs. abundant compute/backpropagation in AI) arrive at the same solutions.
• The Core Question: What substrate-independent physical constraint channels both evolved and engineered learning systems toward similar representational solutions?

2. Methodology and Theoretical Framework

The authors propose a unified framework based on two core principles: the Information-Theoretic Imperative (ITI) and the Compression Efficiency Principle (CEP).

A. The Information-Theoretic Imperative (ITI)

• Definition: For "active persisters" (systems that maintain organized states in uncertain environments via regulation), persistence under finite resources (energy, memory, time) creates a strong selection pressure toward predictive compression.
• Logic: To survive, a system must predict future states. Under resource constraints, storing raw history is impossible; the system must compress sensory data into a compact internal structure that preserves predictive relevance.
• Scope: Applies to both biological evolution (metabolic budgets) and AI training (compute/data budgets).

B. The Compression Efficiency Principle (CEP)

• Definition: Under recurring distribution shifts (changes in environment, context, or regime), efficient compression penalizes "shortcuts" (shift-unstable correlations) and favors shift-stable invariants.
• The Mechanism (The "Exception Tax"):
  • Shortcuts: Representations relying on correlations that flip across environments (e.g., a background texture correlating with an object class) incur a growing "exception tax." Formally, this manifests as linear excess codelength (regret) relative to environmental diversity ($E$).
  • Invariants: Representations encoding shift-stable structure (e.g., object shape regardless of lighting) amortize the cost of adaptation. They incur only sublinear regret (dominated by estimation error, not patch accumulation).
• The Crossover ($E^*$): There exists a threshold of environmental diversity ($E^*$) beyond which the cost of maintaining environment-specific "patches" (shortcuts) exceeds the structural cost of learning the invariant mechanism. Beyond $E^*$, invariant representations become the compression-efficient solution.

C. The Causality Bridge (Conditional)

• The paper clarifies that invariance alone does not guarantee causality.
• Condition: If the environment provides intervention-rich shifts (perturbing factors independently) and exhibits approximately modular causal structure (Independent Causal Mechanisms, ICM), then the shift-stable invariants selected by CEP align with causal mechanisms.
• Result: Under these conditions, the system learns representations that support localized updates and intervention-invariant prediction.

3. Key Contributions

1. Unification of Biology and AI

The paper provides a single physical explanation for the convergence of brains and AI. It argues that both are navigating the same rate-distortion frontier under the constraint of predictive compression.

• Biology: Metabolic constraints force the brain to minimize the "exception tax" of mismatch signaling and plasticity.
• AI: Optimization objectives (Negative Log-Likelihood) minimize codelength; distribution shifts reveal the inefficiency of shortcut-based compression.

2. Operationalization via Codelength and Regret

Instead of relying on difficult-to-estimate high-dimensional mutual information (common in Information Bottleneck theories), the authors operationalize the theory using predictive codelength (negative log-likelihood) and excess codelength (regret) under shift.

• Metric: $\Delta L = L_{shifted} - L_{baseline}$.
• Prediction: Shortcut strategies show $\Delta L \propto E$ (linear growth); Invariant strategies show $\Delta L \propto \log E$ or sublinear growth.

3. Resolution of Biological Mysteries

The framework explains three specific biological phenomena:

• Metabolic Budget: The brain's high energy cost is justified by the steep marginal cost of signaling/plasticity; inefficient strategies (requiring constant re-patching) are selected against.
• Sensory Efficiency: Early sensory systems (e.g., fly motion neurons) approach theoretical limits because tight resource budgets force frontier-seeking optimization.
• Ventral Hierarchy: The progression from V1 to IT (increasing tolerance/invariance) is a progressive extraction of shift-stable invariants to amortize description length across the natural shift family generated by active sensing.

4. Testable Predictions and Falsifiability

The paper outlines four specific, falsifiable predictions (Section 6):

1. Efficiency-Robustness Coupling: Higher frontier-relative efficiency should correlate with better Out-of-Distribution (OOD) robustness.
2. Regret Scaling Separation: Under "shortcut-flipping" shifts, the codelength gap between shortcut and invariant strategies should grow linearly with environmental diversity ($E$).
3. $\epsilon$-Tracking: Learned invariances should track the specific shift family ($\epsilon$) experienced during training, not a universal prior.
4. Active Sensing Advantage: Systems with active/interactive data generation (richer $\epsilon$) should show reduced shortcut reliance compared to passive systems.

4. Results and Evidence

The paper synthesizes existing literature and proposes new experimental protocols rather than presenting a single new dataset.

• Biological Evidence:
  • Fly Photoreceptors: Demonstrate near-optimal coding efficiency under naturalistic stimuli, consistent with tight metabolic constraints.
  • Ventral Stream: Hierarchical tolerance (V1 $\to$ IT) aligns with the need to discard nuisance variation to maintain predictive stability.
  • Metabolic Cost: The high cost of synaptic plasticity supports the "exception tax" hypothesis; frequent reconfiguration is metabolically expensive.
• AI Evidence:
  • Shortcut Learning: DNNs often exploit spurious correlations (e.g., texture bias) that fail under distribution shift (ImageNet-C, ObjectNet), incurring high "regret."
  • Augmentation: Data augmentation acts as an engineered proxy for active sensing, forcing the network to learn shift-stable invariants.
  • Grokking: The phenomenon where networks transition from memorization to generalization is interpreted as a shift from linear regret (patching) to sublinear regret (invariant structure).
  • RLHF: Reinforcement Learning from Human Feedback is framed as narrowing the shift family, potentially leading to preference-stable but shift-fragile representations (analogous to domestication).

5. Significance and Implications

• Physics of Intelligence: The paper posits that intelligence is not a unique property of carbon or silicon but a physical consequence of matter organized to persist under uncertainty. Compression becomes necessary, and shift-stability becomes advantageous.
• Beyond "Black Box" Alignment: It moves the explanation of brain-AI alignment from "coincidence" or "shared data" to a shared constraint channel. This suggests that if we build systems with similar constraints (finite resources + shifting environments), they must converge on similar representations.
• Design Principles for AI:
  • To improve robustness, training must explicitly penalize cross-environment regret (not just in-distribution loss).
  • Active data generation and diverse shift families are critical for discovering causal mechanisms.
  • Overparameterization may delay the "crossover" ($E^*$), allowing shortcuts to persist longer, but does not eliminate the fundamental trade-off.
• Neuroscience: Provides a quantitative framework to measure neural "regret" (via integrated mismatch negativity or metabolic cost of adaptation) and predicts that neural systems should exhibit linear scaling of mismatch costs under shortcut reliance.

Conclusion

The paper argues that the convergence of brains and deep networks is evidence of a substrate-independent basin of attraction shaped by predictive compression under distribution shift. The Information-Theoretic Imperative forces the need for prediction, and the Compression Efficiency Principle dictates that under shifting environments, only representations encoding shift-stable invariants (and, under specific conditions, causal mechanisms) can survive the "exception tax" of linear regret. This framework unifies biological constraints (metabolism) and engineering objectives (loss minimization) into a single physical theory of representational learning.