Technical Summary: Exact Solutions to the Dynamics of Abstraction

Problem Statement

In artificial and biological neural networks, high-level concepts are frequently observed to behave as approximately linear directions within representation space. This phenomenon, formalized as the Linear Representation Hypothesis (LRH), underpins many interpretability and control methods, such as linear probes for concept detection and activation steering. While prior theoretical work has established that abstract representations (where concept vectors are perfectly aligned across contexts) can emerge at the global minima of the loss landscape, the dynamics of how these representations emerge during training remain poorly understood. Empirical observations suggest that abstraction often exhibits non-trivial, non-monotonic trajectories and frequently fails to reach the perfect abstraction ($\alpha=1$) predicted by simple asymptotic theories.

This paper addresses the central questions: What is the trajectory of abstraction during training? How is it influenced by dataset geometry, network depth, and initialization? How do nonlinearities affect these dynamics?

Methodology

The authors develop a dynamical framework to study the alignment of concept vectors, termed "abstraction," using a minimal, analytically solvable setting.

1. Minimal Linear Network Model:

   • Setup: A two-layer linear network trained on a dataset with two binary latent variables ("shape" and "color"), creating four classes.
   • Abstraction Score ($\alpha$): Defined as the cosine similarity between concept vectors in different contexts (e.g., the vector for "square minus circle" in the red context vs. the blue context). $\alpha=1$ implies perfect linearity.
   • Assumptions:
     • Variable-Projected Readout: The readout weights ($W_r$) are assumed to be at their ridge-optimum at all times, effectively regularizing only the feature weights. This simplifies the dynamics to a function of the feature kernel $Q = Z^\top Z$.
     • Two-Factor Symmetry (2FS): The input ($\Sigma_x$), target ($\Sigma_y$), and feature kernels are assumed to possess a specific symmetry structure invariant under permutations within classes and global relabeling of latents. This allows the kernel dynamics to be diagonalized into five eigenmodes: Global mean ($G$), Shape ($S$), Color ($C$), Shape-Color interaction ($SC$), and Residual ($I$).
   • Dynamics: The authors derive exact Ordinary Differential Equations (ODEs) for the eigenvalues of the kernel $Q$ under gradient flow. The abstraction score is expressed as a function of the ratio of the $S$ and $SC$ eigenvalues (inverse Signal-to-Noise Ratio).

2. Extension to Nonlinear Networks:

   • The theory is extended to infinite-width two-layer nonlinear networks using Neural Tangent Kernel (NTK) / Neural Network Gaussian Process (NNGP) limits.
   • Two nonlinearities are analyzed: the error function ($\text{erf}$) and Leaky ReLU.
   • The authors derive the kernel maps that transform the preactivation kernel $Q$ to the feature kernel $K$ and analyze how these maps affect the abstraction score instantaneously and dynamically.

3. Empirical Validation:

   • Experiments are conducted on synthetic data and real-world datasets (3dshapes) using ResNets and Transformers (DINOv3, Gemma 4).
   • A "local GELU ablation" procedure is applied to transformer models to test theoretical predictions regarding nonlinearities.

Key Contributions and Results

1. Exact Solutions for Linear Network Dynamics

The paper provides exact implicit solutions for the trajectory of abstraction $\alpha(t)$ during training.

• Terminal Abstraction Law: The final abstraction $\alpha_\infty$ is determined solely by the geometric mean of the inverse Signal-to-Noise Ratios (SNR) of the input and target kernels:
  $$\alpha_\infty = \frac{1 - \sqrt{\nu(\Sigma_x)\nu(\Sigma_y)}}{1 + \sqrt{\nu(\Sigma_x)\nu(\Sigma_y)}}$$
  where $\nu = \lambda_{SC}/\lambda_S$. Perfect abstraction ($\alpha_\infty=1$) is achieved if either inputs or targets are noiseless.
• Initialization Scale Controls Maximum Abstraction: While the terminal value is independent of initialization, the maximum abstraction reached during training depends critically on the initialization scale ($\kappa$).
  • In the rich limit (small initialization), the network can achieve near-perfect abstraction ($\alpha \approx 1$) for a parametrically long time (metastability) before decaying to the terminal value.
  • In the lazy limit (large initialization), abstraction never exceeds its terminal value.
• Non-monotonicity: In signal-dominant regimes, abstraction can overshoot its terminal value, reaching a peak before converging.

2. Depth and Layerwise Interpolation

Extending the model to deep linear networks with $L$ layers under a "layerwise balancing" assumption:

• Layerwise Interpolation: The abstraction at layer $\ell$ linearly interpolates between the input abstraction and the final layer abstraction in arctanh space:
  $$\text{arctanh}(\alpha^{(\ell)}) = \left(1 - \frac{\ell}{L}\right)\text{arctanh}(\alpha_{\Sigma_x}) + \frac{\ell}{L}\text{arctanh}(\alpha^{(L)})$$
• Depth Improves Abstraction: If the target geometry is more abstract than the input geometry, increasing network depth monotonically improves the terminal abstraction of the final layer.

3. The Attenuation Law in Nonlinear Networks

A striking theoretical result for infinite-width nonlinear networks:

• Attenuation Law: Nonlinearities (specifically $\text{erf}$ and Leaky ReLU) weaken abstraction in the feature space ($H$) relative to the preactivation space ($Z$). Formally, $|\alpha_K| \leq |\alpha_Q|$.
• Differential Effects:
  • Erf Networks: The dynamics closely approximate the linear theory.
  • ReLU Networks: The dynamics depend less on target geometry and more on input geometry. For pure ReLU, even with noiseless targets, perfect abstraction is not guaranteed if the input noise is high; the system can converge to "anti-abstraction" ($\alpha \approx -1$).

4. Empirical Applications

• Transformers (DINOv3, Gemma 4): Applying the theory, the authors show that ablating nonlinearities (replacing GELU with identity locally) in transformer blocks increases abstraction and improves the generalization of linear probes. This validates the attenuation law in finite-width, deep models.
• Neuroscience (Macaque Ventral Stream): Analyzing recordings from V4 and IT regions, the study finds that abstraction increases along the ventral visual hierarchy, consistent with the theoretical prediction that deeper layers (closer to behavioral outputs) should exhibit greater abstraction.

Significance and Claims

The paper claims to provide the first dynamical theory of abstraction that captures the full trajectory of learning, rather than just the asymptotic endpoint.

• Bridging Theory and Practice: It explains why abstraction often fails to reach perfection in real-world settings (due to input/target noise and initialization scale) and why it exhibits non-monotonic behavior.
• Interpretability and Control: The results suggest that the quality of linear probes and steering vectors is fundamentally linked to the abstraction dynamics. The finding that nonlinearities attenuate abstraction offers a theoretical basis for why ablating them can improve probe generalization.
• Neuroscience Implications: The theory provides a mechanistic explanation for the emergence of abstract representations in the brain, predicting that deeper cortical hierarchies should exhibit higher abstraction scores, a prediction confirmed by macaque data.

The authors emphasize that while their exact solutions rely on simplifying assumptions (linear networks, 2FS symmetry, infinite width), the qualitative predictions (overshoot, initialization dependence, depth improvement, and attenuation by nonlinearities) hold robustly in realistic, finite-width deep learning settings and biological data.