🤖 machine learning

Learning reshapes power-law anisotropy in internal representations

This paper elucidates the mechanism behind the emergence of power-law anisotropy in neural representations by demonstrating that, unlike in the regime where the spectral exponent remains static, feature learning in linear and nonlinear networks drives a nonmonotonic evolution of this exponent through the dynamic interplay of input statistics and task structure.

Original authors: Asahi Nakamuta, Jun-nosuke Teramae

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Asahi Nakamuta, Jun-nosuke Teramae

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Neural networks, the engines behind modern artificial intelligence, are often described as black boxes that transform raw data into complex decisions. To understand how they work, scientists look inside at the "internal representations"—the high-dimensional patterns the network creates as it processes information. A key way to measure the shape of these patterns is by looking at how the importance of different directions of information is distributed. In many systems, from the brains of mice to the most advanced language models, this distribution follows a specific rule: a few directions hold a massive amount of information, while the rest hold progressively smaller amounts, dropping off in a predictable mathematical curve known as a power law. This unevenness, or anisotropy, is thought to be crucial for how efficiently a network learns and generalizes. For a long time, researchers assumed this pattern was simply a static reflection of the data the network was fed, like a fingerprint left behind by the input. However, a new study challenges this view, suggesting that the pattern is not just inherited but actively reshaped by the learning process itself.

A team of researchers at Kyoto University set out to uncover exactly how this reshaping happens. They focused on a simplified model of a neural network—a two-layer system with a wide hidden layer—placed in a "teacher-student" scenario. In this setup, the student network tries to learn a task defined by a teacher, using data that already possesses a power-law structure. The researchers wanted to know: does the student simply copy the teacher's pattern, or does the act of learning alter the geometry of the internal representation? By mathematically solving the equations that govern how the network changes over time, they discovered that the answer depends entirely on how the network is trained. They found that the network does not settle into a single, fixed pattern. Instead, the shape of its internal information evolves dynamically, passing through distinct phases as it learns.

The study reveals that the evolution of these patterns is not a straight line but a journey through different regimes. When the network is trained in a "feature-learning" mode, where it actively reorganizes its internal structure to solve the task, the pattern of information undergoes a dramatic transformation. Initially, the network's internal state mirrors the input data. But as learning begins, the pattern shifts. The researchers identified that the network passes through up to four different stages of organization. In the earliest phase of learning, the pattern becomes even more skewed, with information concentrating even more sharply in a few directions. As learning continues, this steepness relaxes, and the pattern settles into a new, stable shape that is determined by the specific task the teacher is asking the network to perform. This new shape is a blend of the original data structure and the demands of the task, resulting in a final pattern that is different from both the raw input and the initial state.

Crucially, the researchers showed that this dynamic reshaping only happens when the network is allowed to learn features deeply. If the network is trained in a regime where it barely changes its internal structure and relies mostly on its initial random setup, the pattern remains frozen. In this state, the internal representation stays almost exactly as it was at the beginning, merely reflecting the input data without any significant reorganization. This distinction is vital because it proves that the power-law patterns seen in real neural networks are not inevitable consequences of the data alone. Instead, they are the result of a specific, active interaction between the statistics of the input and the specific goals of the learning task.

To ensure these findings were not just a quirk of their simplified mathematical model, the team tested their theory on more complex, nonlinear networks that behave more like real-world artificial intelligence. The results held true: even in these more realistic systems, the internal representations shifted their shape during training in the feature-learning regime, while remaining static in the regime where the network relies mostly on its initial random setup. The study suggests that the power-law anisotropy observed in biological and artificial brains is a fluid property, constantly being rewritten by the learning process. It is not a fixed trait of the data, but a dynamic signature of how a system adapts to its environment. This work provides a clear, analytical explanation for how the microscopic geometry of a network's internal state is forged, offering a new way to understand the relationship between the data a system sees and the intelligence it develops.

Technical Summary: Learning Reshapes Power-Law Anisotropy in Internal Representations

Problem Statement
Power-law anisotropy in the eigenspectra of internal representations is a ubiquitous geometric property observed in diverse neural systems, ranging from the mouse cerebral cortex to large language models. This anisotropy, where eigenvalues decay as a power law with rank, is linked to coding efficiency, robustness, and generalization. However, existing theoretical frameworks largely treat the power-law exponent as a fixed quantity inherited from input statistics or assume it remains static during training. Consequently, the mechanism by which task-driven learning reshapes these spectra, and how the exponent evolves dynamically in response to input structure and teacher signals, remains unclear. Specifically, it is unknown whether internal representations merely inherit input power laws or undergo non-trivial deformation, and how such changes depend on the learning regime (feature-learning vs. linearized).

Methodology
To address these questions, the authors analyze a wide two-layer linear neural network in a teacher–student setting. The model is defined with input dimension $D$ , hidden width $N$ , and output dimension $P$ , where $P=D \ll N$ and $N \to \infty$ .

Input and Teacher Structure: Both the input covariance matrix and the teacher Gram matrix are assumed to be diagonal in the same orthonormal basis, with eigenvalues decaying as power laws with exponents $\alpha$ (input) and $\beta$ (teacher).
Dynamics: The network is trained via population gradient flow on mean-squared error. While the input-output map is linear, the joint training of weights induces nonlinear dynamics in the internal representation.
Analytical Approach: The authors derive the exact temporal evolution of the eigenvalues of the internal-representation kernel (Gram matrix of first-layer weights) in the infinite-data and infinite-width limits. They decompose the spectrum into an initialization component (inherited from input) and a teacher-derived component (formed through learning).
Regime Analysis: The study distinguishes between the feature-learning regime (where weights change significantly) and the linearized regime (where weights remain close to initialization), controlled by the parameter $\gamma_0$ .
Validation: Theoretical predictions are validated against numerical simulations of finite-width linear networks and nonlinear networks with nonlinear activation functions.

Key Contributions and Results
The paper provides an exact analytical characterization of how the local power-law exponent of internal representations evolves during training.

Spectral Decomposition: The internal representation spectrum $\rho_k(t)$ is shown to consist of two additive components: one derived from the input covariance (present at initialization) and one induced by the teacher signal through learning. The relative dominance of these components determines the spectral shape.
Four Asymptotic Exponent Regimes: In the feature-learning regime, the local effective power-law exponent $\alpha_{\text{eff}}(k, t)$ is not static. As training progresses, the spectrum exhibits up to four distinct asymptotic regimes depending on the mode index $k$ and training time $t$ :
- Region I (Initialization-dominated): $\alpha_{\text{eff}} \approx \alpha$ . Occurs when the learning-induced change is negligible compared to initialization.
- Region II (Early learning, first-layer dominant): $\alpha_{\text{eff}} \approx 3\alpha + \beta$ . Occurs when learning has begun but the mode is not yet fully learned, and the first layer dominates the update.
- Region IIIa (Learned, first-layer dominant): $\alpha_{\text{eff}} \approx \alpha + \beta$ . Occurs when the mode is learned, and the first layer carries the primary learning signal.
- Region IIIb (Learned, both layers): $\alpha_{\text{eff}} \approx \alpha + \beta/2$ . Occurs when the mode is learned and learning is distributed across both layers.
The transition between these regimes is non-monotonic; for a fixed mode, the exponent may first steepen (transiently) and then decrease as the teacher signal is incorporated.
Linearized Regime Stability: In the linearized regime, the learning-induced component never exceeds the initialization component. Consequently, the local exponent remains essentially unchanged ( $\alpha_{\text{eff}} \approx \alpha$ ) throughout training, indicating that the internal representation does not significantly reshape the input anisotropy.
Generalizability to Nonlinear Networks: Numerical experiments with nonlinear networks demonstrate that the predicted spectral transitions and the distinction between feature-learning and linearized regimes persist beyond the linear model. In nonlinear feature-learning settings, the spectrum similarly transitions from $\alpha$ to $\alpha + \beta$ over finite mode ranges.

Significance
The paper claims to provide a general mechanism explaining the formation of power-law internal representations. The results suggest that these representations are not merely static reflections of input statistics but are dynamically reshaped by the interaction between input anisotropy, the anisotropy of the teacher signal, and mode-dependent learning timescales.

By demonstrating that the power-law exponent can take multiple asymptotic values and evolve non-monotonically, the work challenges the assumption of a fixed exponent in theoretical analyses. It establishes that the geometric properties of internal representations are a consequence of the specific interplay between data structure and task-driven learning dynamics. This analytical framework offers a starting point for understanding how microscopic spectral geometry relates to macroscopic performance and scaling laws in neural networks.

Technical Summary: Learning Reshapes Power-Law Anisotropy in Internal Representations

More like this