🔢 mathematics

Informative Fisher Modes in Neural Networks as a Result of Explicit Symmetry Breaking

This paper identifies task-dependent contingent symmetries and their explicit breaking as a fundamental organizing principle in neural network learning, demonstrating how this mechanism generates informative, non-flat Hessian modes and drives symmetry-breaking transitions like BBP, thereby offering a measurable route to study symmetry breaking in deep learning systems.

Original author: Ibrahim Talha Ersoy, Björn Ladewig, Karoline Wiesner

Published 2026-11-03

📖 8 min read🧠 Deep dive

CC BY 4.0

Original author: Ibrahim Talha Ersoy, Björn Ladewig, Karoline Wiesner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the author. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of modern science, the study of how machines learn sits at a crossroads between mathematics, statistics, and physics. At its heart lies a simple but profound challenge: understanding how a computer, starting with random guesses, gradually figures out the rules of a task. Imagine a student trying to solve a complex puzzle; the path they take is not a straight line but a winding journey across a rugged terrain of possibilities, where some paths lead to dead ends and others to the solution. Scientists have long known that this terrain is shaped by the structure of the data and the design of the machine itself. A key concept in this field is symmetry, which in this context refers to the many different ways a machine can be arranged internally while still producing the exact same output. Just as a cube can be rotated in space without changing its shape, a neural network can often be reconfigured in countless ways without changing what it does. These symmetries create "flat" directions in the learning landscape, where the machine can move without improving or worsening its performance. For years, researchers have wondered if these symmetries were the only organizing principle at work, or if there were hidden rules governing the specific directions where real learning happens.

A team of researchers at the University of Potsdam has now uncovered a new layer of organization that explains how machines learn specific features of a task. They discovered that learning is not just about the machine's internal design, but about how the task itself restricts the machine's freedom. When a machine is asked to learn a specific part of a problem, it loses the ability to change certain internal settings without affecting the result. The researchers found that this loss of freedom is the key to understanding how learning directions emerge. They showed that when a machine is forced to satisfy a specific requirement, a previously flat and unimportant direction in its internal structure suddenly becomes "massive" or stiff. This stiffness is a sign that the machine has begun to learn that specific feature. Before this constraint is applied, the machine could change that setting freely; after, any change would ruin the solution. This process acts like a filter, revealing which parts of the machine's internal structure are actually carrying information about the task.

To prove this, the scientists built a simple, solvable model of a learning machine with just a few adjustable knobs. They started with a task that was only partially defined, leaving the machine with many ways to be correct. In this state, the machine had a lot of "contingent symmetry," meaning it could change its internal settings in many ways without changing the outcome of the partial task. The researchers then added a second requirement to the task. As soon as this new requirement was imposed, the symmetry broke. The machine could no longer change its internal settings freely; one specific direction became rigid. This rigidity was not random; it corresponded exactly to the new piece of information the machine was now required to learn. The researchers demonstrated that this breaking of symmetry creates a measurable signal: the machine's internal curvature changes, and it gains the ability to distinguish between different settings based on the data. This finding suggests that learning is a step-by-step process where each new piece of information removes a specific freedom, turning a flat, uninformative direction into a steep, informative one.

The team also explored how this process plays out in more complex, deep networks and how it can be detected in real-world scenarios. They found that by adjusting a standard control parameter used in training—essentially a penalty for having large internal values—they could force the machine to learn features in a specific order. As they relaxed this penalty, the machine would learn the strongest features first, then the next strongest, and so on. This created a cascade of transitions, where the machine moved from a state of knowing nothing about the task to a state of knowing everything, passing through distinct phases where it knew only a subset of the features. They also showed that adding noise to the data, similar to static on a radio, could delay this learning process. The machine would only learn a feature once the signal from that feature became strong enough to rise above the noise. This allowed them to map out a "phase diagram" of learning, showing exactly when and how the machine acquires different pieces of knowledge based on the strength of the data and the strictness of the training rules.

What makes this discovery significant is that it provides a way to see the learning process without needing to know the exact mathematical symmetries of the machine. The researchers showed that these changes in learning directions leave clear signatures in the machine's behavior that can be measured directly. They do not need to know the hidden group of symmetries to see that a feature has been learned; they only need to observe how the machine's internal structure stiffens in response to the data. This approach offers a new tool for studying how neural networks learn, moving beyond the idea that learning is just a chaotic search through a high-dimensional space. Instead, it reveals a structured hierarchy where the task itself dictates the order in which the machine acquires knowledge. The work suggests that the path to learning is paved by the breaking of these contingent symmetries, where each new constraint transforms a free, unimportant direction into a critical, information-carrying one.

The researchers also clarified what does not happen during this process. They showed that not all changes in the machine's internal structure are due to learning. Some directions become stiff simply because of the training rules, not because the machine has learned anything new about the data. These are "gauge" directions, which are related to the machine's internal redundancy rather than the task itself. The study carefully distinguished between these two types of stiffness, ensuring that the observed changes were truly linked to the information being learned. By separating the effects of the task from the effects of the training method, the researchers provided a clearer picture of what is actually happening inside the machine. They found that while the machine can be reconfigured in many ways without changing its output, the moment it learns a specific feature, that freedom is lost, and the machine becomes locked into a specific configuration for that feature.

This understanding has implications for how we think about the generalization of learning. The study suggests that the ability of a machine to generalize to new data is tied to the specific way it has broken its symmetries to learn the task. The researchers did not claim to have solved all the mysteries of deep learning, but they have provided a concrete mechanism for how learning directions emerge. They showed that the process is not a single, sudden event but a sequence of transitions, each corresponding to the acquisition of a new feature. This sequence is governed by the interplay between the task requirements and the machine's internal structure. The findings were demonstrated analytically in simple models and numerically in deeper networks, showing that the principle holds across different levels of complexity.

The work also highlights the role of noise in the learning process. The researchers showed that noise can act as a barrier, preventing the machine from learning a feature until the signal is strong enough to overcome it. This is similar to how a radio needs a strong signal to tune out static, but in this case, the "signal" is the information about the task, and the "static" is the random fluctuations in the data. The researchers used this insight to create a map of learning, showing how the machine's ability to learn changes as the noise level and the training penalty vary. This map reveals regions where the machine learns nothing, regions where it learns only the strongest features, and regions where it learns everything. The boundaries between these regions are sharp and predictable, suggesting that the learning process is more orderly than previously thought.

In the end, the study offers a new perspective on the geometry of learning. It moves away from the idea of a static landscape and toward a dynamic process where the landscape itself changes as the machine learns. The breaking of symmetry is the mechanism that drives this change, transforming flat directions into steep ones and revealing the path to the solution. The researchers have shown that this process is measurable and predictable, providing a way to track the learning of individual features. While the study focused on linear networks and simple models, the principles are expected to apply to more complex, nonlinear systems as well. The key takeaway is that learning is a process of constraint, where the task itself shapes the machine's internal structure, removing freedoms and creating the specific directions needed to solve the problem. This insight brings us closer to understanding how machines learn, not just as a black box, but as a system governed by clear, observable principles.

Technical Summary: Informative Fisher Modes in Neural Networks as a Result of Explicit Symmetry Breaking

Problem Statement
Understanding the learning dynamics of neural networks requires analyzing the geometry of high-dimensional, non-convex loss landscapes. While architectural symmetries (reparametrizations that leave the realized function unchanged) are known to create flat directions (near-zero Hessian modes) in the loss landscape, it remains an open question whether additional organizing principles govern the remaining degrees of freedom. Specifically, how do task constraints restrict model freedom, and when does the resulting curvature in the loss landscape correspond to statistical information (Fisher information)? The paper investigates how imposing task-specific features removes "contingent" freedoms—degrees of freedom preserved by partial tasks but broken by full task specifications—and how this breaking generates informative learning directions.

Methodology and Conceptual Framework
The authors develop a framework distinguishing three types of symmetries:

Architectural Symmetries: Reparametrizations preserving the realized function $f_\theta(x)$ regardless of the task (e.g., weight scaling in linear networks).
Task/Data Symmetries: Invariances of the loss function due to data distribution properties (e.g., label invariance).
Contingent Symmetries: Transformations that preserve the features already imposed by a partial task but allow other aspects of the function to change. These symmetries are "contingent" because they are broken when additional feature constraints are added.

The core analytical tool is Feature Filtration. The authors construct a sequence of partial tasks, $M_k$ , where $k$ features are constrained to their target values. This creates a nested hierarchy of solution sets ( $M_0 \supseteq M_1 \supseteq \dots \supseteq M_r$ ). At each step, adding a feature constraint explicitly reduces the contingent symmetry group $G_k \to G_{k+1}$ .

The analysis connects this symmetry reduction to the Hessian (curvature of the error) and Fisher Information (statistical distinguishability). The authors posit that at a regular exact solution, imposing an independent feature constraint removes one degree of freedom from the contingent group, turning a previously flat direction into a "massive" (positive curvature) direction in both the Hessian and the Fisher metric.

Key Contributions

Distinction of Symmetries: The paper formally distinguishes contingent symmetries (freedoms left open by partial tasks) from architectural symmetries and task invariances. It introduces feature filtration to isolate these freedoms analytically.
Rank Increase Mechanism: It is shown that at a common regular exact solution, imposing one independent feature constraint increases the rank of both the error Hessian and the Fisher information matrix by exactly one. This identifies the newly constrained parameter variation as an "emergent learning direction."
Role of Regularization ( $L_2$ Penalty): The authors demonstrate that an $L_2$ penalty can lift architectural directions (making them massive in the Hessian) without making them Fisher-positive. These "penalty-lifted" directions correspond to reparametrization (gauge) modes, not learning directions. In contrast, directions associated with learning a feature become Fisher-positive.
Phase Diagram of Learnability: By combining $L_2$ regularization strength ( $\beta$ ) with data noise amplitude ( $T$ ), the authors construct a phase diagram. This diagram maps regions where specific subsets of features are learned. It reveals that regularization and spectral recovery (via BBP transitions) can order features differently if penalty weights are unequal, leading to competing phase boundaries.
Extension to Deep Linear Networks: The analysis is extended to deep linear networks, where the contingent group is identified as a parabolic subgroup. The Hessian spectrum at a balanced critical point is decomposed into four sectors: flat (architectural), onset (unlearned features), massive (learned features), and pseudo-Goldstone (penalty-lifted architectural modes).

Results

Minimal Model: In a solvable two-feature bilinear model, the authors show that imposing the first feature breaks a subset of the symmetry group, creating one massive Hessian/Fisher mode. Imposing the second feature breaks the remaining contingent symmetry, creating a second massive mode.
Symmetry Breaking and Fisher Information: The transition from a flat to a massive direction coincides with the direction becoming statistically distinguishable (Fisher-positive). This occurs when the parameter variation changes the observation distribution to the first order.
Regularization Cascade: In deep linear networks with distinct feature strengths, lowering the $L_2$ penalty $\beta$ triggers a cascade of continuous (1 hidden layer) or discontinuous (2+ layers) transitions. Each transition corresponds to the activation of a new feature, ordered by the feature strength $\eta_i$ .
BBP Transitions: In the presence of noise, feature directions separate from the noise bulk only when the signal-to-noise ratio exceeds a Baik-Ben-Arous-Péché (BBP) threshold. This provides a statistical counterpart to exact feature inclusion.
Phase Diagram: The $\beta-T$ phase diagram shows that for uniform penalties, regularization and noise thresholds order features identically. However, with unequal penalty weights, the ordering can compete, causing phase boundaries to cross.

Significance and Claims
The paper claims to identify task-dependent contingent symmetries as an organizing principle in deep learning, distinct from architectural symmetries. The primary significance lies in linking the geometric structure of the loss landscape (Hessian modes) directly to statistical information (Fisher modes) through the mechanism of explicit symmetry breaking.

The authors argue that:

Learning a feature is fundamentally the breaking of a contingent symmetry that preserves a partial task.
This breaking generates "emergent learning directions" that are both geometrically massive and statistically informative.
Crucially, these signatures (the emergence of massive, Fisher-positive modes) are measurable without requiring an explicit description of the symmetry group. This makes the principle testable in nonlinear networks and other learning systems where symmetry groups are unknown or intractable.
The work unifies concepts from optimization (regularization cascades), statistical inference (Fisher information), and random matrix theory (BBP transitions) to explain how neural networks acquire features in a structured, hierarchical manner.

The paper remains modest regarding nonlinear networks, noting that while the specific group descriptions and closed-form spectra derived for linear networks may not hold, the phenomenology of symmetry breaking and the resulting Hessian/Fisher signatures are expected to persist. The proposed mechanism offers a route to study symmetry breaking as a general principle in learning systems beyond the solvable linear case.

Technical Summary: Informative Fisher Modes in Neural Networks as a Result of Explicit Symmetry Breaking

More like this