🤖 machine learning

The Role of Symmetry in Optimizing Overparameterized Networks

This paper demonstrates that overparameterization in neural networks optimizes training by introducing weight-space symmetries that act as diagonal preconditioning to improve Hessian conditioning and increase the probability mass of global minima near typical initializations, thereby accelerating convergence.

Original authors: Kusha Sareen, Mohammad Pedramfar, Sékou-Oumar Kaba, Mehran Shakerinava, Siamak Ravanbakhsh

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kusha Sareen, Mohammad Pedramfar, Sékou-Oumar Kaba, Mehran Shakerinava, Siamak Ravanbakhsh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex puzzle, like fitting a specific shape into a jigsaw board. In the world of deep learning, the "puzzle" is finding the perfect set of numbers (weights) inside a neural network that makes it solve a task correctly.

For a long time, researchers noticed a strange phenomenon: making the network bigger (adding more "pieces" or parameters) actually makes it easier and faster to solve the puzzle, even if the puzzle itself is small enough to be solved by a tiny network. This is called "overparameterization."

This paper asks: Why does having extra pieces help? The authors argue it's not just about having more raw power; it's about symmetry. They use the idea of "symmetry" to explain two main ways that making a network bigger improves the search for the solution.

Here is the breakdown using simple analogies:

1. The "Mirror Maze" of Solutions (Symmetry)

First, imagine that many different arrangements of puzzle pieces result in the exact same picture. In a neural network, you can swap two neurons, or split one neuron into two smaller ones that work together, and the network still does the exact same job.

The authors call these groups of "different settings that do the same thing" symmetry orbits. Think of it like a flat valley in a mountain range. If you are at the bottom of the valley, you can walk in any direction along the floor, and your altitude (the error) doesn't change. These are the "flat directions."

2. Mechanism One: The "Magic Rescaling" (Diagonal Preconditioning)

When you make a network wider, you introduce new ways to split and rearrange these neurons. The authors prove that these new arrangements act like a magic rescaling tool for the landscape.

The Problem: Imagine the valley floor is shaped like a long, skinny ellipse. If you are trying to roll a ball to the bottom, it's easy to get stuck or bounce back and forth because the path is narrow in one direction and wide in another. This is a "badly conditioned" landscape.
The Symmetry Fix: By using the new symmetries available in a wider network, you can effectively "squash" the long side and "stretch" the short side of that ellipse. You are reshaping the valley into a perfect circle.
The Result: Now, when you roll the ball (run the optimization algorithm), it rolls straight to the bottom without wobbling. The authors call this diagonal preconditioning. It's like putting on special glasses that make the path look perfectly round, even if it was actually oval.

3. Mechanism Two: The "Bigger Target" (Volume Growth)

The second way overparameterization helps is by making the "good" solutions much easier to find by chance.

The Problem: Imagine you are throwing darts blindfolded at a wall. If the "winning" spot is a tiny dot, you will almost never hit it.
The Symmetry Fix: When you add more neurons, the symmetry orbit expands. It's as if the tiny dot on the wall suddenly grew into a large, fluffy cloud. Because there are so many different ways to arrange the extra neurons to get the same result, the total "volume" of winning spots becomes huge.
The Result: Even if you start with a random guess (a random throw), you are now much more likely to land inside that big cloud of good solutions. The paper shows that as the network gets wider, the probability of starting your journey right next to a perfect solution increases significantly.

4. The "Splitting" Trick

The paper details specific mathematical tricks called splitting symmetries.

Post-activation splitting: Imagine you have a worker (a neuron) who is doing a job. You can hire two new workers who each do half the job, and they split the paycheck. The total work done is the same, but now you have more flexibility in how you pay them.
Pre-activation splitting (for ReLU networks): This is like splitting the input to the worker. If the worker is a "ReLU" (a specific type of switch), you can split the incoming signal into two smaller signals that add up to the original.

These splits create new "flat directions" in the landscape. The authors show that while the total "volume" of the solution stays roughly the same (if you split evenly), the shape of that volume changes to be much rounder and easier to navigate.

5. What the Experiments Show

The authors didn't just do math; they ran experiments to prove this:

Curvature: As they made networks wider, the "Hessian" (a mathematical map of the landscape's bumps and valleys) became better conditioned. The "bumps" became smoother and more uniform.
Speed: The networks with more parameters converged (found the solution) faster.
Universality: This worked for different types of networks (MLPs, CNNs, Transformers) and different tasks, suggesting this is a fundamental geometric rule of how these networks work.

Summary

In simple terms, the paper argues that making a neural network bigger doesn't just give it more muscle; it gives it more "degrees of freedom" to reshape the terrain it walks on.

By adding extra parameters, you create a landscape where the "good" solutions are:

Rounder and easier to roll down (better conditioning/preconditioning).
Larger and easier to hit by accident (increased volume/probability).

The "symmetry" of the network is the tool that allows us to reshape the landscape into a friendlier form, making the optimization process much more efficient.

1. Problem Statement

Overparameterization (using neural networks with significantly more parameters than necessary to represent the target function) is a cornerstone of modern deep learning, consistently improving optimization speed and final performance. However, the precise mechanisms driving these improvements remain incompletely understood.

Existing theories often rely on:

Large-width limits: Viewing networks as linearized kernels (NTK).
Curvature analysis: Observing structured Hessian spectra with large near-zero subspaces.
Symmetry: Recognizing that networks possess weight-space symmetries (permutations, rescalings) creating manifolds of functionally identical solutions.

The Gap: While symmetry is known to create flat directions in the loss landscape, its specific role in improving the conditioning of the optimization landscape (making minima easier to reach and converge to) via overparameterization has not been comprehensively characterized. This paper aims to bridge the gap between overparameterization, weight-space symmetries, and the geometric properties of the loss landscape.

2. Methodology

The authors propose a unified framework analyzing weight-space symmetries induced by increasing network width. They utilize a Matrix Groupoid formalism to describe how symmetries operate across different network widths, moving beyond fixed-width symmetries.

Key Theoretical Tools:

Symmetry Orbits: Defining equivalence classes of parameters $[\theta]_T$ that produce the same function.
Splitting Operations: Characterizing how adding neurons (overparameterization) corresponds to "splitting" existing neurons via duplication matrices ( $M$ $M$ ) and pseudoinverses ( $M^+$ $M^{+}$ ).
- Post-activation splitting: Duplicating neurons and redistributing output weights.
- Pre-activation splitting (ReLU specific): Splitting input weights while preserving the ReLU linearity.
Hessian Analysis: Analyzing how the Hessian matrix ( $H$ ) transforms under these splitting operations to understand changes in curvature, condition numbers, and local volume.
Teacher-Student Setting: Using a setup where a "student" network (width $n$ ) is trained to match a "teacher" network (minimal width $n_f$ ) to guarantee the existence of global minima and isolate optimization dynamics from representation issues.

3. Key Contributions

The paper identifies two distinct mechanisms by which overparameterization aids optimization:

Mechanism 1: Diagonal Preconditioning of the Hessian

The authors prove that symmetry transformations act as diagonal preconditioners on the Hessian.

Theorem: If a network is expanded via symmetry, the new Hessian $H'$ has the same non-zero eigenvalues as $K^{1/2} H K^{1/2}$ , where $K$ is a diagonal matrix determined by the splitting structure.
Implication: This transformation effectively rescales the axes of the loss landscape. If the eigenvectors of the original Hessian are approximately axis-aligned (which empirical evidence suggests they often are), this rescaling significantly improves the condition number ( $\kappa$ ), leading to faster convergence for first-order methods.
Trace Reduction: They derive a formula showing that while the product of non-zero eigenvalues (pseudo-determinant/volume) is preserved under even splits, the trace (sum of eigenvalues/average curvature) can be reduced. This leads to improved average-case conditioning.

Mechanism 2: Volume Growth and Reachability

Overparameterization increases the probability mass of global minima near typical random initializations.

Theorem: As width increases, the total volume of global minima near the initialization grows. Specifically, the probability of initializing within an $\epsilon$ -neighborhood of a global minimum increases with width once the network is sufficiently wide.
Combinatorial Effect: Adding parameters introduces new symmetries that expand the equivalence class of solutions. Even if individual minima shrink slightly in volume due to splitting, the combinatorial explosion of possible split configurations (permutation copies) results in a net increase in the probability of landing in a "good" basin.

4. Experimental Results

The authors validate their theoretical predictions through extensive experiments:

Theory Validation:
- Hessian Interlacing: Confirmed that eigenvalues of the student Hessian interlace with the teacher's, with additional near-zero eigenvalues corresponding to the new flat symmetry directions.
- Pseudo-determinant & Trace: Empirical measurements of the Hessian trace and pseudo-determinant matched theoretical predictions within 0.03% error.
- Neuron Clustering: Visualized that at convergence, student neurons cluster around teacher neuron weights, with output weights summing to the teacher's values, confirming the "splitting" structure.
Optimization Metrics:
- Conditioning: Across MLPs, CNNs, and Transformers, the condition number, top eigenvalue, and AM/GM ratio (a measure of average-case conditioning) improved monotonically as width increased.
- Convergence Speed: The number of steps required to converge decreased as width increased.
- Generalization: These trends held true not only in teacher-student settings but also in direct data fitting scenarios (e.g., California Housing regression, CIFAR-10 classification), even in mildly underparameterized regimes.

5. Significance and Implications

Unified Framework: The paper provides a geometric explanation for overparameterization that unifies symmetry, curvature, and initialization probability. It moves beyond the "lazy training" (NTK) regime to explain finite-width dynamics.
Why Overparameterization Works: It suggests that overparameterization does not just provide more capacity; it fundamentally reshapes the loss landscape to contain better-conditioned minima that are more abundant and easier to reach from random initialization.
Generalization: The authors propose a "counting argument" for generalization: simpler functions (represented by the teacher) have larger symmetry orbits and thus larger volumes of minima compared to complex overfitting solutions. Optimization naturally favors these larger, better-conditioned basins.
Practical Insight: The findings support the use of wider networks and explain why adaptive optimizers (like Adam) work well—they implicitly perform diagonal preconditioning, which synergizes with the symmetry-induced preconditioning of overparameterized networks.
Plasticity Loss: The framework offers a potential explanation for "plasticity loss" in continual learning: optimization may get trapped in a specific symmetry orbit (a low-width diagonal in the groupoid), preventing the network from utilizing dormant width for new tasks.

In summary, the paper demonstrates that overparameterization is a geometric transformation that improves optimization by creating a landscape with more accessible, better-conditioned minima through the expansion of weight-space symmetries.