🤖 machine learning

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

This paper models Stochastic Gradient Descent dynamics as a percolation process where architectural symmetries drive the formation of simpler subnetworks through discrete, simultaneous block merges, manifesting as variance spikes and scaling cascades that also apply to Adam and AdamW under heavy-tailed noise.

Original authors: Sai Niranjan Ramachandran, Suvrit Sra

Published 2026-09-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sai Niranjan Ramachandran, Suvrit Sra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Deep learning has revolutionized how machines learn, yet the internal journey of a neural network during training remains a black box. We know these systems start with millions of adjustable knobs, or parameters, and through a process called training, they tune these knobs to solve problems. A common method for this tuning is stochastic gradient descent, a technique that nudges the network toward better solutions by looking at small, random slices of data at a time. For years, researchers have observed that this process naturally pushes networks toward simpler, more efficient structures, often discarding unnecessary complexity without being explicitly told to do so. This phenomenon, known as implicit bias, suggests that the training method itself acts as a sculptor, carving away excess material to reveal a core structure. However, the precise mechanics of how this sculpting happens—whether it is a smooth, gradual erosion or a series of sudden, dramatic shifts—have remained unclear. Understanding this process is crucial because it could explain why networks sometimes seem to memorize data perfectly for a long time before suddenly "clicking" and learning to generalize, a behavior that has puzzled scientists for years.

A team of researchers has now mapped this hidden journey, revealing that the collapse of a neural network into a simpler form is not a smooth slide but a series of sudden, synchronized jumps. By treating the training process as a physical system where parts of the network merge together, the authors discovered that these merges happen in discrete blocks rather than one by one. Imagine a large group of people in a room who are slowly finding their way to the same spot; in this new view, they do not arrive individually. Instead, entire groups arrive at the exact same moment, fusing together in a single event. The researchers modeled this behavior using a concept from physics called percolation, which describes how fluids flow through porous materials or how connections form in a network. They found that the architecture of the neural network itself forces these groups to merge simultaneously, creating a pattern of sudden structural changes that ripple through the system.

To uncover this pattern, the researchers developed a mathematical framework that tracks the movement of the network's parameters as they drift and diffuse over time. They focused on how different parts of the network, which start out independent, eventually get trapped in the same simplified state. When these parts merge, they form a larger, unified block. The researchers showed that because of the symmetries built into the network's design, these blocks cannot merge one at a time. Instead, they must merge in groups of two, three, or more, all at once. This creates a "variance cascade," a sequence of spikes in the system's instability that signals these major structural shifts. By measuring the fluctuations in the network's behavior across many different training runs, the team could detect these spikes and see a clear, repeating pattern. The time intervals between these spikes followed a strict geometric rule, where each event happened at a predictable multiple of the previous one. This pattern, known as discrete scale invariance, acts like a fingerprint of the underlying symmetry, proving that the network is collapsing in a highly organized, step-by-step fashion rather than a chaotic mess.

The study went beyond simple models to test these ideas on complex, real-world scenarios, including a famous phenomenon called "grokking." In grokking, a neural network trained on a specific logic puzzle will memorize the training data for thousands of steps, showing no sign of true understanding, before suddenly and dramatically improving its ability to solve new problems. The researchers found that this sudden leap in performance coincides exactly with the final stage of their predicted cascade. Just before the network "clicks" into a generalizing solution, the system undergoes a final, massive topological shift where the remaining complex parts of the network fuse into a simple, low-rank structure. This suggests that the network was not slowly learning the rule but was instead waiting for the right moment to collapse its internal complexity into the correct, simple form. The team also demonstrated that this mechanism holds true for advanced training methods like Adam and AdamW, which are widely used in modern artificial intelligence, provided the noise in the system follows certain statistical patterns.

The findings offer a new way to look at how artificial intelligence learns, shifting the focus from a continuous, smooth optimization to a series of discrete, phase-transition-like events. The researchers showed that these transitions are not random accidents but are driven by the fundamental geometry of the network itself. By tracking the relative variance of the network's parameters, they could predict when these major shifts would occur, seeing the system move through a series of distinct stages before reaching its final, simplified state. In simulations and on various datasets, from simple mathematical puzzles to image recognition tasks, the predicted pattern of sudden merges appeared consistently. The work suggests that the path to intelligence in these machines is paved with sudden, synchronized collapses of complexity, where the network sheds its unnecessary layers in a single, decisive motion. This insight could help researchers better understand the timing of learning in deep networks and potentially guide the design of training algorithms that harness these natural, structural shifts to achieve faster and more reliable results.

Technical Summary: Percolation Dynamics in Optimization

Problem Statement

Deep learning systems often achieve strong generalization without explicit regularization, a phenomenon attributed to the implicit biases introduced by optimizers like Stochastic Gradient Descent (SGD). It is established that SGD steers networks toward invariant sets corresponding to simpler subnetworks. However, the temporal dynamics of how these invariant sets are reached remain poorly understood. This gap limits the mechanistic explanation of anomalous training behaviors, such as "grokking" (delayed generalization where a network memorizes data perfectly before suddenly discovering a generalizing solution after thousands of epochs). The paper posits that the topological structure of these dynamics contains crucial information about optimization progress, specifically questioning whether the collapse into simpler subnetworks occurs smoothly or through discrete, sudden transitions.

Methodology

The authors model the optimization process as a continuous Stochastic Gradient Flow (SGF) using Stochastic Differential Equations (SDEs). They integrate concepts from stochastic dynamics, topological data analysis, and statistical physics to analyze the collapse of neural networks.

Stochastic Gradient Flow and Invariant Sets: The discrete SGD update is approximated by an Itô SDE. The authors define "invariant sets" as regions in parameter space (often affine subspaces generated by architectural symmetries like neuron permutation) that trap trajectories. They prove that if a set is an invariant set for discrete SGD, it remains invariant for the continuous SGF under specific Lipschitz conditions.
Stochastic Attractivity and Trapping: The paper introduces the concept of "stochastic attractivity," where inward deterministic gradient drift overpowers outward stochastic diffusion near invariant sets. This creates a "trapping" mechanism where decoupled subnetworks are forced to synchronize and collapse into shared invariant subspaces.
Topological Dynamics via Reeb Graphs: To track the evolution of these collapses, the authors map the optimization dynamics onto a Reeb graph. In this framework, distinct subnetworks are nodes, and edges represent "Attractor Equivalence" (synchronization). The dynamics are described as a process of condensation (merging nodes) and fragmentation (splitting nodes).
Percolation Model: The authors reframe the merging of subnetworks as a percolation process. Unlike classical Erdős-Rényi percolation where edges attach continuously, architectural symmetries force discrete, simultaneous block-merges (e.g., $n$ subnetworks merging at once).
Discrete Scale Invariance (DSI): Because these merges occur in discrete blocks rather than continuously, the system breaks continuous scale invariance. Instead, it exhibits Generalized Discrete Scale Invariance (DSI), where microtransitions (variance spikes) occur at geometrically scaled intervals leading up to the global phase transition.
Extension to Adaptive Optimizers: The framework is extended to Adam and AdamW. This requires handling heavy-tailed gradient noise (common in attention-based architectures) via truncation and analyzing the joint state of parameters, first moments, and second moments. The authors prove that under heavy-tailed noise models and specific non-degeneracy conditions, the trapping mechanism and DSI cascade persist.

Key Contributions

Percolation Model of Topological Condensation: The paper constructs a framework mapping stochastic gradient flow near invariant sets to a percolation process on a Reeb graph. It demonstrates that architectural symmetries cause discrete, simultaneous block-merges rather than continuous edge attachments. The authors derive the precise conditions under which these discontinuities survive in the thermodynamic limit (large network size).
Variance Divergence and DSI Cascades: The authors show that relative variance across training trajectories can isolate discrete microtransitions. They prove that multi-body block-merges yield Discrete Scale Invariance (DSI), shaping the phase transition into a geometrically scaling cascade. This allows the prediction of the global structural collapse point based on the spacing of preceding variance spikes.
Extension to Adaptive Optimizers: The trapping mechanism and DSI cascade are theoretically extended to Adam and AdamW under an explicit heavy-tailed noise model, providing a theoretical basis for phenomena like grokking in AdamW-trained Transformers.
Empirical Validation: The framework is validated across diverse settings:
- Toy Models: Controlled kinematic simulations confirm the theoretical DSI scaling factor ( $\lambda \approx 2$ ).
- Tabular Data: Experiments on UCI datasets (e.g., Heart Disease) reveal DSI variance cascades, distinguishing true topological constraints from optimization noise via spectral null models.
- Vision and Grokking: In Transformer networks trained on modular arithmetic (the grokking setting), the authors observe a 3-peak DSI cascade immediately preceding the generalization spike, with a scaling factor $\lambda \approx 2.11$ . Similar patterns are observed in vision benchmarks (FashionMNIST).

Results

Theoretical: The paper proves that stochastic attractivity drives trajectories into invariant sets, creating a supermartingale trapping effect. It establishes that symmetry-induced percolation leads to discontinuous jumps in the order parameter (size of the largest connected component), which manifest as divergences in the relative variance of the order parameter across an ensemble of runs.
Scaling Laws: The microtransitions (variance peaks) follow a geometric progression $p_{n} - p_{c} = \lambda^{-1}(p_{1} - p_{c})$ , where $\lambda = n^\sigma$ . In pairwise merges under mean-field assumptions, $\lambda = 2$ .
Empirical:
- In a Transformer grokking experiment, a 3-peak DSI cascade with $\lambda = 2.11$ was observed just before the performance spike, with a False Positive Rate of 0.1% against phase-randomized null models.
- Tabular datasets showed varying scaling factors (e.g., $\lambda = 1.71$ for Heart Disease), which the authors interpret as evidence of fractional spectral variance mass collapsing, suggesting higher-order or partial merges.
- Task shifts in toy models demonstrated "reactive fragmentation," where reducing noise and increasing signal curvature caused previously bound parameters to split, confirming the reversibility of the trapping mechanism.

Significance and Claims

The paper claims to provide a mechanistic explanation for how optimization progresses through topological phase transitions. By modeling SGD as a percolation process, it offers a new lens to understand why networks suddenly generalize (grokking) and how they collapse into sparse, low-rank representations.

The authors assert that the observed "discrete scale invariance" is a fundamental property of optimization dynamics driven by architectural symmetries, distinct from classical continuous phase transitions. They suggest that these variance spikes (DSI cascades) can serve as early indicators of structural collapse and generalization.

The paper remains modest regarding its scope:

It acknowledges that the discontinuity of the phase transition depends on the network size and the scaling of merging components (macroscopic vs. microscopic regimes).
It notes that while the theory is proven for SGD and extended to Adam/AdamW under specific noise assumptions, a systematic study of these dynamics in large-scale practical models remains an open direction for future work.
It frames the DSI cascade as a potential tool for guiding learning rate scheduling or understanding scaling laws, but does not claim to have solved the generalization problem entirely.

In summary, the paper posits that the "journey" of deep learning optimization is not a smooth descent but a series of discrete, symmetry-driven topological reorganizations, detectable through variance cascades that obey discrete scale invariance.