🤖 machine learning

Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias

This paper demonstrates that the practical inductive bias of trained deep networks is determined not just by their architecture but by how training dynamics—specifically learning rate, optimizer choice, and regularization—filter or erase the memory of random initialization, with low-learning-rate SGD preserving this initial bias while adaptive methods and explicit norm control effectively erase it.

Original authors: Mohua Das, Pierfrancesco Beneventano, Shibshankar Dey, Gareth H. McKinkey, Tomaso Poggio

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mohua Das, Pierfrancesco Beneventano, Shibshankar Dey, Gareth H. McKinkey, Tomaso Poggio

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a chef to cook a complex dish. You give them a specific set of ingredients (the initialization) and a recipe (the training pipeline).

This paper asks a simple but profound question: Does the final taste of the dish depend on exactly how the chef chopped the onions at the very beginning, or does the cooking process itself erase those initial differences?

In the world of AI, "chopping onions" is called initialization. It's the random starting point of a neural network before it has learned anything. The "cooking process" is the training, where the network learns from data.

Here is what the paper discovered, broken down into everyday concepts:

1. The "Memory" of the Chef

The researchers introduced a concept called "Initialization Memory." This is a way to measure how much the final AI still "remembers" its random starting point.

High Memory: The final AI performs very differently depending on how it was started. If you start with a slightly different random seed, the AI becomes a totally different (and potentially worse) chef.
Low Memory: The final AI forgets its starting point. No matter how you chop the onions initially, the cooking process smooths everything out, and you get the same great dish.

2. The Two Types of Cooks (Optimizers)

The paper tested different "cooking methods" (optimizers) to see which ones forget and which ones remember.

The Slow, Careful Cook (Low-Learning-Rate SGD):
Imagine a chef who takes tiny, cautious steps. The paper found that this chef remembers everything. Even after cooking for a long time and perfectly memorizing the recipe (training accuracy), the final dish still tastes different depending on how the ingredients were initially arranged.
- The Result: If you start with a "large" random chop, the dish might taste terrible. If you start with a "small" chop, it tastes great. The chef never fully forgets the initial chaos.
The Adaptive, Fast Cook (Adam, AdamW, Muon):
Imagine a chef who adjusts their knife speed and pressure based on the food they are cutting. These methods forget the starting point very quickly.
- The Result: No matter how wildly different the initial chop was, the adaptive chef adjusts their technique so well that the final dish tastes almost exactly the same. They "erase" the memory of the start.

3. Time Doesn't Always Help

A common belief is that if you just let the slow cook (SGD) work longer, they will eventually forget the bad start.

The Paper's Finding: No. Even if you let the slow cook work for 5,000 hours (epochs), they still remember the initial chop. The "bad start" lingers.
The Fix: To make the slow cook forget, you don't just need more time; you need to change the style of cooking. You need to add regularization (like adding a specific spice or constraint, e.g., L2 weight decay) or cook with larger steps (higher learning rate). These changes act like a "reset button" that wipes the slate clean.

4. The "Clock" of Forgetting

The paper argues that we shouldn't measure forgetting by how many hours (epochs) the chef worked. Instead, we should measure it by the total amount of "effort" or "regularization" applied.

Think of it like a bucket of water (the memory of the start).
Slow cooking (low learning rate) has a tiny hole in the bucket. The water (memory) leaks out so slowly that even after a long time, the bucket is still full.
Adaptive cooking or adding weight decay creates a big hole. The water drains out fast, and the bucket is empty (forgotten) quickly.

5. The Big Takeaway

The "bias" or "personality" of an AI isn't just built into its architecture (the pot and pan). It is also shaped by how the training process handles its starting point.

If you use a "forgetful" training recipe (like Adam or SGD with weight decay), the AI becomes robust. It doesn't matter if you start with a lucky or unlucky random seed; it will learn the same good solution.
If you use a "remembering" recipe (like slow SGD without extra help), the AI is fragile. A bad start can ruin the final performance, even if the AI learned the training data perfectly.

In short: The paper proves that good generalization (making good predictions on new data) is tightly linked to the ability of the training process to "forget" its random start. The mechanisms that help an AI generalize well are the exact same mechanisms that make it forget its initialization.

Technical Summary: "Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias"

Problem Statement
Modern neural networks are highly overparameterized, meaning their capacity alone cannot explain generalization. While the training pipeline (architecture, optimizer, regularization, etc.) selects a specific function from the hypothesis class, the role of the initial random weights in determining the final predictor remains ambiguous. Existing literature presents two conflicting views:

Initialization as a Source of Bias: Theoretical work suggests random networks possess an inherent "simplicity bias" (a prior over functions) that may survive training and drive generalization.
Initialization as a Trainability Mechanism: Practical large-scale training often treats initialization merely as a tool to stabilize signal propagation and prevent exploding/vanishing gradients, assuming the final predictor is largely independent of the initial seed once training converges.

The paper investigates a precise subquestion: To what extent does the final predictor selected by a modern training pipeline depend on the scale of the random initialization? Specifically, does the training process "forget" the initial condition, or does the initial bias persist in the final solution?

Methodology
The authors conduct a controlled empirical study using ResNets on the CIFAR-10 dataset. They introduce a diagnostic metric called Initialization Memory, defined as the dependence of the validation-selected predictor on the initialization scale $\sigma_w$ .

Experimental Design: They sweep the global scale $\sigma_w$ of random kernel initialization while holding all other factors (architecture, data split, optimizer, batch size, learning rate schedule, and training horizon) constant.
Metrics:
- Test Accuracy Spread: The difference between the best and worst test accuracy across different $\sigma_w$ values at the best-validation-loss checkpoint.
- Interpolation Epoch ( $\tau_{interp}$ ): The epoch where training accuracy reaches 99.5%.
- Frobenius Norm ( $\|W\|_F$ ): Used to measure "radial memory" (whether the final weight norm remains coupled to the initial norm).
- Repair Gap: The change in validation accuracy between interpolation and the final checkpoint.
Variables: The study varies optimizers (SGD, SGD with momentum, Adam, AdamW, Muon), batch sizes ( $b \in \{16, 32, 64, 128, 256\}$ ), initialization scales ( $\sigma_w \in [0.1, 2.5]$ ), network depths (ResNet-9, 56, 110), and explicit regularization (L2 weight decay).
Theoretical Framework: The authors employ backward error analysis to interpret discrete optimizers as gradient flows on modified objectives. They propose "forgetting timescales" based on the accumulated strength of regularization mechanisms rather than simple epoch counts.

Key Results

Optimizer Dependence:
- SGD Remembers: Under low learning rates, vanilla SGD retains strong initialization memory. On ResNet-9 with batch size $b=128$ , test accuracy varies by 26.5 percentage points across initialization scales, despite achieving $\ge 99.5\%$ training accuracy. The final weight norm remains coupled to the initial norm.
- Adaptive Methods Forget: Adam, AdamW, and Muon largely erase the dependence on initialization scale. Under the same conditions, their test accuracy spread is minimal (e.g., ~4.0 pp for Muon). They converge to a common final norm scale regardless of the initial scale.
Hyperparameter Dependence:
- Training Time is Insufficient: Extending low-learning-rate SGD to 5,000 epochs does not significantly reduce the initialization spread; the memory persists.
- Regularization and Learning Rate: SGD can be made to forget initialization if the training recipe includes larger learning rates or explicit L2 regularization (weight decay). For instance, adding L2 decay with $\eta=10^{-2}$ reduces the spread to near zero.
- Batch Size: Larger batch sizes slow down the forgetting process for SGD, increasing the spread.
Depth and Architecture:
- In deeper networks (ResNet-56, ResNet-110), poor forgetting dynamics manifest differently: instead of just interpolating with poor generalization, the networks often fail to train well (fail to interpolate) at large initialization scales.
Forgetting Timescales:
The authors identify that forgetting is governed by cumulative "clocks" rather than epoch count:
- $T_{SGD} = \frac{1}{b} \sum \eta_k^2$ : The timescale for stochastic finite-step effects (minibatch noise). Low-LR SGD has a tiny $T_{SGD}$ , leading to memory retention.
- $T_{L2} = \lambda \sum \eta_k$ : The timescale for explicit norm decay.
- $T_{adapt} = \sum \eta_k$ : The timescale for adaptive preconditioning.
  Adaptive methods operate on the $T_{adapt}$ scale, which is significantly larger (faster forgetting) than the $T_{SGD}$ scale of low-LR SGD.

Significance and Claims
The paper claims that the "practical inductive bias" of a trained network is not solely the architectural prior present at initialization, but rather the architectural prior filtered by the forgetting dynamics of the training pipeline.

Resolution of the Paradox: The authors resolve the apparent contradiction between theories of simplicity bias and practical training stability. They argue that the mechanisms that improve generalization (e.g., weight decay, adaptive optimizers, larger learning rates) are precisely those that erase the memory of initialization.
Role of Initialization: In regularization-rich, modern training pipelines, initialization serves primarily as a trainability device (stabilizing optimization, enabling depth) rather than a persistent source of inductive bias. The "Occam's razor" that matters in practice is imposed by the accumulated regularizers along the trajectory, not the initial parameter map.
Limitations: The authors acknowledge their findings are based on BatchNorm ResNets on CIFAR-10. The proposed timescale expressions are organizing principles derived from linear and scale-invariant arguments, not universal theorems for nonlinear BatchNorm networks. However, the empirical hierarchy observed (SGD remembers, adaptive methods forget, regularization erases memory) aligns with these theoretical clocks.

In summary, the paper demonstrates that deep networks do not automatically forget initialization; they only forget it when the specific training dynamics (stochasticity, adaptivity, or explicit regularization) provide sufficient "forgetting time" to overwrite the initial geometric biases.