🤖 machine learning

Beyond a Single Explanation of the Adam--SGD Gap

Through a controlled empirical study across diverse domains, this paper demonstrates that the performance gap between Adam and SGD arises from complex data-architecture interactions rather than a single factor, but is consistently characterized by a theoretical "crossover batch size" where the relative advantage shifts from SGD to Adam as batch size scales.

Original authors: Chenxiang Zhang, Rustem Islamov, Enea Monzio Compagnoni, Jun Pang, Aurelien Lucchi, Antonio Orvieto

Published 2026-06-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chenxiang Zhang, Rustem Islamov, Enea Monzio Compagnoni, Jun Pang, Aurelien Lucchi, Antonio Orvieto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to learn a new skill, like writing a story or recognizing a cat in a photo. To do this, you need a "teacher" (an optimizer) to guide the robot's learning process. For a long time, there have been two main teachers: SGD (a steady, traditional teacher) and Adam (a modern, adaptive teacher).

For years, researchers have argued about which teacher is better. Some said Adam wins because of the data (like how words are used in language). Others said Adam wins because of the architecture (the specific design of the robot's brain). Some said it was about the batch size (how many examples the robot sees at once).

This paper is like a massive, controlled experiment where the researchers put all these theories to the test across different worlds: language, vision, genomics (DNA), and graphs. Here is what they found, explained simply:

1. There is no single "Magic Bullet"

The biggest takeaway is that no single factor explains why Adam is usually better. It's not just the data, and it's not just the robot's design.

The Data Myth: People thought Adam only won because language data is "heavy-tailed" (meaning a few words are used constantly, while most are rare, like in a Zipf distribution). The researchers tested this on DNA data, where the "vocabulary" is tiny (just 4 letters: A, C, G, T) and used very evenly. Surprisingly, Adam still won. So, weird data isn't the only reason.
The Design Myth: People thought Adam only won because of complex "Transformer" designs (like the attention mechanism in modern AI). The researchers stripped away these complex parts and used simpler, older designs. Adam still won.
The "One-Size-Fits-All" Myth: They also found that sometimes, the traditional teacher (SGD) is actually better! If you change the robot's design slightly (like swapping one type of activation function for another), the advantage can flip, and SGD becomes the winner.

2. The "Crossover" Point: It's All About the Class Size

If neither data nor design is the sole answer, what is? The paper suggests the answer lies in how many examples the robot sees at once (the "batch size").

Imagine a classroom:

Small Class (Small Batch Size): When the robot learns from just a few examples at a time, the noise is high. In this chaotic environment, the traditional teacher (SGD) often does just as well, or even better, than the adaptive teacher (Adam).
Large Class (Large Batch Size): As you increase the number of examples the robot sees at once, the "noise" smooths out. At a certain point—called the crossover batch size—the adaptive teacher (Adam) suddenly pulls ahead and starts winning.

The paper shows that this "crossover point" changes depending on the data and the design.

For language tasks, the crossover happens relatively early (Adam wins even with moderate class sizes).
For vision tasks (like recognizing images), the crossover happens much later. You need a huge class size before Adam starts beating SGD. This explains why SGD is still very popular for image tasks.

3. The Theory: A Map of the Terrain

The researchers built a mathematical model to explain why this crossover happens. They compared the "terrain" the robot has to climb.

Sometimes the terrain is bumpy and uneven (high noise).
Sometimes it's smooth.

Their model predicts that if the terrain is "rough" enough and you have a large enough class size, the adaptive teacher (Adam) can navigate it much faster. But if the terrain is smoother or the class is small, the steady teacher (SGD) is just fine. The model confirms that the "winner" depends on the interaction between the shape of the data, the design of the robot, and the size of the class.

4. The "One-Epoch" Reality

The paper also highlights a difference between old-school training and modern training:

Old School (Overparametrized): If you let the robot train for a long time on a small dataset, it eventually learns everything perfectly. In this case, the gap between the teachers shrinks because both can reach the bottom of the hill.
Modern (Underparametrized): In modern Large Language Models, we often train on massive datasets for just one pass (one epoch). Here, the robot doesn't have time to learn everything perfectly. In this race, the adaptive teacher (Adam) consistently finishes with a lower error rate than the traditional teacher.

Summary

The paper concludes that the debate over "Adam vs. SGD" isn't about picking a winner based on one rule. Instead, it's about finding the crossover point.

Think of it like driving a car:

On a narrow, bumpy dirt road (small batch, specific data), a sturdy, simple truck (SGD) might handle it better.
On a wide, smooth highway (large batch, different data), a high-performance sports car (Adam) will zoom ahead.

The "winner" depends entirely on the road conditions (data), the car's design (architecture), and how fast you are driving (batch size). There is no single car that wins every race.

Technical Summary: Beyond a Single Explanation of the Adam–SGD Gap

Problem Statement

For over a decade, the performance disparity between the adaptive optimizer Adam and Stochastic Gradient Descent (SGD) has been a central question in deep learning, particularly in the context of Transformer-based Large Language Models (LLMs). While Adam is the de facto standard for modern foundation models, existing literature offers fragmented explanations for this "Adam–SGD gap." These explanations are typically studied in isolation and attribute the gap to specific factors:

Data Properties: Heavy-tailed vocabulary distributions (Zipfian) in text data creating anisotropic optimization landscapes.
Architecture: Component heterogeneity in Transformers (e.g., softmax-attention, embedding, normalization layers).
Optimization Dynamics: Batch size effects and gradient noise characteristics (e.g., heavy-tailed noise).

The core problem addressed by this work is the lack of a unified understanding of how these factors interact. It remains unclear whether a single factor drives the gap universally, or if the phenomenon arises from complex interactions between data, architecture, and training hyperparameters across different domains.

Methodology

The authors conduct a controlled, large-scale empirical study spanning four distinct domains: language, vision, genomics, and graph learning. The study utilizes both modern Transformer architectures and classical non-Transformer models (e.g., ResNet, GCNN, GAT).

Experimental Design:

Unified Tuning Protocol: To control for confounders, the authors jointly sweep learning rates ( $\eta$ ) and momentum parameters ( $\beta$ ) for both optimizers. For Adam, they use a tied parameterization ( $\beta_1 = \beta_2 = \beta$ ) to align with SignSGD theory, while also testing independent tuning in ablation studies.
Global Constraints: Experiments employ global gradient-norm clipping (to 1) and consistent warmup/cosine schedules.
Gap Metric: The performance gap is defined as $\Delta^* = L^*_{Adam} - L^*_{SGD}$ , where $L^*$ is the final training loss of the best hyperparameter configuration. A negative $\Delta^*$ indicates Adam outperforms SGD.
Theoretical Modeling: The authors complement empirical findings with a theoretical gap model based on non-Euclidean convergence bounds (Kovalev, 2025). They compare Normalized SGD (NSGD) with momentum against a SignSGD proxy for Adam, analyzing how the gap depends on the ratio of smoothness constants ( $r = L_\infty / L_2$ ) and batch size ( $B$ ).

Key Contributions and Results

1. No Single Factor Explains the Gap

The study systematically refutes the hypothesis that any single factor (vocabulary distribution, attention mechanisms, or architecture type) is necessary or sufficient to explain the Adam–SGD gap.

Vocabulary Distribution: The gap persists even on genomics data (HG38) with a near-uniform vocabulary distribution ( $v=5$ ) and small vocabulary size, contradicting the heavy-tailed class imbalance hypothesis.
Softmax-Attention: Replacing softmax-attention layers in GPT with gated convolutions (GCNN) does not eliminate the gap; Adam still outperforms SGD, suggesting attention mechanisms are not the sole driver.
Architecture Heterogeneity: While component heterogeneity (e.g., embedding vs. head layers) influences the gap, it is not deterministic. A Vision Transformer (ViT) trained on ImageNet21K can exhibit a reverse trend where SGD outperforms Adam, despite having similar architectural complexity to models where Adam wins.

2. The Crossover Batch Size Phenomenon

The primary empirical finding is the existence of a crossover batch size. As the batch size ( $B$ ) scales, the relative advantage shifts from SGD to Adam.

Data and Architecture Interaction: The specific point of crossover and the slope of the transition depend jointly on the data modality and the architecture. For instance, image data (I21K) requires significantly larger batch sizes to shift the advantage to Adam compared to language or genomics data, consistent with SGD's historical success in vision tasks.
Architectural Sensitivity: Small architectural changes, such as replacing ReLU with GeLU or BatchNorm with LayerNorm, can shift the optimizer preference from SGD to Adam on the same dataset.

3. Training Regime Dependence

The gap behaves differently depending on the optimization regime:

Overparametrized (Multi-epoch): In settings where the model can reach near-zero training loss (e.g., small datasets trained for many epochs), the gap shrinks as training progresses, and SGD can eventually match Adam's performance.
Underparametrized (One-epoch): In modern LLM settings (large datasets, one epoch), the gap persists throughout training, with Adam maintaining a consistent advantage over SGD.

4. Theoretical Gap Model

The authors propose a theoretical model predicting two regimes based on the geometry smoothness ratio $r = L_\infty / L_2$ :

Regime I ( $1 < r < d^{1/3}$ ): A crossover occurs where SGD is favored at small batch sizes, and Adam becomes favored as $B$ increases. This matches most experimental setups.
Regime II ( $d^{1/3} < r < d$ ): SGD (NSGD) is favored across all batch sizes. This regime corresponds to the ResNet on ImageNet21K experiments where SGD outperforms Adam even at large batch sizes.
The model suggests the gap is determined by the interplay of optimization geometry, dataset properties, and gradient noise variance.

Significance

The paper claims to offer a unified perspective that reconciles previously conflicting hypotheses. Rather than attributing the Adam–SGD gap to a single cause (e.g., "text data is Zipfian" or "Transformers are complex"), the authors demonstrate that the gap is an emergent property of the interaction between data distribution, architectural design, and batch size.

The identification of the crossover batch size provides a practical heuristic: the choice between Adam and SGD should not be absolute but should consider the specific batch size regime and the data-architecture pair. The theoretical model further validates these empirical observations, predicting that the relative performance is contingent on the smoothness ratio of the loss landscape and the noise characteristics.

The work concludes that while Adam offers a consistent practical advantage in modern, large-scale, underparametrized settings (especially with large batch sizes), SGD remains competitive or superior in specific regimes (e.g., vision tasks with large batches or overparametrized settings), challenging the notion of a universal optimizer winner.