🤖 machine learning

Why Muon Outperforms Adam: A Curvature Perspective

This paper demonstrates that Muon outperforms Adam in large language model training by achieving a larger one-step loss decrease primarily through a lower Normalized Directional Sharpness (NDS) curvature penalty, a geometric advantage that is amplified by data imbalance and sustained by reduced within-layer curvature.

Original authors: Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Dirk Bergemann, Zhuoran Yang

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Dirk Bergemann, Zhuoran Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find the lowest point in a vast, foggy mountain range. This is what training a giant AI model (like a Large Language Model) is like: the "mountain" is the error rate, and the "lowest point" is the perfect, most accurate model. To get there, you need an optimizer—a guide that tells you which way to step.

For a long time, Adam has been the standard guide. Recently, a new guide called Muon arrived and started getting to the bottom of the mountain twice as fast. But why? This paper acts like a detective, using a "curvature perspective" (looking at the shape of the terrain) to solve the mystery.

Here is the story of their findings, explained simply:

1. The Two Guides Take the Same Step Size, But Muon is Smoother

Imagine you and a friend are hiking down a hill. You both decide to take a step of the exact same length.

The First Step (The Push): Both you and your friend push forward with the same amount of force. In math terms, the paper calls this the "first-order gain." They are equal.
The Bump (The Curvature Penalty): The ground isn't perfectly flat; it's bumpy. If you step too hard in a bumpy direction, you bounce back up a little, wasting energy.
- Adam tends to step into the "bumpiest" parts of the terrain. It hits a high bump, bounces back, and loses some of its forward progress.
- Muon is smarter. It looks at the shape of the ground and steers its step toward the smoother, flatter paths. It still takes a step of the same length, but it hits a much smaller bump.

The Result: Because Muon hits fewer bumps (a smaller "curvature penalty"), it actually travels further down the hill in a single step than Adam does, even though they pushed with the same initial force.

2. The Secret Weapon: "Normalized Directional Sharpness" (NDS)

The paper introduces a fancy term called NDS, which essentially measures "how sharp the ground is in the direction you are walking."

The Finding: Muon and Adam take steps of the same size (same "update norm"), but Muon's steps are always in directions where the ground is less sharp.
The Analogy: Imagine walking through a field of tall, sharp corn.
- Adam walks straight through the thickest, sharpest stalks. It gets cut up (high NDS), and the corn pushes back against it.
- Muon uses a special compass to find the gaps between the corn. It walks through the same distance, but the corn is much less dense (low NDS). It glides through with less resistance.

3. Why Does Muon Find the Gaps Better? (Data Imbalance)

The paper discovered that the "terrain" gets bumpier when the data is imbalanced.

The Scenario: Imagine a library where 90% of the books are about "cats," and only 10% are about "dogs." This is an imbalanced dataset.
The Effect: In these imbalanced libraries, the ground becomes extremely jagged and full of sharp spikes.
The Winner: Adam gets really stuck in the "cat" spikes. Muon, however, is much better at navigating these jagged, uneven landscapes. The more imbalanced the data, the bigger the gap between Muon's smooth path and Adam's bumpy path.

4. Where Does the Magic Happen? (Inside the Layers)

AI models are built like a stack of layers (like a multi-layer cake). The paper looked at whether Muon's advantage came from the whole cake or just specific slices.

The Discovery: As training goes on, Muon's advantage shifts. It stops worrying about how the layers interact with each other (cross-layer) and focuses entirely on making the steps smooth within each individual layer (within-layer).
The Analogy: Think of a relay race. At the start, everyone is worried about passing the baton between runners (cross-layer). But as the race heats up, Muon realizes the secret to speed is just running its own leg perfectly smooth (within-layer), ignoring the noise of the other runners.

5. The Theoretical Proof: The "Equalizer"

Finally, the authors built a simple mathematical model (a "stylized quadratic problem") to prove why this works.

The Problem: Imagine the mountain has a few very steep, sharp cliffs (high curvature) and a huge area of gentle slopes (low curvature).
Adam's Mistake: Because the "cliffs" are so steep, Adam gets drawn to them. It spends all its energy trying to climb down the steep cliffs, but because they are so sharp, it bounces around wildly.
Muon's Strategy: Muon uses a "spectral normalization" trick. Imagine it has a magic equalizer that forces it to spend the same amount of energy on the steep cliffs as it does on the gentle slopes.
The Outcome: By not over-focusing on the dangerous cliffs, Muon avoids the wild bouncing. It distributes its energy evenly, allowing it to make steady, efficient progress down the mountain that Adam can't match.

Summary

Muon beats Adam not because it takes bigger steps or pushes harder, but because it is a better navigator. It avoids the "sharp" parts of the mathematical landscape that cause the AI to bounce back. By smoothing out its path, especially when the data is messy or imbalanced, it reaches the bottom of the mountain (the best model) much faster.

Technical Summary: Why Muon Outperforms Adam: A Curvature Perspective

Problem Statement

Muon has emerged as a highly efficient optimizer for Large Language Model (LLM) pretraining, demonstrating approximately 2 $\times$ faster training convergence compared to the standard Adam optimizer across various model scales. While recent works have attributed this advantage to mechanisms like associative memory and data long-tailedness, the local geometric source of Muon's superiority remains unclear. This paper addresses two primary questions:

What specific property of the optimization landscape supports Muon's advantage over Adam?
How do pretraining factors, specifically training data distribution and model structure, influence this property?

Methodology

The authors employ a combination of empirical analysis on LLM training trajectories and theoretical analysis on stylized quadratic optimization problems.

Empirical Analysis:

Second-Order Taylor Decomposition: The authors analyze the one-step loss decrease ( $\Delta D$ ) by decomposing it into a first-order gain (gradient alignment) and a second-order curvature penalty using a Taylor expansion: $\Delta D \approx \langle G, Z \rangle - \frac{1}{2}\langle Z, H[Z] \rangle$ .
Metric Definition: To isolate the effect of update direction from update scale, they introduce Normalized Directional Sharpness (NDS), defined as $SF(W; Z) = \langle Z, H[Z] \rangle / \|Z\|_F^2$ . This allows the curvature penalty to be factored into the update norm and NDS.
Experimental Setup: Experiments are conducted on a 124M-parameter NanoGPT model trained on the FineWeb dataset. Comparisons between Muon and Adam are made at matched validation loss levels to ensure fairness.
Data and Structure Variations:
- Data Imbalance: Synthetic data is generated using a Zipf-Probabilistic Context-Free Grammar (PCFG) with controlled imbalance levels (Zipf exponent $s$ ) to test the effect of heavy-tailed distributions.
- Layer Decomposition: The NDS is decomposed into within-layer (diagonal Hessian blocks) and cross-layer (off-diagonal interactions) contributions to understand how different model components contribute to the curvature advantage.

Theoretical Analysis:

Stylized Quadratic Model: The authors construct a quadratic optimization problem mirroring LLM landscape characteristics, based on four empirically verified assumptions:
1. Low Kronecker-Rankness: The Hessian admits a low-rank Kronecker approximation.
2. Simultaneous Diagonalization: The Kronecker factors share common orthogonal eigenbases.
3. Curvature Heterogeneity: The Hessian spectrum exhibits strong heterogeneity (long-tailed distribution of eigenvalues).
4. Gradient Alignment: Gradients align strongly with high-curvature eigendirections.
Comparison: Theoretical proofs compare Muon (spectrally normalized updates) against Gradient Descent (GD). Adam is noted to behave similarly to GD in this specific stylized setting, allowing the authors to focus on the contrast between spectral normalization and gradient-based updates.

Key Contributions and Results

1. The Source of Muon's Advantage: Reduced Curvature Penalty
The analysis reveals that Muon achieves a larger one-step loss decrease than Adam primarily because it incurs a significantly smaller second-order curvature penalty, not because it achieves a larger first-order gradient gain.

Observation 1: At matched validation losses, Muon and Adam show comparable first-order decreases, but Muon's curvature penalty is substantially lower.

2. The Mechanism: Lower Normalized Directional Sharpness (NDS)
The reduced curvature penalty is driven by Muon's update direction rather than its step size.

Observation 2: Muon and Adam have comparable update norms ( $\|Z\|_F$ ). However, Muon exhibits a notably lower NDS (approximately 1.76 $\times$ lower than Adam on average). This indicates that Muon's spectral normalization steers updates toward directions with lower local curvature.

3. The Role of Data Imbalance
The NDS advantage of Muon is amplified by data imbalance.

Observation 3: Using Zipf-PCFG data, the authors show that as the dataset becomes more imbalanced (higher Zipf exponent $s$ ), the NDS for both optimizers increases, but the gap between Adam and Muon widens significantly (increasing by 1.8 $\times$ from $s=0$ to $s=1$ ). Muon is more robust to the increased sharpness induced by heavy-tailed data.

4. The Role of Model Structure
The curvature advantage shifts over the course of training.

Observation 4: While both within-layer and cross-layer NDS components are lower for Muon, the within-layer component becomes increasingly dominant in Muon's advantage during the middle and late stages of training. Muon's cross-layer contribution drops rapidly, whereas Adam maintains a stable balance between within- and cross-layer sharpness.

5. Theoretical Justification
The paper proves that on quadratic problems with heterogeneous curvature and gradient alignment toward high-curvature modes:

Muon attains a smaller average NDS than GD because its spectral normalization balances update energy across curvature groups, whereas GD concentrates energy on high-curvature modes.
When curvature heterogeneity is sufficiently strong, Muon achieves a lower local quadratic loss than GD after the same number of steps.

Significance and Claims

The paper claims to take the "first step" toward demystifying Muon's superiority from a local curvature perspective. The central conclusion is that Muon's efficiency stems from its ability to perform spectrally normalized updates that inherently possess lower directional sharpness, thereby minimizing the second-order cost of optimization steps.

The authors explicitly position this work as complementary to existing "memory-centric" explanations (e.g., Wang et al., 2025b), offering a distinct geometric mechanism: at the same iterate, Muon pays a smaller directional sharpness penalty than Adam. The work is modest in scope, focusing on causal LLMs and identifying a local mechanism rather than claiming global convergence guarantees or applicability to all model classes (such as diffusion models), which are left for future work.