🤖 machine learning

When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study

This paper presents a theoretical and empirical study demonstrating that the Muon optimizer, which orthogonalizes matrix-valued updates, enhances adversarial training robustness across diverse architectures and threat models by imposing spectral-norm stability, thereby positioning optimizer geometry as a critical factor in model security.

Original authors: Jun Yan, Weiquan Huang, Jiankai Zuo, Yujian Mo, Xi Fang, Chengliang Wu, Zeming Wei

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jun Yan, Weiquan Huang, Jiankai Zuo, Yujian Mo, Xi Fang, Chengliang Wu, Zeming Wei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Building a Fort Against Sneaky Attackers

Imagine you are training a robot to recognize cats and dogs. You want it to be so smart that even if someone sneaks a tiny, invisible sticker on a cat's ear (an "adversarial attack"), the robot still correctly identifies it as a cat. This process of training the robot to ignore these sneaky tricks is called Adversarial Training (AT).

For a long time, the standard tool for building this "fortress" was a method called SGD (Stochastic Gradient Descent). Think of SGD as a very steady, reliable hiker who takes small, careful steps up a mountain. It's slow, but it rarely falls off a cliff.

Recently, a new, faster tool called AdamW became popular. It's like a hiker with a jetpack who adjusts their speed based on the terrain. It gets to the top of the mountain (standard learning) much faster, but in the "sneaky attack" training scenario, the jetpack sometimes causes the hiker to wobble and fall off the cliff (poor security).

The Question: Is there a way to get the speed of the jetpack without the wobble?

Enter the Muon Optimizer: The "Dance Instructor"

The paper introduces a new tool called Muon. The authors compare Muon to a dance instructor who specializes in keeping dancers perfectly aligned.

The Problem with Standard Tools: When AdamW tries to learn, it sometimes makes the robot's internal "muscles" (mathematical weights) grow too strong or unbalanced, like a weightlifter who accidentally lifts a barbell that is too heavy and breaks their back. This instability makes the robot vulnerable to attacks.
The Muon Solution: Muon acts like a strict dance instructor. Every time the robot tries to take a step (update its learning), Muon checks the move. If the move is too wild or unbalanced, Muon forces the robot to "orthogonalize" the step.
- The Analogy: Imagine the robot is trying to push a heavy box. AdamW might push it at a weird, jagged angle that causes the box to tip over. Muon ensures the robot pushes the box in a perfectly straight, balanced line, like a dancer gliding across a stage. It keeps the "shape" of the movement stable, preventing the robot from developing "bad habits" (unstable weights) that attackers can exploit.

What the Paper Found (The Results)

The researchers tested this "Dance Instructor" (Muon) against the "Steady Hiker" (SGD) and the "Jetpack Hiker" (AdamW) in three different scenarios:

1. Small Datasets (Like CIFAR-10)

The Result: Muon was a huge success. It was much better than AdamW (which struggled) and was just as good as, or sometimes better than, the steady SGD.
The Takeaway: On smaller, manageable tasks, Muon's "dance moves" kept the training stable and secure, preventing the robot from getting confused by the sneaky attacks.

2. Vision Transformers (ViTs)

The Result: This is where AdamW really failed. When trying to train complex "Vision Transformer" models, AdamW often crashed completely, resulting in a robot that was essentially guessing randomly. Muon, however, kept the training stable and produced a secure robot.
The Takeaway: For these complex, modern AI architectures, the "Dance Instructor" was essential. Without Muon, the training was too chaotic to be secure.

3. Large Datasets (Like ImageNet)

The Result: When they moved to a massive dataset (millions of images), the "Steady Hiker" (SGD) was still the champion. Muon was good, but it struggled a bit more with the sheer scale and required very specific tuning to work well.
The Takeaway: Muon is a powerful tool, but for the biggest, heaviest lifting jobs, the old reliable method (SGD) still has the edge right now.

The "Why" Behind the Magic

The paper doesn't just say "it works"; it explains why using some math (which we can translate):

The Speed Limit: Muon puts a "speed limit" on how much the robot's internal weights can grow in any single direction. It prevents the "uncontrolled growth" that happens with AdamW.
The Safety Net: Think of AdamW as a car with no brakes that speeds up on curves. Muon is like a car with a governor that limits the speed so it never spins out of control, even if the road gets bumpy. This "speed limit" on the math ensures the robot doesn't become fragile.

The Bottom Line

The paper concludes that how you choose to "drive" the learning process matters just as much as the car itself.

Muon is a new, exciting driver that keeps the car stable and secure, especially when the road is tricky (complex models) or the traffic is light (small datasets).
It is not a magic wand that fixes everything (it still needs tuning for huge datasets), but it is a major improvement over the current "jetpack" method (AdamW) for security.
The authors suggest that in the future, security experts should pay more attention to the "driver" (the optimizer) they choose, not just the "car" (the model architecture).

In short: If you want your AI to be tough against sneaky attacks, using the Muon "Dance Instructor" is a great way to keep your training stable, especially for modern, complex AI models.

Technical Summary: When Muon Optimizer Meets Adversarial Training

Problem Statement

Adversarial Training (AT) remains the most reliable empirical defense against adversarial attacks, yet its robustness is critically dependent on how the underlying min-max optimization problem is solved. While Stochastic Gradient Descent (SGD) is the default optimizer for AT due to its observed robustness, adaptive optimizers like AdamW often improve standard training but yield inferior robustness. Recently, the Muon optimizer has emerged as a high-performance alternative for large-scale training, utilizing orthogonalized matrix-valued updates via approximate polar decomposition. However, the impact of this specific optimizer geometry on adversarial robustness under strong and heterogeneous threat models remains unexplored. This paper addresses the question: Can orthogonalized optimization improve AT, and what are the theoretical and empirical mechanisms governing this behavior?

Methodology

The study employs a dual approach combining theoretical analysis of the Muon update rule with comprehensive empirical evaluation across multiple architectures and threat models.

Theoretical Analysis

The authors analyze the Muon update rule, where the momentum-accumulated gradient $M_t$ is transformed into an approximate polar factor $U_t$ (orthogonalized) before updating weights $W_{t+1} = (1-\eta\lambda)W_t - \eta U_t$ .

Spectral-Norm Stability Ceiling: The paper proves that the polar update imposes a structural ceiling on the spectral norm of matrix updates. Unlike adaptive optimizers which can exhibit unbounded spectral growth, Muon's orthogonalized updates ensure that the spectral norm of weights remains bounded by a function of the learning rate, weight decay, and approximation error ( $\|W_t\|_2 \leq \max(\|W_0\|_2, \frac{1+\bar{\epsilon}}{\lambda})$ ).
Gradient Dynamics: Theoretical results indicate that Muon descends the adversarial loss along the nuclear-norm direction. This provides a structurally larger per-step descent guarantee compared to standard gradient steps, particularly in the presence of high-variance adversarial gradients.
Robustness Bounds: By bounding the layer spectral norms, the authors derive a structural bound on the Lipschitz constant of the margin function, linking the optimizer's update geometry to the model's robustness against $\ell_p$ -bounded attacks.

Empirical Evaluation

The authors adapted Muon for AT using Auto Projected Gradient Descent (APGD) as the inner attack loop. Experiments were conducted on:

Datasets: CIFAR-10 and ImageNet.
Architectures: PreActResNet-18, WideResNet (WRN-34-10, WRN-34-20), and Vision Transformers (ViT-B, ViT-L).
Threat Models: $\ell_\infty$ , $\ell_1$ , $\ell_2$ , and their unions.
Comparators: SGD, AdamW, and Sharpness-Aware Minimization (SAM).

Key Contributions

Revisiting Optimizer Choice as a Security Factor: The paper identifies orthogonalized matrix-update geometry as a critical, underexplored factor in adversarial training, moving beyond the conventional SGD vs. AdamW comparison.
Mechanistic Analysis of Muon: The authors provide a theoretical explanation for Muon's behavior, demonstrating that its polar update rule imposes an operator-norm ceiling on matrix-valued updates. This prevents the unstable spectral growth often observed with adaptive optimizers, offering a diagnostic explanation for robustness gains.
Controlled Multi-Norm Evaluation: The study presents a systematic comparison of SGD, AdamW, and Muon across diverse settings. It clarifies the trade-offs between orthogonalized updates (Muon) and sharpness-aware perturbation (SAM), showing that Muon acts as an optimizer-geometry alternative rather than a direct replacement.

Key Results

Performance vs. AdamW: Muon substantially outperforms AdamW in AT across both CNNs and ViTs. On ViT architectures, AdamW often suffers from training instability or collapse to near-random performance, whereas Muon maintains stability and competitive robustness.
Performance vs. SGD: On small-scale datasets (CIFAR-10) with overparameterized CNNs (e.g., WRN-34-10/20), Muon achieves union robustness on par with or superior to SGD. However, on large-scale datasets like ImageNet, SGD currently performs better, suggesting Muon is more sensitive to hyperparameter tuning (learning rate, warmup) at scale.
Comparison with SAM: Muon improves update geometry without the additional computational cost of the weight-perturbation step required by SAM. While SAM often achieves slightly higher robustness in specific settings, Muon offers a more efficient alternative with comparable stability.
Robust Overfitting: The study observes that Muon, like SGD, is susceptible to "robust overfitting" in later training stages. Early stopping remains a necessary strategy for Muon-based AT.
Black-Box Robustness: Muon-trained models demonstrate strong transferability, maintaining high robustness against black-box transfer attacks, indicating that the gains are not merely artifacts of white-box optimization paths.

Significance and Claims

The paper claims that optimizer geometry is a security-relevant factor in adversarial training. By introducing Muon, the authors demonstrate that orthogonalized updates can provide a stable alternative to SGD, particularly in scenarios where adaptive optimizers fail (e.g., Transformers) or where computational efficiency is a concern compared to SAM.

The authors modestly conclude that Muon does not universally dominate SGD; rather, it offers a "stable orthogonalized update geometry" that often improves early robust optimization. The findings highlight that while Muon is a powerful tool for AT, its effectiveness at large scales (ImageNet) requires further hyperparameter and schedule design. The work establishes a theoretical link between polar decomposition and spectral stability, suggesting that controlling the geometry of updates is as important as the loss landscape itself for achieving robustness.