🤖 machine learning

When to use what Schatten- $p$ norm in deep learning?

This paper resolves conflicting observations about Schatten- $p$ norm optimizers by demonstrating that while Schatten- $\infty$ methods like Muon excel in high-dimensional settings, smaller Schatten- $p$ geometries are actually optimal in low-dimensional regimes such as Chinchilla scaling, a finding supported by new noise-robust acceleration results for $p>2$ that also explain Muon's lack of warmup requirement and its preference for large batches.

Original authors: Thomas Pethick

Published 2026-06-16

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Thomas Pethick

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Which "Ruler" Should We Use?

Imagine you are trying to find the lowest point in a vast, foggy valley (this is the AI model trying to learn). To do this, you need a "ruler" to measure how steep the ground is and decide which direction to step.

In the world of deep learning, there are different types of rulers, called Schatten-p norms.

Schatten-2 (Euclidean): This is the standard, straight-line ruler we usually use (like the optimizer called Adam).
Schatten-∞ (The "Muon" ruler): This is a special, rigid ruler that only cares about the single steepest cliff in the landscape. Recently, a method called Muon (using this ruler) became very popular because it seemed to work incredibly fast in some tests.
Schatten-p (The "Goldilocks" rulers): These are rulers in between, measuring the slope in a way that is neither too soft nor too rigid.

The Confusion:
People were arguing about which ruler is best.

Some said: "Muon (Schatten-∞) is amazing! It's the fastest!"
Others said: "No, Muon is overrated. Standard methods work better when we train on huge amounts of data."

The Paper's Answer:
The author, Thomas Pethick, says: "You are both right, but you are looking at different sizes of the valley."

The best ruler depends entirely on the size of the problem relative to the amount of data you have.

The Two Regimes: Small Pools vs. Big Oceans

The paper divides the world into two main scenarios:

1. The "Low-Dimensional" Regime (The Small Pool)

What it is: This happens when you have a relatively small model or a short training run (like the "speedruns" on small games mentioned in the paper). The number of steps you take (data) is much larger than the complexity of the model.
The Analogy: Imagine you are walking through a small, well-lit garden. You can see the whole path.
The Best Ruler: In this scenario, the Schatten-∞ ruler (Muon) is actually not the best. Surprisingly, a smaller p-ruler (closer to the standard Euclidean ruler) is faster.
Why? Because in a small garden, you don't need to be hyper-focused on the single steepest cliff. You need a smoother, more balanced approach to accelerate quickly. The paper proves that using a "softer" ruler here gives you a speed boost.

2. The "High-Dimensional" Regime (The Big Ocean)

What it is: This is the massive training runs used for huge AI models (like LLMs) where the model is incredibly complex and the data is vast.
The Analogy: Imagine you are navigating a massive, dark ocean with thousands of islands. You can't see the whole map.
The Best Ruler: Here, the Schatten-∞ ruler (Muon) shines. It is designed to handle the specific "roughness" of these massive, complex landscapes.
The Catch: To make Muon work in this big ocean, you need a huge batch size (looking at a very wide slice of the ocean at once). If you try to use Muon with a small batch size in this regime, it fails.

The "Batch Size" Secret

One of the paper's key findings is about Batch Size (how many examples the AI looks at before taking a step).

The Rule: The paper derives a mathematical rule showing that as you change your ruler (from $p=2$ to $p=\infty$ ), the ideal batch size must change too.
The Metaphor: Think of the ruler as a boat.
- A small boat (Standard Ruler) can navigate with a small crew (small batch size).
- A massive cruise ship (Muon/Schatten-∞) needs a huge crew (massive batch size) to stay stable and move fast. If you try to sail a cruise ship with only two people, it will just spin in circles.
The Insight: This explains why Muon works great for "speedruns" (where people use massive batches on small models) but struggles in some large-scale benchmarks (where the batch size might not be scaled up enough for the specific geometry).

Why Do We Need "Warmups"?

You might have heard that some AI trainers need a "warmup" (starting slow and speeding up).

The Paper's Finding: The Schatten-∞ ruler (Muon) is so robust that it doesn't need a warmup. It can start at full speed immediately.
The Contrast: If you use a smaller $p$ -ruler in the low-dimensional regime, you do benefit from a warmup phase to get the acceleration going.

The "Chinchilla" Connection

The paper connects this to a famous rule in AI called Chinchilla scaling, which says: "As you get more data, you should make your model bigger."

The Twist: The paper argues that even with Chinchilla scaling, we are often still in the "Low-Dimensional Regime" (the small garden).
The Result: Because we are often in this "small garden" regime, using the standard Muon (Schatten-∞) might actually be less efficient than using a finite $p$ -norm optimizer (like HTMuon or Soft-Muon). This explains why newer methods using "softer" rulers are starting to beat Muon in some large-scale tests.

Summary: How to Choose Your Ruler

The paper concludes with a simple guide:

If you are in a "Low-Dimensional" regime (small models, or when the amount of data is huge compared to the model size):
- Don't automatically use the extreme Schatten-∞ (Muon).
- Do consider using a finite Schatten- $p$ norm (a "softer" ruler). It accelerates faster and handles noise better.
- Note: You might even want to switch rulers as you train! Start with a "hard" ruler and switch to a "softer" one later.
If you are in a "High-Dimensional" regime (massive complexity):
- The Schatten-∞ (Muon) is likely the right choice, BUT you must use a very large batch size to make it work.

The Bottom Line: There is no single "best" optimizer. The best tool depends on the size of the problem and how much data you have. The confusion in the community existed because people were testing the tools in different "regimes" without realizing it.

Technical Summary: "When to use what Schatten-p norm in deep learning?"

Problem Statement

Schatten- $\infty$ based optimizers, such as Muon, have demonstrated promising empirical performance in specific deep learning settings, particularly those involving short training runs or large batch sizes. However, conflicting observations exist regarding their general utility. While they excel in certain regimes (e.g., CIFAR-10 speedruns), they do not consistently outperform standard methods like Adam or SGD in large-scale training scenarios with extensive token budgets. Furthermore, recent variants utilizing finite Schatten- $p$ norms (e.g., HTMuon, Soft-Muon) have shown competitive or superior results, challenging the necessity of exclusively using the Schatten- $\infty$ norm.

The core question addressed is: What Schatten- $p$ norm should an optimizer use, and under what training regimes is it optimal? Existing theoretical frameworks, specifically the SODA (Schedule-free Optimistic Dual Averaging) framework, lacked the ability to analyze $p$ -norm geometries for $p > 2$ under stochastic noise, leaving the mechanism behind Muon's success and its limitations theoretically unexplained.

Methodology

The paper extends the SODA framework (Pethick et al., 2026), which unifies modern optimizer components like optimistic dual averaging, Nesterov momentum, and weight decay. The authors generalize the analysis of SODA to $p$ -uniformly convex reference functions for $p \ge 2$ , covering both vector $\ell_p$ and matrix Schatten- $p$ geometries.

Key methodological components include:

Algorithmic Template: The analysis utilizes a primal-dual update rule where the dual map is defined by the gradient of the Fenchel conjugate of a reference function $h(x) = \frac{1}{p}\|x - z_0\|_p^p$ $h (x) = \frac{1}{p} ∥ x - z_{0} ∥_{p}^{p}$ .
- For $p=2$ , this recovers Euclidean geometry.
- For $p \to \infty$ , it recovers the linear minimization oracle (LMO) used in Muon ( $UV^\top$ ).
- For finite $p > 2$ , it corresponds to methods like HTMuon and Soft-Muon.
Noise Assumptions: The analysis incorporates heavy-tailed noise assumptions, specifically assuming gradients satisfy an $r$ -moment condition ( $r \in (1, 2]$ ). This allows the derivation of convergence rates that depend on the dual exponent $q = p/(p-1)$ .
Regime Analysis: The authors derive oracle complexity bounds under two distinct regimes:
- Deterministic: Where noise is negligible.
- Stochastic: Where noise is present, and batch sizes are scaled to manage variance.
Scaling Laws: The theoretical bounds are applied to modern deep learning scaling rules (e.g., Chinchilla), relating the total token budget $N$ to the effective dimension $d$ of model matrices.

Key Contributions

1. Noise-Robust Acceleration for Arbitrary $p$

The authors prove that SODA achieves accelerated convergence rates for optimization over $p$ -norm geometries while remaining robust to stochastic noise. They establish a dimension-free rate:
$O\left( \frac{L_p R_p^2}{n^{1+2/p}} + \frac{\sigma_q R_p}{n^{1/p}} \right)$
This result reveals a fundamental connection between the choice of geometry $p$ and the heaviness of the noise tails (via the dual exponent $q$ ). This is presented as the first noise-robust acceleration result for arbitrary $p$ -norm geometries.

2. Batch Size Scaling Rules

By analyzing the oracle complexity under bounded variance ( $r=2$ ), the paper derives a norm-dependent batch size scaling rule. For a fixed total oracle budget $N$ , the optimal batch size $B_p$ scales as:
$B_p \asymp N^{2(p-1)/(3p-2)}$

For $p=2$ (Euclidean), $B_2 \asymp N^{1/2}$ .
For $p \to \infty$ (Schatten- $\infty$ ), $B_\infty \asymp N^{2/3}$ .
This explains why Schatten- $\infty$ methods like Muon empirically require larger batch sizes to realize their benefits.

3. Regime-Dependent Optimality

The paper resolves the conflict regarding Muon's utility by showing that the optimal choice of $p$ is regime-dependent, determined by the relationship between the effective dimension $d$ and the oracle budget $N$ :

High-Dimensional Regime ( $N \lesssim d$ ): Matching the geometry to the problem smoothness is favorable. If the objective is smooth in the Schatten- $\infty$ norm, using $p \to \infty$ is optimal.
Low-Dimensional Regime ( $N \gtrsim d$ ): Smaller values of $p$ $p$ are preferred.
- In deterministic settings, $p=2$ (Euclidean) offers stronger acceleration.
- In stochastic settings with heavy-tailed noise, the smallest admissible geometry $p = r/(r-1)$ is optimal due to better alignment with noise assumptions.

4. Implications for Scaling Laws

The authors connect these theoretical regimes to modern scaling laws (e.g., Chinchilla). Under typical scaling recipes where model size scales linearly with tokens ( $N$ ), the effective rank of weight matrices grows as $d \propto N^{1/3}$ . Consequently, $N \gg d$ , placing standard large-scale training firmly in the low-dimensional regime.

Conclusion: In this regime, smaller $p$ (finite Schatten- $p$ ) is theoretically preferable over Schatten- $\infty$ . This provides a theoretical explanation for why Muon's advantage diminishes at scale and why finite- $p$ optimizers (HTMuon, Soft-Muon) are emerging as strong contenders.

Results and Significance

The paper claims to resolve the apparent contradiction in the empirical performance of Schatten- $\infty$ optimizers. The significance of the work lies in:

Theoretical Unification: Providing a single framework (SODA) that explains the behavior of Muon, Soft-Muon, and Euclidean optimizers through the lens of $p$ -norm geometry.
Explaining Empirical Phenomena:
- Warmup: The analysis explains why Schatten- $\infty$ methods do not require stepsize warmup (the effective stepsize decays as $k^{-1}$ ), whereas smaller $p$ geometries exhibit a warmup-like phase.
- Batch Size: It quantifies the necessity of large batches for Schatten- $\infty$ methods and provides a scaling rule for arbitrary $p$ .
- Scaling: It offers a theoretical basis for the observation that Schatten- $\infty$ methods may not scale as well as expected under Chinchilla-like recipes, suggesting that the "low-dimensional" nature of modern LLM training favors finite $p$ .
Practical Guidance: The results suggest a potential switching strategy where $p$ could be adjusted during training (starting near $\infty$ and switching to a finite $p$ as the budget $N$ exceeds the dimension $d$ ), aligning with recent empirical schedules like Soft-Muon.

The authors maintain a modest tone, noting that their analysis relies on worst-case norm conversions and that sharper, problem-dependent relations could refine the boundaries between regimes. They leave the practical exploration of dynamic $p$ -switching as future work.