🤖 machine learning

Reassessing Muon for Matrix Factorization

This paper reevaluates the Muon optimizer on low-rank matrix factorization to isolate its update rule from confounding factors, revealing that it does not consistently outperform AdamW and that its reported advantages are often sensitive to hyperparameter choices.

Original authors: Ali Parviz, Gal Mishne, Alex Cloninger

Published 2026-07-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ali Parviz, Gal Mishne, Alex Cloninger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to learn. To do this, you need a "coach" that tells the robot which way to step next to get better. In the world of artificial intelligence, this coach is called an optimizer. For years, the most popular coach has been a method called AdamW, which is great at adjusting its steps based on how slippery or bumpy the path is. But recently, a new, flashy coach named Muon arrived. Muon claims to be special because it doesn't just look at the path; it tries to "straighten out" the robot's steps using a fancy mathematical trick called orthogonalization. Think of it like a dance instructor who forces the dancer to move in perfect, non-overlapping directions, hoping this makes the dance more efficient.

The big question on everyone's mind is: Is Muon actually a better coach than AdamW for the massive, complex tasks modern AI does, like writing stories or generating images? Or is Muon just looking good because the dance floor is huge and chaotic? Scientists have been running huge tests on giant AI models, and Muon seems to win. But when you are dealing with billions of variables, it's hard to tell if the coach is doing the work or if the sheer size of the problem is just hiding the coach's mistakes. This is where the story gets interesting: to find the truth, you can't just watch the big show; you have to test the coaches in a quiet, controlled room.

This paper, titled "Reassessing Muon for Matrix Factorization," takes Muon and AdamW out of the chaotic, billion-parameter dance floor and drops them into a simple, well-lit practice room. The researchers decided to test them on a specific, fundamental task called matrix factorization. If you imagine a giant spreadsheet of numbers that you want to break down into two smaller, simpler spreadsheets that multiply back together, that's matrix factorization. It's a clean, mathematical puzzle where we know exactly what the solution looks like and can measure every tiny step the coaches take.

The researchers ran a massive experiment, tweaking the "learning rate" (how big a step the robot takes) for both coaches across many different versions of the puzzle. They tested everything from easy, smooth puzzles to extremely difficult, "ill-conditioned" ones where the numbers are tricky and decay rapidly. They also looked at different types of puzzles, including some where the numbers had to stay positive (Non-negative Matrix Factorization).

Here is what they found, and it might surprise you: Muon is not a magic bullet. When the researchers gave both coaches a fair chance to find their perfect step size, Muon did not consistently beat AdamW. In fact, for standard low-rank factorization tasks, AdamW and even plain old Gradient Descent often matched or even outperformed Muon. The "superiority" of Muon that people saw in giant AI models seems to vanish when you control the variables. The paper suggests that Muon's success in the real world might be an artifact of the specific scale and architecture of those huge models, rather than the optimizer itself being fundamentally better at the math.

However, Muon isn't useless. The paper found that Muon does have a clear advantage in one specific scenario: Non-negative Matrix Factorization (NMF). In these puzzles, where the numbers must stay positive, Muon's unique way of straightening out the steps helped it avoid redundant solutions and find more diverse answers. It seems Muon is a specialist, not a generalist. It shines when the problem has specific geometric quirks (like the NMF constraints) but struggles to dominate when the problem is just a standard, well-behaved math puzzle.

The study also revealed that the "conditioning" of the problem (how tricky the numbers are) changes the winner. In very difficult, ill-conditioned scenarios, the ranking of the coaches can flip completely depending on the shape of the data. The authors conclude that we can't just look at one test or one default setting to decide which optimizer is best. We need to test them across a wide range of conditions and step sizes. The takeaway is that while Muon is a powerful tool, it doesn't automatically dethrone the old champions like AdamW in every situation. Its success depends heavily on the specific shape of the problem it's trying to solve.

Technical Summary: Reassessing Muon for Matrix Factorization

Problem Statement

Muon (MomentUm Orthogonalized by Newton–Schulz) has recently emerged as a powerful optimizer for large-scale deep learning, particularly in training large language models (LLMs), where it reportedly outperforms standard adaptive methods like Adam and AdamW. Theoretical literature interprets Muon as performing steepest descent under the spectral norm, leveraging approximate orthogonalization to reshape gradient updates.

However, a critical gap remains: it is unclear whether Muon's empirical success stems from its intrinsic update rule or is an artifact of the specific scale, architecture, and data distributions of modern deep networks. Existing theoretical analyses often rely on idealized settings or local properties, failing to specify concrete conditions under which Muon should theoretically or empirically outperform classical optimizers.

This paper addresses this uncertainty by isolating the optimizer from confounding factors (architecture, scale, data) by evaluating Muon on low-rank matrix factorization. This problem is chosen for its simplicity, well-understood optimization landscape, and explicit spectral structure, allowing for systematic and exhaustive evaluation of how Muon's spectral nature interacts with curvature and singular-value structures.

Methodology

The authors conduct a controlled, systematic comparison of Muon against adaptive baselines (AdamW, plain Gradient Descent, and SignGD) across five distinct matrix-recovery problem settings. The evaluation protocol is designed to eliminate bias from hyperparameter selection, which the authors identify as a primary source of misleading conclusions in prior work.

Experimental Design

Problem Variants:
- Symmetric Low-Rank Factorization: Recovering a rank- $r$ positive semidefinite matrix $M^\star$ from $UU^\top$ .
- Matrix Completion: Recovering a matrix from a subset of observed entries (20% observation rate).
- Non-negative Matrix Factorization (NMF): Constrained factorization where both target and factors are non-negative.
- Tensor-Train Factorization: A bilinear factorization setup applied to symmetric PSD targets, serving as a minimal model for product parameterizations in deep learning.
- Gaussian-Kernel Factorization: A smooth, well-conditioned landscape using an RBF kernel matrix.
Controlled Variables:
- Conditioning ( $\kappa$ ): Experiments sweep condition numbers from well-conditioned ( $\kappa=1$ ) to severely ill-conditioned ( $\kappa=625$ or $10^4$ ).
- Spectral Shape: The distribution of eigenvalues is varied (uniform, decayed, heavy-tailed, clustered) while holding the condition number fixed to test sensitivity to spectral density.
- Parameterization: Experiments cover under-parameterized, exactly parameterized, and over-parameterized regimes.
Evaluation Protocol:
- Hyperparameter Tuning: Instead of using a single "default" learning rate, the authors perform exhaustive learning rate sweeps (logarithmic grids) for every optimizer and problem configuration.
- Metric: Recovery is measured via Normalized Mean-Squared Error (NMSE).
- Comparison: Optimizers are compared based on their "stable learning-rate range" (the band of step sizes where convergence to the loss floor occurs) and their best-tuned final loss.

Key Contributions and Findings

1. Hyperparameter Sensitivity and Tuning

The paper demonstrates that optimizer rankings are highly sensitive to learning rate selection.

Default vs. Tuned: Rankings derived from a single default learning rate (e.g., $10^{-3}$ ) frequently contradict rankings obtained after individual tuning.
No Universal Dominance: When each method is individually tuned, no single optimizer dominates across all settings.
- AdamW and GD often match or exceed Muon on standard low-rank factorization and matrix completion tasks.
- Muon shows a consistent advantage primarily in Non-negative Matrix Factorization (NMF) and specific highly ill-conditioned completion problems.

2. The Role of Conditioning and Spectral Structure

Conditioning Reorders Methods: As the condition number ( $\kappa$ ) increases, the optimal learning rate shifts, and the relative performance of optimizers changes. For example, GD performs well in well-conditioned regimes but fails in extreme ill-conditioning, where AdamW excels.
Spectral Shape Matters: Even at a fixed, extreme condition number ( $\kappa=10^4$ ), the interior distribution of eigenvalues significantly impacts performance. AdamW adapts effectively to heavy-tailed and clustered spectra, whereas Muon underperforms across most spectral shapes in these extreme conditions.

3. Specific Performance in NMF

Muon retains a distinct advantage in NMF tasks. The authors suggest this is because Muon's orthogonalized updates discourage redundant factors and promote more diverse representations, a benefit not observed in unconstrained or symmetric factorization tasks.

4. Well-Behaved Landscapes

On the Gaussian-kernel factorization task (a smooth, well-conditioned landscape), Muon's complex orthogonalization mechanism provides no benefit over a well-tuned AdamW. Both converge to the same global floor, reinforcing that Muon's advantages are not universal but context-dependent.

Significance and Claims

The paper argues that the perceived superiority of Muon in large-scale deep learning may be conflated with other factors such as model scale, architecture, and data characteristics. By isolating the optimizer on controlled problems, the authors provide a more nuanced picture of Muon's capabilities:

Context-Dependent Utility: Muon is not a universally superior optimizer. Its benefits are tied to specific problem structures, particularly those involving non-negativity constraints or specific spectral degeneracies found in NMF.
Evaluation Standards: The work underscores the necessity of evaluating modern optimizers across tuned hyperparameter ranges and diverse conditioning regimes rather than relying on default settings or isolated end-to-end benchmarks.
Theoretical Implications: The results challenge the assumption that spectral-norm steepest descent (the theoretical interpretation of Muon) is inherently superior to coordinate-wise adaptivity (AdamW) in all structured settings. Instead, the "best" approach depends heavily on the specific geometric and spectral features of the objective function.

In conclusion, the authors do not claim Muon is ineffective, but rather that its advantages are problem-dependent. The study advocates for a more rigorous, controlled benchmarking approach to understand when and why spectrum-aware orthogonalization aids optimization.