🤖 machine learning

Muon as a Residual Connection

This paper proposes a mechanistic interpretation of the Muon optimizer as an implicit residual connection that sacrifices immediate gradient fidelity to improve representation preservation for downstream layers, thereby offering a conceptual explanation for its effectiveness and a new design perspective for balancing local descent with downstream usability.

Original authors: Hao Huang

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hao Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Muon is a "Hidden Shortcut"

Imagine you are trying to teach a team of workers (a neural network) to solve a complex puzzle. Usually, you tell each worker exactly what to do based on the immediate mistake they made. This is how standard optimizers work: they focus on fixing the current error as fast as possible.

The paper introduces a new optimizer called Muon. While Muon looks like a simple mathematical trick (it forces the workers' updates to be "orthogonal," or at right angles to their current path), the authors propose a new way to understand why it works so well.

They argue that Muon acts like a hidden "Residual Connection."

In deep learning, a "Residual Connection" is like a conveyor belt that carries the original raw ingredients from the start of the kitchen all the way to the end, skipping the chopping and mixing. This helps the final chef (the last layer) see the original ingredients and decide how to use them, rather than just relying on the chopped-up version.

The authors claim Muon does this implicitly. It doesn't add a physical conveyor belt to the network. Instead, by changing how it updates the weights, it naturally preserves the "shape" of the information so that later layers can still use it easily, even if the current layer didn't fix its own mistake perfectly.

The Trade-Off: Speed vs. Usability

To understand Muon, you have to look at a trade-off between two goals:

Gradient Fidelity (The "Now" Goal): Fixing the current layer's error as fast as possible.
Representation Preservation (The "Later" Goal): Making sure the information passed to the next layer is still easy to understand and use.

The Analogy of the Sculptor:
Imagine a sculptor (the optimizer) trying to carve a statue.

Standard Optimizer (SGD): The sculptor chisels away at the stone exactly where the error is. They get the shape of the current section perfect very quickly. However, they might accidentally carve the stone in a way that makes it hard for the next sculptor to attach the next piece.
Muon: The sculptor takes a slightly different chisel path. They might not get the current section perfect as fast as the first sculptor. However, they carve the stone in a way that leaves a smooth, flat surface for the next sculptor to grab onto.

The Paper's Claim:
Muon is willing to be slightly slower at fixing the current layer's mistake (sacrificing "gradient fidelity") if it means the result is much easier for the next layer to work with (improving "representation preservation").

The Experiments: The Two-Phase Test

The authors tested this idea using a simple, controlled experiment with two layers of math (like two workers in a line).

Phase 1: The Local Test
They asked the first worker to learn a specific pattern.

Result: The standard optimizer (SGD) learned the pattern faster and more accurately than Muon. Muon was "slower" here.
Takeaway: Muon did indeed sacrifice immediate speed on the local task.

Phase 2: The Downstream Test
They froze the first worker's output and asked a second worker to learn how to turn that output into a final target.

Result: Even though the first worker (trained by Muon) was slightly less accurate at their specific job, the second worker learned the final task much faster than when the first worker was trained by SGD.
Takeaway: The "imperfect" output from the Muon-trained worker was actually easier for the next person to use.

The "Tau" Schedule (The Real-World Test)
They also tested what happens when both workers learn at the same time (end-to-end training).

Result: Even though Muon was slower at fixing the first worker's mistakes in the moment, the whole system finished the puzzle faster overall.
Why? The "hidden shortcut" Muon created allowed the second worker to help the first worker later on. It created a feedback loop where a better-conditioned downstream layer made the upstream layer's job easier.

The "Why": The Shape of the Data

Why does Muon leave the data easier to use?

The paper explains that Muon tends to make the "spectrum" (the distribution of importance) of the data flatter.

The Analogy of a Flat Table vs. a Rocky Hill:

Standard Optimizer: Might create a representation that looks like a steep, rocky hill. It's very precise at the top, but if you try to roll a ball (data) down it, it gets stuck or bounces unpredictably.
Muon: Creates a representation that looks like a flat, smooth table. It might not be the "steepest" path down, but a ball can roll across it smoothly and predictably.

Because the "table" is flatter, the next layer (the next worker) doesn't have to struggle to figure out how to process the data. They can just roll the ball right to the finish line.

Summary

The paper concludes that Muon isn't just a mathematical curiosity; it's a mechanism that acts like an implicit residual connection.

It doesn't add a new wire to the network.
It changes the path the network takes to learn.
The result: It accepts a small delay in fixing the current problem to ensure the solution is "user-friendly" for the next part of the system.

In short: Muon teaches the network to be a good teammate, not just a fast individual. It sacrifices a tiny bit of immediate speed to make sure the next person in line can do their job easily, which ultimately makes the whole team finish the race faster.

Technical Summary: Muon as a Residual Connection

Problem Statement

Muon has emerged as a highly effective optimizer for training large neural networks, achieving state-of-the-art performance across architectures ranging from CNNs to Large Language Models (LLMs). Despite its empirical success, existing explanations for Muon's efficacy rely on diverse and sophisticated mathematical frameworks, including spectral-norm steepest descent, non-Euclidean trust-region optimization, Frank–Wolfe methods on spectral-norm balls, long-tail learning, and curvature analysis.

The paper addresses the need for a more intuitive, mechanistic explanation that does not require extensive mathematical training. Specifically, it seeks to explain why Muon works well by reframing the optimization dynamics in terms of representation usability for downstream layers, rather than solely focusing on immediate gradient fidelity or local loss reduction.

Methodology

The authors propose a novel interpretation: Muon functions as an implicit residual connection.

Theoretical Framework

Orthogonalization as Residuality: In standard residual networks, the update is explicit: $x_{l+1} = x_l + \text{Block}(x_l)$ , or in weight space, $W \leftarrow W + I$ (or an orthogonal matrix $R$ ). Muon replaces the standard update $W \leftarrow W + \Delta W$ with an orthogonalized update $W \leftarrow W + \text{Orth}(\Delta W)$ .
The Trade-off: The paper posits a trade-off between gradient fidelity (staying close to the raw gradient for immediate loss reduction) and representation preservation (maintaining a representation structure that is easy for downstream layers to utilize). Orthogonalizing the update sacrifices some immediate gradient fidelity but implicitly injects a residual-like transformation that preserves the upstream representation's usability.

Experimental Validation

To test this hypothesis, the authors conducted controlled experiments using linear models, avoiding the complexities of nonlinear deep networks to isolate the mechanism.

Two-Phase Linear Experiment:
- Phase 1 (Local Optimization): A two-layer linear network ( $\hat{y} = W_2 W_1 x$ $\overset{y}{^} = W_{2} W_{1} x$ ) is trained to match a target transformation $A$ $A$ . $W_2$ $W_{2}$ is frozen, and $W_1$ $W_{1}$ is optimized toward a local target $W_1^*$ $W_{1}^{*}$ .
  - Hypothesis: SGD should outperform Muon here because SGD follows the steepest descent direction for the local objective, while Muon orthogonalizes the update.
- Phase 2 (Downstream Adaptation): The learned $W_1$ $W_{1}$ is frozen, and $W_2$ $W_{2}$ is reinitialized and optimized using SGD to minimize $\|W_2 W_1 - A\|_F$ $∥ W_{2} W_{1} - A ∥_{F}$ .
  - Hypothesis: If Muon's "implicit residual" effect holds, the $W_1$ learned by Muon should be easier for $W_2$ to adapt to, despite being further from the local target in Phase 1.
$\tau$ -Schedule Experiment:
- To simulate end-to-end training, the authors introduced a schedule where layers are updated in alternating segments of length $\tau$ (e.g., $\tau$ steps on $W_1$ , then $\tau$ steps on $W_2$ ).
- This setup allows the observation of how the "local cost" of Muon (during $W_1$ updates) interacts with the "downstream benefit" (during $W_2$ updates) in a continuous training loop.

Key Results

Two-Phase Linear Experiment

Phase 1: As predicted, SGD achieved a lower local reconstruction error ($0.0312$) compared to Muon ($0.0447$) after 600 steps, confirming Muon's sacrifice of local gradient fidelity.
Phase 2: When optimizing $W_2$ with the frozen $W_1$ , the model trained with Muon in Phase 1 required fewer steps (13,321) to reach the target error threshold compared to the model trained with SGD (13,436 steps).
Interpretation: The representation learned by Muon, while suboptimal for the local target, possessed a "flatter" singular spectrum (higher spectral flatness), resulting in a better-conditioned problem for the downstream layer.

$\tau$ -Schedule Experiments

Muon consistently reached the target composition loss faster than SGD across all tested schedules, including fully joint updates ( $\tau=0$ ) and alternating updates ( $\tau > 0$ ).
Dynamics Analysis:
- During "blue segments" (updating only $W_1$ ), SGD often reduced the loss faster initially, reflecting Muon's gradient-fidelity cost.
- During "red segments" (updating only $W_2$ ), the Muon-trained trajectory allowed for faster convergence.
- Crucially, the improved $W_2$ in Muon's trajectory made subsequent $W_1$ subproblems easier to solve, creating a feedback loop that compensated for the initial local slowdown.

Key Contributions

Mechanistic Interpretation: The paper provides a simple, intuitive explanation for Muon's success: it acts as an implicit residual connection. It balances the immediate need for gradient descent with the long-term need for representation preservation.
New Design Perspective: It shifts the optimizer design paradigm from judging updates solely by local loss decrease to considering how updates shape representations for downstream layers.
Empirical Evidence: Through controlled linear experiments, the authors demonstrate that sacrificing local optimization speed can yield faster end-to-end convergence by improving the condition number of the downstream optimization problem.

Significance and Claims

The paper claims that this interpretation complements existing explanations (spectral, curvature, trust-region) rather than replacing them. By framing Muon as an implicit residual connection, the authors suggest that:

Optimizers should balance local and downstream objectives: An effective optimizer need not be judged only by immediate loss decrease on a layer-wise subproblem.
Context matters: The benefits of Muon-like behavior (sacrificing local fidelity for downstream usability) are most valuable when downstream layers have sufficient capacity and training time to exploit better-conditioned representations, particularly in deep chains of composed transformations.
Future Directions: This perspective suggests adaptive optimizer designs that control the strength of orthogonalization or spectral flattening based on layer spectra, conditioning, or downstream progress.

The authors maintain a modest tone, acknowledging that their experiments are deliberately controlled and linear. They do not claim to have proven this mechanism in all modern nonlinear networks but argue that it offers a plausible and intuitive mechanism for Muon's observed empirical advantages.