🤖 machine learning

LionMuon: Alternating Spectral and Sign Descent for Efficient Training

The paper introduces LionMuon, an efficient optimizer that alternates between Lion's and Muon's updates while sharing a single momentum buffer, achieving superior performance and lower computational costs across various model scales compared to existing methods like AdamW, Lion, and Muon.

Original authors: Arman Bolatov, Artem Riabinin, Nikita Kornilov, Andrey Veprikov, Samuel Horváth, Martin Takáč, Aleksandr Beznosikov

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Arman Bolatov, Artem Riabinin, Nikita Kornilov, Andrey Veprikov, Samuel Horváth, Martin Takáč, Aleksandr Beznosikov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a giant, complex robot (a Large Language Model) how to speak by showing it millions of examples. To do this, you need an "optimizer"—a coach that tells the robot how to adjust its brain (its parameters) after every single example it sees.

The problem is that this coach has to make a decision billions of times. If the coach is too slow or too expensive to run, the whole project becomes too costly. If the coach is too cheap but makes bad decisions, the robot learns slowly or gets stuck.

This paper introduces a new coach called LionMuon. It's a clever hybrid that tries to get the best of two very different coaching styles.

The Two Existing Coaches

To understand LionMuon, we first need to meet the two coaches it combines:

The "Sign" Coach (Lion/Signum):
- How it works: This coach is incredibly fast and cheap. It doesn't look at the exact size of the mistake; it just looks at the direction (positive or negative). It's like a GPS that only tells you "Turn Left" or "Turn Right" without calculating the exact distance.
- Pros: It's super efficient. You can run it billions of times without breaking the bank.
- Cons: Because it ignores the "magnitude" (how big the turn needs to be), it sometimes takes a weak or slightly off-course path. It's fast, but not always the most accurate.
The "Spectral" Coach (Muon):
- How it works: This coach is a genius mathematician. It looks at the entire map of the mistake at once and calculates the perfect, strongest possible direction to fix it. It uses complex matrix math (like a high-end GPS calculating the absolute shortest route considering traffic, terrain, and speed limits).
- Pros: It finds the best path very quickly. The robot learns faster per step.
- Cons: It is computationally expensive. Doing this complex math takes a lot of time and energy. If you use this coach for every single step, the training bill skyrockets.

The New Solution: LionMuon

The authors asked a simple question: "Can we have the speed of the Sign Coach but the accuracy of the Spectral Coach?"

Their answer is LionMuon.

Instead of choosing one or the other, LionMuon acts like a smart switchboard. It alternates between the two coaches on a fixed schedule (let's say, every 2 steps):

Step 1: It uses the Muon coach to find that perfect, strong direction.
Step 2: It uses the Lion coach to make a cheap, quick adjustment based on the momentum from the first step.
Step 3: It goes back to Muon, and so on.

The Secret Sauce:
Usually, if you switch coaches, you have to reset their memory or use two different memory banks. LionMuon is clever because it shares a single memory bank between both coaches. This means it doesn't need extra computer memory (RAM) to run. It uses the same amount of memory as the cheap Lion coach, which is half the memory of the standard industry coach (AdamW).

Why is this a big deal?

The paper claims that by alternating these steps, LionMuon gets the best of both worlds:

It's Cheaper: Because it only does the expensive "perfect math" (Muon) once every few steps, the total cost of training drops significantly (about 5% less computing power needed for the same result).
It's Smarter: Even though it skips the expensive math sometimes, the "cheap" steps are guided by the strong direction found in the previous step. The result is that the robot learns faster and reaches a lower error rate than using either coach alone.
It Scales: The researchers tested this on models of different sizes (from 124 million to 720 million parameters). In every case, LionMuon was the winner, reaching the best results with the least amount of computing power.

The Theory (The "Why" it works)

The authors didn't just guess; they did the math. They proved that under certain conditions (specifically when the data is noisy, which is common in AI), there is a "sweet spot" for how often you should switch.

They found that if you switch too often (doing Muon every step), it's too expensive. If you switch too rarely, you lose the accuracy boost. Their math shows that for most modern AI models, switching every 2 steps (one Muon, one Lion) is the perfect balance.

Summary Analogy

Imagine you are hiking up a mountain in the fog.

Lion is a fast runner who just guesses the direction based on the wind. He moves fast but might zigzag.
Muon is a slow, expensive helicopter pilot who uses a thermal camera to find the absolute steepest, safest path.
LionMuon is a team where the helicopter pilot flies once to find the best path, and then the fast runner sprints along that path for a while before the helicopter checks in again.

The result? You reach the summit faster and with less fuel than if you relied on the helicopter for the whole trip or just guessed the whole way.

The Bottom Line: LionMuon is a new, efficient way to train AI that saves money and time while making the AI smarter, without needing extra computer memory.

Technical Summary: LionMuon

Problem Statement

Training Large Language Models (LLMs) is a massive optimization problem where the per-step computational cost (FLOPs) and convergence speed are the primary determinants of the final compute bill. Current optimizers face a trade-off between update quality and computational efficiency:

Sign-based methods (e.g., Lion, Signum): These produce cheap, coordinate-wise sign updates ( $W_{t+1} = W_t - \eta \cdot \text{sign}(\hat{G}_t)$ ). They are computationally inexpensive and memory-efficient but often provide weaker update directions compared to spectral methods.
Spectral methods (e.g., Muon): These compute a matrix sign update via Newton-Schulz iterations ( $W_{t+1} = W_t - \eta \cdot \text{msign}(\hat{G}_t)$ ). While they provide stronger spectral directions that often lead to faster convergence, they incur a significantly higher per-step cost due to multiple matrix multiplications and increased communication overhead in distributed settings (requiring full matrix all-gather).

The central question addressed is whether it is possible to combine the low cost of sign-based updates with the high-quality directions of spectral updates to achieve a superior cost-quality trade-off.

Methodology: LionMuon

The authors propose LionMuon, an optimizer that alternates between Lion-style sign updates and Muon-style spectral updates on a fixed period $P$ .

Core Algorithm

Alternating Schedule: The algorithm performs a Muon step (Newton-Schulz spectral update) every $P$ -th iteration and Lion-style sign updates for the remaining $P-1$ iterations.
Shared Momentum: Both update types share a single dual-Exponential Moving Average (EMA) momentum buffer ( $M_t$ ). The direction $\hat{G}_t$ is computed via Lion's interpolation: $\hat{G}_t = \beta_1 M_{t-1} + (1-\beta_1)G_t$ .
State Efficiency: LionMuon maintains only one momentum buffer, matching the memory footprint of Lion/Signum and being exactly half that of AdamW.
Special Cases:
- $P=1$ : Pure Muon.
- $P=\infty$ : Pure Lion (or Signum if $\beta_1=\beta_2$ ).
- $P \in (1, \infty)$ : The proposed alternating method.
- SignMuon: A variant using single-EMA momentum ( $\beta_1 = \beta_2$ ) with the same alternating schedule.

Theoretical Analysis

The authors provide convergence bounds for LionMuon under heavy-tailed noise, a condition common in LLM training.

Interpolated Constants: The convergence bound depends on period-averaged smoothness ( $\bar{L}$ ) and noise levels ( $\bar{\rho}$ ) that interpolate between the constants of pure Muon (spectral norm) and pure Lion (infinity norm).
Optimal Period: The theory suggests an optimal period $P^*$ exists that minimizes a trade-off factor involving the ratios of smoothness constants ( $L_\infty / L_2$ ) and noise levels ( $\rho_1 / \rho_{nuc}$ ).
Complexity: The derived complexity bounds show that LionMuon can achieve the same accuracy as pure Muon or Lion with fewer operations under specific configurations, particularly when the trade-off factor $\phi(P) < 1$ .

Key Contributions

SignMuon: An alternating optimizer performing a Muon step every $P$ iterations and a Signum-style sign step otherwise. The authors show that even this single-EMA variant improves the loss-vs-FLOPs trade-off over pure Muon at $P=2$ .
LionMuon: The main contribution, replacing Signum's momentum with Lion's dual-EMA rule. It preserves the cost savings of SignMuon while further improving convergence. It requires only one additional integer hyperparameter ( $P$ ) and is a near drop-in replacement for Lion.
Theoretical Bounds: Sharp complexity bounds under heavy-tailed noise that explicitly model the interpolation between Muon and Lion regimes, predicting the compute-optimal period $P$ .
Empirical Validation: Extensive experiments across multiple datasets (FineWeb, SlimPajama, WikiText-103) and architectures (GPT-2, LLaMA) at scales of 124M, 355M, and 720M parameters.

Experimental Results

124M Scale: LionMuon (specifically with $P=2$ ) Pareto-dominates Muon, Lion, Signum, and AdamW on all tested datasets and architectures. It achieves lower validation loss with approximately 5.5% fewer total training FLOPs compared to pure Muon.
Scaling (355M & 720M): The convergence-quality advantage persists at larger scales. At 355M, LionMuon and SignMuon ( $P=2$ ) outperform pure Muon. At 720M, SignMuon ( $P=2$ ) still beats pure Muon, though the dual-EMA advantage of LionMuon over SignMuon did not reproduce in this specific under-trained budget.
Distributed Training: The method offers structural communication savings. Since Lion steps are element-wise and local to shards, setting $P=2$ halves the frequency of the expensive all-gather operations required by Muon's Newton-Schulz step, regardless of batch size.

Significance and Claims

The paper claims that LionMuon offers a structurally superior approach to LLM training by decoupling the frequency of expensive spectral updates from the iteration count.

Efficiency: It retains the effectiveness of Muon steps while cutting the averaged iteration cost, matching the memory efficiency of sign-based methods.
Robustness: The method is robust across different model sizes and datasets, consistently defining the Pareto frontier for loss-vs-FLOPs.
Theoretical Alignment: The empirical results align with the theoretical predictions regarding the optimal period $P$ , suggesting that intermediate values (e.g., $P=2$ or $P=5$ ) often provide the best balance between the expensive spectral direction and the cheap sign direction.

The authors conclude that LionMuon is a practical, drop-in extension for modern LLM training that strictly improves upon the current state-of-the-art (Muon) in terms of both convergence quality and computational efficiency. Future work is suggested for adaptive $P$ selection and scaling beyond 720M.