🤖 machine learning

Can Muon Fine-tune Adam-Pretrained Models?

This paper investigates the performance degradation caused by switching from Adam to Muon for fine-tuning pretrained models, demonstrating that the resulting optimizer mismatch disrupts learned knowledge and can be effectively mitigated by constraining update strength through methods like LoRA.

Original authors: Xingyu Qu, Peigeng Huang, Samuel Horvath

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xingyu Qu, Peigeng Huang, Samuel Horvath

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Wrong Tool" for the Job

Imagine you have a highly skilled artisan (the AI model) who spent years learning a craft using a specific set of tools: a hammer and chisel (this is the Adam optimizer). They built a beautiful, complex sculpture (the pre-trained model) that knows how to speak, write, and reason.

Now, a new, faster, and more efficient tool arrives: a laser cutter (this is the Muon optimizer). The paper's authors wanted to see if they could use this laser cutter to fine-tune (make small adjustments to) the sculpture the artisan built with the hammer.

The Discovery: When they tried to use the laser cutter on the hammer-built sculpture, the results were terrible. The sculpture started to crack, lose its shape, or perform worse than before.

Why? The paper explains that the hammer and the laser cutter "think" differently.

The Hammer (Adam) builds structures based on individual, tiny adjustments (like chipping away one grain of wood at a time).
The Laser (Muon) builds structures by looking at the whole block and smoothing out the entire surface at once.

When you try to use the laser on a structure built by the hammer, the laser's "smooth" adjustments clash with the "chipped" texture of the original work. This clash is called an optimizer mismatch. It disrupts the knowledge the model already learned, causing it to "forget" things or perform poorly.

The Solution: The "Sticky Note" Method (LoRA)

The authors asked: Is there a way to use the laser cutter without breaking the hammer-built sculpture?

They found the answer in a technique called LoRA (Low-Rank Adaptation).

The Analogy:
Instead of trying to carve directly into the original stone sculpture (Full Fine-Tuning), imagine you take a piece of clear, flexible plastic (LoRA) and tape it over the sculpture.

You leave the original stone sculpture exactly as it is (frozen).
You only carve your new adjustments into the plastic sheet.
The laser cutter (Muon) works on the plastic sheet.

Why this works:
Because the laser cutter is only touching the new plastic sheet, it doesn't have to fight against the old hammer-chipped texture of the stone underneath. The plastic sheet is flexible enough to adapt to the laser's style without breaking the foundation.

The paper shows that when they used this "plastic sheet" method (LoRA) with the laser cutter (Muon), the results were just as good as, or sometimes even better than, using the hammer (Adam). The mismatch problem disappeared.

Key Findings in Plain English

The Mismatch is Real: If you take a model trained with Adam and try to fine-tune it directly with Muon, it gets worse. It's like trying to drive a car with a steering wheel designed for a different car; the car won't handle well.
The "Plastic Sheet" Fixes It: By using LoRA (the plastic sheet), Muon can fine-tune Adam-trained models effectively. The performance gap between the two methods vanishes.
Less is More: The paper found that the more you try to change the original model (Full Fine-Tuning), the more the mismatch hurts. The less you change (using a thin plastic sheet/LoRA), the better the laser cutter works.
Forgetting Less: When the laser cutter tried to carve directly into the stone (Full Fine-Tuning), the model "forgot" its original knowledge (like how to do math or common sense). When using the plastic sheet (LoRA), the model remembered its original skills much better.
Efficiency: Muon is already known to be faster and use less memory than Adam during the initial training phase. This paper proves it can also be used for fine-tuning if you use the LoRA method, saving even more memory because you don't need to store as many "state" variables.

The Takeaway

You don't need to throw away all the models trained with the "Hammer" (Adam). You can still use the faster, more efficient "Laser" (Muon) to improve them, but you have to be gentle. Instead of trying to reshape the whole model, just add a small, flexible layer on top (LoRA). This allows the new tool to work without breaking the old work.

Technical Summary: Can Muon Fine-tune Adam-Pretrained Models?

Problem Statement

Muon (MomentUm Orthogonalized by Newton-Schulz) has emerged as a highly efficient optimizer for pretraining large language models (LLMs), offering approximately $2\times$ compute efficiency over Adam and reduced memory usage by eliminating the second-moment estimate. However, its practical application to fine-tuning remains limited. The primary obstacle is the optimizer mismatch problem: most publicly available open models are pretrained with Adam, and naively switching to Muon for fine-tuning leads to degraded performance. Conversely, using Adam to fine-tune Muon-pretrained models also yields suboptimal results. This incompatibility severely restricts Muon's adoption in the dominant fine-tuning paradigm, despite its pretraining successes.

Methodology and Analysis

The authors investigate this mismatch through controlled experiments and theoretical analysis, focusing on the distinct implicit biases of Adam and Muon.

Implicit Bias Characterization:
- Adam utilizes element-wise preconditioning based on first and second moments, converging to solutions that maximize the max-norm margin ( $\|W\|_{\max}$ ).
- Muon applies matrix-level preconditioning via Newton-Schulz orthogonalization, converging to solutions that maximize the spectral-norm margin ( $\|W\|_2$ ).
- Theoretical analysis on a simplified linear regression problem demonstrates that these optimizers converge to fundamentally different solutions ( $W^*_{\max}$ vs. $W^*_2$ ). Empirically, Muon-trained weights exhibit higher stable rank and more uniform singular value distributions compared to Adam-trained weights.
Mechanism of Degradation:
- The mismatch disrupts pretrained knowledge. When an optimizer with a different implicit bias is applied, it forces the model weights into a structural direction incompatible with the pretraining phase.
- Experiments show that mismatched fine-tuning increases sensitivity to update strength. Stronger updates cause greater disruption to the pretrained structure, leading to higher perplexity and more severe catastrophic forgetting.
Proposed Mitigation: LoRA:
- The authors hypothesize that constraining the extent of updates can mitigate this mismatch. They employ Low-Rank Adaptation (LoRA), which freezes the pretrained weights ( $W_0$ ) and restricts updates to a low-rank subspace ( $\Delta W = BA$ ).
- This approach limits the magnitude of weight changes, preventing the optimizer from aggressively altering the pretrained structure. Theoretical analysis suggests that the "mismatch inflation" (the gap between matched and mismatched performance) is bounded by the LoRA rank $r$ , vanishing as $r \to 1$ .

Key Contributions

Reproduction and Analysis: The paper reproduces the optimizer mismatch phenomenon at accessible scales (NanoChat, 561M parameters) and links it to the distinct implicit biases (max-norm vs. spectral-norm) of Adam and Muon.
LoRA as a Solution: The authors demonstrate that LoRA effectively mitigates the mismatch. By constraining updates, LoRA allows Muon to fine-tune Adam-pretrained models with performance comparable to, or exceeding, LoRA-Adam.
Comprehensive Validation: The findings are validated across:
- Natural Language Understanding (NLU): GLUE benchmark with T5-Base.
- Natural Language Generation (NLG): Instruction tuning of Llama 2-7B/13B on math, code, and commonsense reasoning.
- Computer Vision: Fine-tuning CLIP ViT-B/32 on six image classification tasks.
Rank and Forgetting Studies:
- Rank Sensitivity: The benefit of LoRA-Muon is most pronounced at low-to-moderate ranks. As rank increases (approaching full fine-tuning), the mismatch re-emerges, and performance degrades relative to Adam.
- Catastrophic Forgetting: Full fine-tuning with Muon on Adam-pretrained models causes more severe forgetting of pretrained knowledge than Adam. LoRA significantly reduces this forgetting, preserving the pretrained structure.
LoRA Variants: The study evaluates existing LoRA variants (e.g., rsLoRA, PiSSA, LoRA-One). It finds that variants designed to make LoRA updates closer to full fine-tuning (like PiSSA) do not necessarily benefit Muon and may even exacerbate the mismatch, suggesting a need for Muon-specific adaptations.

Results

Performance Gap Reduction: In full fine-tuning, Muon consistently underperforms Adam on Adam-pretrained models (e.g., +0.009 to +0.023 perplexity gap on WikiText-2). Under LoRA, this gap shrinks significantly (by 39% for Muon-pretrained and 78% for Adam-pretrained models).
Superiority in LoRA: Across NLU, NLG, and vision tasks, LoRA-Muon matches or outperforms LoRA-Adam. For instance, on the GLUE benchmark, LoRA-Muon-PE achieved the highest average accuracy (89.20%), surpassing both LoRA-Adam (88.93%) and Full-Adam (89.14%).
Efficiency: While Muon incurs a modest per-step overhead due to Newton-Schulz iteration (1.1–1.2 $\times$ slower than Adam in LoRA settings), its memory efficiency (storing only one momentum buffer vs. two for Adam) remains a practical advantage, saving ~50% of optimizer state memory.

Significance and Claims

The paper claims to provide the first in-depth analysis of the optimizer mismatch problem between Adam and Muon. Its primary significance lies in enabling the practical use of Muon for fine-tuning the vast ecosystem of Adam-pretrained models.

The authors conclude that:

Optimizer mismatch is structural: It arises from conflicting implicit biases that disrupt pretrained knowledge when update strength is unconstrained.
Constrained updates are the key: Techniques like LoRA, which limit the magnitude of weight changes, effectively neutralize this mismatch.
Practical Recommendation: Practitioners can use Muon as a drop-in replacement for Adam in LoRA fine-tuning to gain memory efficiency and competitive performance without suffering from the mismatch penalties observed in full fine-tuning.

The paper modestly notes that while LoRA mitigates the issue, existing LoRA variants optimized for Adam do not automatically transfer to Muon, suggesting future work is needed to develop Muon-specific adaptations. Additionally, the theoretical characterization of the mismatch remains an open question.