🤖 machine learning

When Does Muon Help Agentic Reinforcement Learning?

This paper demonstrates that applying the Muon optimizer specifically to hidden weight matrices during Group-in-Group Policy Optimization significantly outperforms AdamW in sparse-reward agentic RL tasks, with performance gains heavily dependent on the interplay between the advantage estimator, learning rate, and policy optimization strategy.

Original authors: Kai Ruan, Jinghao Lin, Zihe Huang, Ziqi Zhou, Qianshan Wei, Xuan Wang, Hao Sun

Published 2026-07-20

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Kai Ruan, Jinghao Lin, Zihe Huang, Ziqi Zhou, Qianshan Wei, Xuan Wang, Hao Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to navigate a giant, messy house to find a hidden treasure. The robot is a "brain" made of code, and every time it takes a step, it gets a tiny nudge from a teacher telling it whether that step was good or bad. This is the world of Reinforcement Learning (RL), where an agent learns by trial and error. The "teacher" in this story is a mathematical tool called an optimizer, which decides how much the robot's brain should change after each mistake or success. For years, the most popular teacher has been a method called AdamW, which is like a careful, steady coach who adjusts the robot's steps based on how loud the feedback was.

Recently, a new, flashier coach called Muon arrived. Muon is famous for being incredibly fast and efficient when teaching robots from scratch (a process called pre-training), often doing the job with half the energy. But here's the big question: Does Muon work just as well when teaching a robot that has already learned the basics and is now trying to master a specific, difficult task? Some early reports suggested Muon might be too aggressive for this "fine-tuning" stage, causing the robot to forget what it knew or get confused by the noise. This paper dives into that mystery, testing whether Muon can be the secret weapon for teaching agents to solve complex, long-horizon puzzles where rewards are rare and hard to find.

The Great Optimizer Showdown

In the lab, the researchers set up a high-stakes race. They took a small but capable AI agent (based on the Qwen2.5-0.5B-Instruct model) and tasked it with solving puzzles in ALFWorld, a simulated environment where the agent must perform household chores like picking up objects, cleaning, or heating things up. The catch? The agent only gets a "success" reward at the very end of a long sequence of actions, and it gets a penalty for doing anything wrong along the way. It's like trying to solve a maze where you only hear "You won!" if you reach the exit, but you get a "Buzz!" for every wrong turn.

To see if Muon could help, the researchers didn't just throw the robot into the maze; they gave it three different types of "feedback coaches" (called advantage estimators) to interpret its performance:

GRPO: This coach only looks at the final result. "Did you win the game? Yes? Great job! No? Try again." It ignores the individual steps.
GiGPO: This coach is smarter. It looks at the final result but also groups together moments where the robot faced the exact same situation multiple times. It asks, "When you were standing in front of the fridge, did you usually open it correctly?"
GraphGPO: The most sophisticated coach, which maps out the entire journey as a graph, analyzing how close every single step was to the goal.

The Surprising Discovery

The team ran the experiments, pitting the old reliable AdamW coach against the new Muon coach. They kept everything else exactly the same, only swapping the optimizer for the robot's "hidden" brain weights (the complex internal connections), while leaving the basic parts (like the dictionary of words) on AdamW.

The results were a mix of "meh" and "wow," depending entirely on which feedback coach was used.

The "Wow" Moment: GiGPO
When they used the GiGPO coach, Muon turned the robot into a super-learner.

With the standard AdamW coach, the robot succeeded in about 29% of the final test runs.
With Muon, that number skyrocketed to 54.6%.
That is an 88% relative improvement.
Even more impressively, while the AdamW robot completely crashed and failed to solve any tasks after a certain point, the Muon robot kept learning and succeeding all the way through 200 updates.

The "Meh" Moment: GRPO
When they used the simpler GRPO coach (which only cares about the final win), Muon still helped, but not as dramatically. It improved success from 16.1% to 26.8%. It was a win, but the robot didn't learn as fast or as well as it did with GiGPO.

The "Almost There" Moment: GraphGPO
With the super-smart GraphGPO coach, the robot was already doing very well with AdamW (around 81% success). Muon pushed it slightly higher to 90.1% at a specific learning rate, and it learned to reach that success level about 30 to 60 steps earlier than AdamW. However, because the robot was already near the top, the gap between the two coaches narrowed as they got closer to the finish line.

Why Did Muon Win? (And Why It Didn't Always)

The researchers had a theory about why this happened. They suspected it had to do with noise.

Imagine the robot's brain is trying to find a path through a foggy forest.

AdamW is like a hiker who follows the strongest, loudest path. If the fog is thick (high noise), it might get stuck or wander off.
Muon is like a hiker who flattens the terrain, making the weak, quiet paths just as visible as the loud ones. This is great if the quiet paths actually lead somewhere useful.

The paper suggests that in the GRPO setup (single-turn, low-reward tasks), the "quiet paths" are mostly just random noise. Muon's flattening effect amplifies this noise, which is why it sometimes fails or struggles in other studies. However, in the GiGPO setup, the "step-level" feedback acts like a lighthouse. It filters out the noise and highlights the truly useful directions. Because GiGPO provides clearer signals about which specific steps were good, Muon's ability to amplify weak signals becomes a superpower rather than a curse.

What the Paper Rules Out

It's important to note what this study didn't find. The researchers tested if the success was just because Muon was using a higher "learning rate" (a bigger step size). They tried cranking up AdamW's speed to match Muon's pace, but the robot immediately crashed and failed. This proves that Muon's success wasn't just about taking bigger steps; it was about how it calculated those steps.

Also, the paper doesn't claim Muon is the perfect solution for everything. In fact, it explicitly notes that in other types of tasks (like single-turn math problems), Muon has failed in the past. The success here is specific to long-horizon, sparse-reward tasks where the feedback is structured carefully (like with GiGPO).

The Bottom Line

This paper suggests that Muon is a powerful tool for agentic Reinforcement Learning, but only when paired with the right kind of feedback.

If you are teaching an AI to solve complex, multi-step puzzles, simply swapping in Muon might not be enough. You need a feedback system (like GiGPO) that breaks the problem down into smaller, clearer pieces. When you combine Muon's ability to see the "quiet" useful paths with GiGPO's ability to filter out the noise, the AI learns significantly faster and solves more puzzles.

The authors admit this is an "exploratory" study using a single seed (one run of the experiment), so more testing is needed to see if this pattern holds up across different models and tasks. But the results are promising: by tuning the optimizer and the feedback system together, we might be able to build AI agents that are much better at navigating the messy, real world.

Technical Summary: When Does Muon Help Agentic Reinforcement Learning?

Problem Statement

While Muon has demonstrated competitive performance against AdamW in large-scale pre-training (matching final loss with roughly 52% of the training FLOPs at billion-parameter scales), its efficacy in Reinforcement Learning (RL) post-training remains an open question. Existing literature presents mixed evidence: some reports suggest minor improvements, while others indicate that vanilla Muon fails under RLVR (Reinforcement Learning from Verifiable Rewards) with GRPO-style objectives, attributing failure to spectral whitening amplifying noise in low signal-to-noise ratio (SNR) policy gradients. Furthermore, studies on supervised fine-tuning (SFT) suggest that switching optimizers from Adam to Muon can disrupt learning-forgetting tradeoffs, particularly at higher learning rates.

The core problem addressed is determining whether Muon's viability in RL post-training depends on the structure of the advantage estimator, specifically in long-horizon, sparse-reward agentic RL scenarios where credit assignment is challenging.

Methodology

The authors conduct a controlled, single-seed study comparing vanilla Muon against AdamW on the ALFWorld environment using a Qwen2.5-0.5B-Instruct model. The experimental design isolates the optimizer's effect by varying the advantage estimator structure while holding other hyperparameters constant.

Experimental Setup

Model: Qwen2.5-0.5B-Instruct.
Environment: ALFWorld (embodied household tasks with sparse terminal rewards and invalid action penalties).
Optimizers:
- Muon: Applied exclusively to 2D hidden weight matrices (attention and MLP weights) using Newton–Schulz iterations for approximate spectral normalization. Non-matrix parameters (embeddings, norms) use AdamW.
- AdamW: Applied to all parameters as the baseline.
Advantage Estimators: Three distinct credit-assignment structures are tested:
1. GRPO: Uses episode-level advantages only.
2. GiGPO (Group-in-Group Policy Optimization): Combines episode-level advantages with a step-level term derived from grouping actions taken from repeated "anchor states" across trajectories.
3. GraphGPO: Aggregates rollouts into a unified state-transition graph, assigning transition-level credit based on successor-state distance to the goal.

Key Controls

Learning Rate Sensitivity: The study tests Muon at $3 \times 10^{-5}$ and $10^{-5}$ , comparing them against AdamW baselines at $10^{-6}$ and high-rate controls at $10^{-5}$ and $3 \times 10^{-5}$ .
Ablation: A matched $2 \times 2$ ablation on GiGPO disables the step-level credit term ( $\omega=0$ ) to isolate the contribution of step-level advantages.
Metrics: Final-window validation success (steps 175–200), normalized validation AUC (learning efficiency), and training dynamics (KL loss, gradient norms, entropy).

Key Results

1. Significant Gains with GiGPO

Under GiGPO, applying Muon to hidden weight matrices yields the most dramatic improvement:

Success Rate: Final-window validation success increases from 0.290 (AdamW) to 0.546 (Muon), an 88% relative improvement.
Learning Efficiency: Muon reaches 0.5 success 20 steps earlier than AdamW (at step 155 vs. 175).
Stability: While high-rate AdamW controls ( $10^{-5}$ and $3 \times 10^{-5}$ ) lose all validation success after the initial steps (dropping to 0.000), Muon at $3 \times 10^{-5}$ retains task success through all 200 updates without catastrophic failure.

2. Variable Performance Across Estimators

The benefit of Muon is not uniform across all credit-assignment structures:

GRPO: Muon improves late-window success (0.161 to 0.268) but shows minimal gain in full-trajectory learning efficiency (normalized AUC increases only from 0.08 to 0.09).
GraphGPO: At $3 \times 10^{-5}$ , the late-window gap narrows as curves approach saturation (0.83 vs. 0.81). However, at a lower rate ( $10^{-5}$ ), Muon significantly improves learning efficiency, raising normalized AUC from 0.40 to 0.56 and reaching success thresholds 30–60 updates earlier.

3. Learning Rate and Step-Level Credit

Learning Rate: The gains are not merely a result of a larger effective step size. High-rate AdamW controls fail completely, whereas Muon succeeds at $3 \times 10^{-5}$ .
Step-Level Credit: The ablation study shows that Muon improves performance both with ( $\omega=1$ ) and without ( $\omega=0$ ) step-level credit. However, the combination of step-level credit and Muon yields the largest gains, suggesting they are complementary.

Key Contributions

Controlled Single-Seed Study: Provides the first matched comparison of vanilla Muon vs. AdamW in long-horizon, sparse-reward agentic RL, demonstrating an 88% relative improvement in success rates under GiGPO.
Learning Rate Decoupling: Shows that the performance gains are not reproducible by simply increasing the AdamW learning rate, as high-rate AdamW controls fail to retain task success.
Estimator-Dependent Analysis: Establishes that Muon's efficacy varies by advantage estimator, with the largest gains observed in estimators incorporating step-level or transition-level credit (GiGPO and GraphGPO) compared to pure episode-level estimators (GRPO).
Gradient SNR Hypothesis: Proposes that Muon's success depends on the signal-to-noise ratio of the gradient spectrum. The authors conjecture that estimators providing per-state contrasts (like GiGPO) may improve the reliability of weak gradient directions, allowing Muon's spectral flattening to be beneficial rather than amplifying noise.

Significance and Claims

The paper claims that Muon can significantly benefit agentic RL, but its utility is contingent on the joint design of the policy optimizer and the advantage estimator. The results challenge the notion that Muon is universally unsuitable for RL, suggesting instead that its failure in previous RLVR studies may be due to the specific structure of episode-level advantage estimation in low-SNR regimes.

The authors emphasize that their findings are exploratory and based on a single seed and a specific model size (0.5B). They do not claim universal applicability but rather motivate further research into the interplay between optimizers and credit-assignment mechanisms. The study suggests that future work should investigate gradient SNR directly and explore whether combining Muon with variance-normalized or high-pass spectral filtering (as proposed in other works like Hopper and Pion) could further stabilize training.

The paper concludes that while Muon shows promise, broader validation across multiple seeds, models, and environments is required to determine the consistency of these patterns.