🤖 machine learning

ISO: An RLVR-Native Optimization Stack

This paper introduces ISO, a novel optimization framework for Reinforcement Learning with Verifiable Rewards (RLVR) that leverages spectral inheritance to optimize only the singular frames of model weights while keeping the base spectrum fixed, thereby enabling highly efficient online training and effective offline merging of specialist models without requiring additional data or gradient updates.

Original authors: Hanqing Zhu, Wenyan Cong, Zhizhou Sha, Sagnik Mukherjee, Xinyuan Song, David González-Martínez, Xiaoxia Wu, Yuandong Tian, Shiwei Liu, David Z. Pan, Zhangyang "Atlas" Wang

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hanqing Zhu, Wenyan Cong, Zhizhou Sha, Sagnik Mukherjee, Xinyuan Song, David González-Martínez, Xiaoxia Wu, Yuandong Tian, Shiwei Liu, David Z. Pan, Zhangyang "Atlas" Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to think better. You don't just give it a textbook; you let it try things, and when it gets the answer right, you give it a high-five (a "reward"). This process is called Reinforcement Learning with Verifiable Rewards, or RLVR for short. It's like a video game where the robot learns by playing, and the game only tells it "good job" or "try again" at the very end of a level.

For a long time, scientists have been using the same old tools to update the robot's brain after these high-fives. They treat the robot's brain like a giant, messy spreadsheet of numbers and just tweak every single number a tiny bit. But here's the catch: the robot's brain is already pretty good at the basics before it starts playing the game. Maybe the old tools are too clumsy, changing things that don't need changing, or missing the specific parts that actually need a nudge. The big question is: Is there a smarter, more precise way to update the robot's brain that respects what it already knows while teaching it new tricks?

This paper, titled "ISO: An RLVR-Native Optimization Stack," dives into that exact question. The authors discovered a hidden pattern in how these robots learn. They found that when a robot learns a new skill through RLVR, it doesn't actually rewrite its entire brain. Instead, it keeps the "volume knobs" (the mathematical values that control the strength of its connections) exactly the same as they were before. It only changes the "directions" (the angles and orientations of those connections).

Think of it like a dance troupe. The "volume knobs" are the number of dancers and their energy levels, which stay the same. The "directions" are the choreography. The paper shows that to learn a new dance, the troupe doesn't need to hire new people or change how energetic they are; they just need to rearrange the steps. The authors call this "Spectral Inheritance." They proved that you can keep the energy levels (the spectrum) fixed and still get amazing results just by tweaking the choreography (the frames).

Based on this discovery, they built a new toolkit called ISO (Isospectral Optimization). It has two main tricks. First, there's "ISO-Merger," which is like a magic blender. If you have three different robots, each a master at a different skill (one is a coding wizard, one is a math genius, and one is a tool expert), you can smash them together into one super-robot without needing to retrain them or feed them new data. It just mixes their choreography while keeping their original energy levels intact.

Second, there's "ISO-Optimizer," which is a faster way to train a robot from scratch. Instead of blindly tweaking the whole brain, it only adjusts the choreography while keeping the energy levels locked. The paper shows this is a huge time-saver. For example, on a specific 8-billion-parameter model, the new method reached the same high score as the old method in just 100 steps, whereas the old method needed 270 steps to get there. Even better, it kept improving and got an even higher score by step 210.

The authors are very careful to say this isn't magic; it's math. They tested it rigorously. They showed that if you try to change only the energy levels (the volume knobs) without changing the choreography, the robot doesn't learn much. But if you keep the energy levels fixed and change the choreography, the robot learns fast. They also checked that this pattern holds true even when the robot learns a second, totally different skill later on.

So, the main takeaway is simple: When teaching these AI models new reasoning skills, don't try to rebuild their entire foundation. Just inherit the solid foundation they already have and focus on rearranging the steps. It's a more efficient, faster, and surprisingly effective way to teach AI how to think.

Technical Summary: ISO: An RLVR-Native Optimization Stack

Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has become a primary mechanism for scaling the reasoning capabilities of large language models (LLMs). However, the optimization layer responsible for converting reward feedback into weight-space updates remains poorly understood. Current RLVR implementations largely inherit optimization strategies (e.g., AdamW, Muon) and parameterizations from pre-training, despite fundamental structural differences between the two regimes. Pre-training relies on dense, token-level supervision, whereas RLVR adapts capable policies using sparse, outcome-level rewards.

Recent analyses suggest that RLVR updates are sparse and off-principal, yet existing frameworks do not explicitly separate what is reused from what is rewritten during adaptation. This paper investigates the "missing layer" of RLVR optimization by analyzing the singular structure of model weights to determine if the base model's spectral properties are functionally necessary or merely incidental.

Methodology: Spectral Inheritance and Isospectral Optimization (ISO)

The paper introduces Isospectral Optimization (ISO), a framework built on the principle of spectral inheritance. This principle posits that RLVR can acquire new behaviors by modifying the input and output singular frames ( $U, V$ ) of weight matrices while reusing the base model's singular value spectrum ( $\Sigma$ ).

1. Spectral Inheritance Analysis

The authors formalize the observation that unconstrained RLVR checkpoints remain close to the "fixed-spectrum family" of their base weights.

Near-Isospectrality: Empirical analysis shows that the distance between learned RLVR weights and the fixed-spectrum family of the base model is minimal (approx. 3% of the total checkpoint displacement), significantly lower than in Supervised Fine-Tuning (SFT).
Functional Reusability: Through interventions, the authors demonstrate that the small spectral changes observed in unconstrained RLVR are not functionally necessary.
- Restoration: Restoring the base spectrum ( $\Sigma_0$ ) after training while retaining learned frames preserves most acquired performance.
- Fixed-Spectrum Training: Keeping $\Sigma_0$ fixed throughout training and optimizing only the frames ( $U, V$ ) yields strong gains, often outperforming standard weight-space optimizers.
- Spectrum-Only Control: Conversely, updating only the spectrum while freezing frames yields limited improvement.
Frame Adaptability: Structural analysis reveals that while the spectrum can remain fixed, both the input ( $V$ ) and output ( $U$ ) singular frames must remain adaptable. Restricting adaptation to only one subspace or remixing only within existing spans leaves a substantial portion of the checkpoint change unexplained.

2. The ISO Framework

ISO operationalizes these findings as an optimization stack with two complementary instantiations:

A. ISO-Merger (Offline)

A data-free method for composing shared-base RL experts.

Mechanism: Instead of using Euclidean task vectors or on-policy distillation, ISO-Merger aligns the singular frames of multiple experts to a shared base spectrum. It projects expert frame changes onto the Stiefel tangent space of the base, masks unstable trailing modes, and aggregates them using a ridge-stabilized linear combination to target unit self-retention.
Process: Project $\to$ Mask $\to$ Merge $\to$ Retract. The final model is reconstructed as $W^* = U^* \Sigma_0 (V^*)^\top$ , ensuring it belongs to the fixed-spectrum family.
Requirement: Requires no post-merge data, rollouts, gradient updates, or distillation.

B. ISO-Optimizer (Online)

A method for online RLVR training that applies standard optimizers to frame variables.

Mechanism: Given a base optimizer (e.g., AdamW, Muon), ISO-Optimizer applies the update rule to the singular frame variables ( $U, V$ ) while keeping the base spectrum $\Sigma_0$ fixed.
Gradient Flow: Gradients are computed with respect to the reconstructed weight matrix $W = U \Sigma_0 V^\top$ and backpropagated to $U$ and $V$ via the chain rule.
Feasibility: After each update, the factors are projected back onto the Stiefel manifold using polar retraction (SVD-based) to ensure orthonormality.
Overhead: The polar retraction adds approximately 7% to the end-to-end step time, which is non-dominant in RLVR systems where rollout generation is the bottleneck.

Key Results

1. Offline Expert Composition (ISO-Merger)

Experiments on Qwen2.5-7B and DeepSeek-R1-Distill-Qwen-1.5B merging coding, tool-use, and memory experts show:

ISO-Merger achieves the strongest aggregate performance among data-free merging methods (e.g., Task Arithmetic, TIES, OrthoMerge).
On Qwen2.5-7B, it reaches an average score of 63.80, outperforming the best baseline (62.88).
On DeepSeek-R1-Distill-Qwen-1.5B, it reaches 44.38, outperforming the best baseline (43.52).
It recovers complementary capabilities without requiring additional training data or rollouts.

2. Online RLVR Training (ISO-Optimizer)

Experiments on mathematical reasoning (AIME, AMC, Olympiad) and coding (LiveCodeBench) across model scales (1.5B to 8B) show:

Accuracy Gains: ISO-AdamW consistently achieves higher aggregate accuracy than standard AdamW and Muon baselines.
Training Efficiency: ISO-Optimizer reaches matched scores with substantially fewer training steps.
- On Qwen3-4B-Base, ISO-AdamW matches the best AdamW run's final accuracy with 2.2 $\times$ fewer steps.
- On Qwen3-8B-Base, ISO-AdamW reaches an accuracy of 0.495 in 100 steps, whereas standard AdamW requires 270 steps to reach the same level (a 2.7 $\times$ reduction).
Generalization: The benefits persist across different optimizers (ISO-Muon) and tasks (coding vs. math), confirming that the fixed-spectrum constraint is a robust inductive bias for RLVR.

Significance and Claims

The paper claims to provide a concrete answer to the "missing optimization layer" in RLVR. Rather than inheriting pre-training optimization wholesale, the authors argue for designing post-training around the structure of reward-driven adaptation: inherit the spectrum, optimize the frames.

Key contributions include:

Formalization of Spectral Inheritance: Demonstrating that RLVR functionally reuses the base model's weight spectra, a property distinct from SFT.
RLVR-Native Stack: Introducing ISO as a framework that unifies offline expert composition and online training under a single fixed-spectrum principle.
Practical Efficiency: Showing that constraining optimization to the fixed-spectrum family improves convergence speed and final accuracy without expanding the feasible model class, effectively acting as a strong inductive bias for RLVR.

The authors emphasize that ISO is not merely a new optimizer but a structural re-parameterization that aligns the optimization trajectory with the intrinsic geometry of RLVR adaptation, offering a pathway to more efficient and stable post-training for reasoning models.