💬 NLP

You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories

This paper introduces RELEX, a compute-efficient method that leverages the discovery that RLVR weight trajectories are highly predictable and low-rank to extrapolate future model checkpoints via linear regression, achieving performance comparable to or exceeding full training with only a fraction of the steps by filtering out stochastic optimization noise.

Original authors: Zhepei Wei, Xinyu Zhu, Wei-Lin Chen, Chengsong Huang, Jiaxin Huang, Yu Meng

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhepei Wei, Xinyu Zhu, Wei-Lin Chen, Chengsong Huang, Jiaxin Huang, Yu Meng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Training AI is Like Climbing a Mountain

Imagine you want to teach a large language model (an AI) to solve complex math problems. Currently, the best way to do this is a process called RLVR (Reinforcement Learning with Verifiable Rewards).

Think of RLVR as sending the AI on a very long, expensive hike up a mountain to reach the peak of "math genius."

The Cost: This hike takes days of powerful computer time (GPU hours).
The Process: The AI takes thousands of small steps, checking its work at every checkpoint.
The Goal: You want the AI to reach the very top (the best performance), but the hike is so long and costly that you'd love to stop halfway and just "guess" where the top is.

The Discovery: The Hike is Surprisingly Straight

The researchers in this paper noticed something strange about how the AI learns. They looked at the "footprints" (the changes in the AI's brain weights) left behind during this hike.

They found two amazing things:

The Path is a Straight Line: Even though the AI is moving through a massive, complex 3D space, its actual path of improvement is incredibly simple. It's like walking through a dense forest but only moving in one single, straight direction.
The Pace is Predictable: The speed at which the AI improves along this straight line is almost perfectly linear. If you draw a graph of its progress, it looks like a ruler-straight line, not a wiggly, chaotic curve.

The Analogy: Imagine you are driving a car through a foggy city. Usually, you expect the road to twist and turn. But the researchers found that for this specific type of AI training, the road is actually a perfectly straight highway, and the car is accelerating at a constant, predictable rate.

The Solution: RELEX (The "Crystal Ball" Method)

Based on this discovery, the authors created a method called RELEX (REinforcement Learning EXtrapolation).

Instead of making the AI finish the whole 500-step hike, RELEX says: "Let's just watch the first 75 steps (about 15% of the trip). Since we know the path is a straight line and the speed is constant, we can mathematically predict exactly where the car will be at step 500, 1,000, or even 2,000."

How it works (The "Denoising" Trick):
The AI's learning process is a bit noisy, like static on a radio.

The Noise: Most of the tiny, random jitters in the AI's brain are just "static" or mistakes that don't actually help it get smarter.
The Signal: The real "genius" is hidden in that single straight line (called a Rank-1 trajectory).
The Magic: RELEX uses a mathematical filter (SVD) to ignore all the noisy static and only look at that one straight line. It then draws a straight line through the early steps and extends it forward.

The Results: Faster, Cheaper, and Just as Good

The researchers tested this on three different AI models (Qwen2.5-Math, Qwen3-4B, and Qwen3-8B).

The Claim: By only training for 15% to 20% of the usual time, RELEX can predict a checkpoint that performs just as well as, or sometimes even better than, the fully trained model.
The Proof: They tested these "predicted" models on math tests. The predicted models scored almost exactly the same as the models that actually finished the full, expensive hike.
The Bonus: Because the method filters out the "noise," the predicted models sometimes generalize better to new, unseen math problems (out-of-domain benchmarks) than the fully trained ones.

Why This Matters (According to the Paper)

No New Learning Required: RELEX doesn't need to train a new AI to predict the future. It just does a simple math calculation (linear regression) on the data it already has.
It's "Minimalist": The paper proves that you don't need complex, fancy predictions. A simple straight line is enough because the AI's learning path is naturally that simple.
Spectral Denoising: The method works so well because it acts like a noise-canceling headphone. It strips away the random, chaotic parts of the training that usually mess up predictions, leaving only the pure signal of improvement.

Summary

Imagine you are watching a rocket launch. Usually, you have to watch it all the way to space to know if it worked. This paper says: "Wait, the rocket is flying in a perfectly straight line at a steady speed. If we watch it for just the first minute, we can calculate exactly where it will be in an hour, and it will be just as successful as if we had waited the whole time."

RELEX is that calculator. It saves massive amounts of time and money by realizing that AI training, despite looking chaotic, is actually following a very simple, predictable path.

Technical Summary: You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories

Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a dominant paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs), particularly in domains like mathematics. However, the standard RLVR pipeline is computationally prohibitive, often requiring days of GPU time to reach performance plateaus. The process involves generating a trajectory of checkpoints over hundreds of optimization steps, with costs scaling linearly with the number of steps. While prior work suggests RLVR amplifies latent behaviors rather than teaching entirely new capabilities, the underlying geometric structure of these parameter trajectories remains underexplored. The central question addressed is whether future RLVR checkpoints can be predicted from a short prefix of training dynamics, thereby bypassing the need for extensive further training.

Methodology: RELEX

The authors propose RELEX (REinforcement Learning EXtrapolation), a training-free method that leverages the geometric regularity of RLVR weight updates to predict future checkpoints. The method operates on the hypothesis that RLVR trajectories are extremely low-rank and near-linear.

The RELEX pipeline consists of three steps, applied to the first $T_{cut}$ observed checkpoints (e.g., the first 15–20% of training steps):

Rank-1 Subspace Estimation:
For each weight tensor $W^{(\ell)}$ , the method computes parameter deltas $\Delta \theta_t = \theta_t - \theta_0$ relative to the base model. These deltas are flattened and stacked into a trajectory matrix $M^{(\ell)}$ . A Singular Value Decomposition (SVD) is performed on $M^{(\ell)}$ , and the top-right singular vector $v_1^{(\ell)}$ is extracted. This vector defines the dominant direction of parameter change across the observation window.
Linear Coefficient Extrapolation:
The observed deltas are projected onto the dominant singular vector to obtain a scalar coefficient trajectory $c_t = \langle \text{flatten}(\Delta \theta_t), v_1 \rangle$ . Empirical analysis shows these coefficients evolve nearly linearly with the training step ( $R^2 > 0.98$ ). A simple linear regression ($c(t) = at + b$) is fitted to these coefficients using the observed prefix.
Future Weight Prediction:
To predict a checkpoint at a target step $T$ , the method extrapolates the coefficient $\hat{c}_T = aT + b$ and reconstructs the weight tensor as $\hat{W}_T = W_0 + \hat{c}_T \cdot v_1$ . This process is repeated for all tensors to assemble the full predicted checkpoint $\hat{\theta}_T$ .

Crucially, RELEX requires no learned predictors, no additional RLVR training, and involves only closed-form operations (truncated SVD and least-squares fitting).

Key Contributions

Empirical Discovery of Low-Rank Linearity: The authors demonstrate that RLVR weight update trajectories are extremely low-rank and predictable. Specifically, a rank-1 SVD approximation captures the vast majority of downstream-relevant parameter changes, and the corresponding scalar coefficients evolve near-linearly with training steps across multiple model families (Qwen2.5-Math-1.5B, Qwen3-4B-Base, Qwen3-8B-Base).
RELEX Framework: The introduction of a simple, compute-efficient extrapolation method that predicts future checkpoints using only the early training prefix.
Spectral Denoising Mechanism: The paper identifies that the rank-1 projection acts as a "spectral denoiser." By projecting updates onto the dominant subspace, the method discards stochastic optimization noise (captured by higher-rank components) that would otherwise degrade performance during extrapolation.
Minimalist Sufficiency: Ablation studies confirm that increasing the subspace rank (e.g., to rank-5 or rank-10) or employing non-linear modeling (polynomials, neural networks) yields no further gains and often degrades performance, validating the sufficiency of the rank-1 linear approach.

Experimental Results

The method was evaluated on three Qwen-based models across in-domain (MATH) and five out-of-distribution (OOD) benchmarks (AIME 2025/2026, HMMT 2025, OlympiadBench, AMC 2023).

Performance vs. Cost: Using only 15–20% of the full RLVR training cost (observing the first 75–125 steps), RELEX produces checkpoints that match or exceed the performance of fully trained RLVR models.
- Qwen2.5-Math-1.5B: RELEX achieved 71.6% on MATH (vs. 71.5% for full RLVR) and outperformed RLVR on 4 of 5 OOD benchmarks.
- Qwen3-4B-Base: RELEX achieved 85.6% on MATH (vs. 85.5% for full RLVR).
- Qwen3-8B-Base: RELEX achieved 87.4% on MATH (vs. 88.5% for full RLVR), staying within 1.1% of the full training performance.
Long-Horizon Extrapolation: RELEX successfully extrapolates far beyond the observation window. For instance, observing only the first 50 steps allowed for stable predictions up to 1000 steps (20× the observation window), with continued performance improvement.
Comparison to Baselines: RELEX uniformly outperformed other extrapolation baselines, including Weight Extrapolation (linear interpolation in raw weight space), Logits Extrapolation, ExPO, and AlphaRL. The superiority over Weight Extrapolation highlights the importance of the SVD step in filtering out noise.

Significance and Claims

The paper claims that RL training, despite its stochastic and non-convex nature, carves surprisingly simple paths in low-rank parameter space. The significance of this work lies in:

Efficiency: It offers a way to achieve near-optimal RLVR performance with a fraction of the computational cost, effectively "skipping" the majority of training steps without retraining.
Theoretical Insight: It provides evidence that the "signal" of RLVR learning is concentrated in a single dominant direction, while higher-rank components primarily represent optimization noise.
Generalization: The method demonstrates that extrapolated checkpoints capture transferable reasoning gains, as evidenced by strong performance on OOD benchmarks, suggesting the model is not merely overfitting the training distribution.

The authors note limitations, acknowledging that the current study is restricted to GRPO on mathematical reasoning tasks with Qwen models. They suggest that the optimal observation window ( $T_{cut}$ ) is model-dependent and that adaptive rank selection may be necessary for broader generalization to other algorithms (e.g., PPO) or model families.