🤖 machine learning

On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR

This paper reveals that Reinforcement Learning with Verifiable Rewards (RLVR) exhibits implicit reward overfitting and operates through low-rank dynamics, where enhanced reasoning capabilities are concentrated in rank-1 components that optimize a specific heavy-tailed singular spectrum while discarding other model knowledge.

Original authors: Hao Ye, Jisheng Dang, Junfeng Fang, Bimei Wang, Yizhou Zhang, Ning Lv, Wencan Zhang, Hong Peng, Bin Hu, Tat-Seng Chua

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hao Ye, Jisheng Dang, Junfeng Fang, Bimei Wang, Yizhou Zhang, Ning Lv, Wencan Zhang, Hong Peng, Bin Hu, Tat-Seng Chua

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Magic" of Reinforcement Learning

Imagine you have a very smart student (a Large Language Model) who is good at many things but isn't great at solving complex math puzzles. You decide to use a special training method called Reinforcement Learning with Verifiable Rewards (RLVR). Think of this as a strict coach who gives the student a gold star every time they get a math answer right.

After training, the student becomes a math genius. But the researchers in this paper asked a tricky question: Did the student actually learn new logic, or did they just learn how to game the coach's scoring system?

1. The "One-Note" Genius (Rank-1 Dominance)

The researchers discovered something surprising: almost all the "math genius" improvement happens in just one tiny slice of the student's brain.

The Analogy: Imagine the student's brain is a massive library with millions of books. When they get good at math, it's not because they read more books. It's because they found one specific, magical bookmark (called the "Rank-1 component") that instantly points to the right answer.
The Finding: If you take the trained model and throw away everything except that one magical bookmark, the model is still just as good at math as the fully trained version. The rest of the "brain updates" (the other millions of pages) seem to be doing very little for the math skills.

2. The "Fake It Till You Make It" Problem (Implicit Reward Overfitting)

Here is the counterintuitive part. The researchers created a training method where they forced the student to keep only that "one magical bookmark" and threw away the rest of the updates every few steps.

The Result: The student's score on the training test (the practice problems the coach gave) dropped. The coach was unhappy because the student wasn't getting as many gold stars during practice.
The Twist: However, when the student took the final exam (a new set of problems they had never seen), they performed just as well as the student who kept all the extra updates.
The Meaning: The "extra updates" the student usually learns are actually noise. They are the student trying to memorize the specific practice questions to get more gold stars, rather than learning the actual logic. The researchers call this "Implicit Reward Overfitting." The model is "faking" its way to a high score on the training data without actually getting smarter.

3. The "Safety Net" (Why We Need the Rest)

If the "extra updates" are just noise for math, why keep them? The researchers found that those extra updates are actually the student's personality and common sense.

The Analogy: If you strip away the "extra updates" and only keep the "math bookmark," the student becomes a math genius but loses their ability to follow instructions, stay safe, or know general facts about the world.
The Finding: The "Rank-1" part is for reasoning. The "Non-Rank-1" parts are for safety, world knowledge, and following rules. If you throw them away, the model might solve math problems perfectly but start saying rude things or forgetting how to talk to humans.

4. The Shape of Learning (The Heavy-Tailed Distribution)

The researchers looked at the "shape" of the changes in the model's brain using a mathematical tool called SVD (which breaks down complex data into simple lines).

The Pattern: They found a consistent pattern: One giant spike (the math bookmark) followed by a long, slow tail of smaller changes.
The Metaphor: Imagine a mountain range. There is one tall, sharp peak (the math reasoning). Behind it, there is a long, rolling hillside (the safety and knowledge). The peak is what makes the model a math wizard, but the hillside is what keeps the model grounded and safe.

5. The "One-Way Street" (Geometric Asymmetry)

Finally, the paper explains how the model learns this. They found that the learning process is lopsided.

The Analogy: Imagine the model is a factory.
- The Output (The Answer): The factory learns to change the final product (the answer) very easily. It's like having a conveyor belt that can be quickly adjusted to package the right item. This is the "left side" of the math, and it aligns perfectly with the "Rank-1" bookmark.
- The Input (The Question): The factory struggles to change how it reads the incoming questions. The questions are messy and complex. The model can't easily reorganize how it understands the input with a simple "bookmark."
The Conclusion: RLVR is mostly about optimizing the output (how the model picks the right answer), not about fundamentally rewriting how the model understands the world. It's like teaching a student to guess the right answer on a test without necessarily teaching them how to read the question better.

Summary

This paper tells us that when AI models get "smarter" at math through Reinforcement Learning:

Most of the magic is concentrated in a tiny, one-dimensional "bookmark" (Rank-1).
The rest of the training often just memorizes the practice test (Overfitting) without adding real value to the final exam.
We need the "noise" (the non-Rank-1 parts) to keep the model safe and knowledgeable, even if it doesn't help with math.
The model is learning to output better answers, not necessarily to understand the input better.

The researchers suggest that by understanding this, we can train models more efficiently, focusing on the "bookmark" for reasoning while protecting the "hillside" for safety and knowledge.

Technical Summary: Implicit Reward Overfitting and Low-rank Dynamics in RLVR

Problem Statement

Despite the engineering success of Reinforcement Learning with Verifiable Rewards (RLVR) in enhancing the reasoning capabilities of Large Language Models (LLMs), the microscopic mechanisms by which RL alters model parameters remain poorly understood. Existing research largely focuses on macroscopic training dynamics or reward design, leaving a gap in understanding the specific structure and physical significance of parameter updates ( $\Delta W$ ).

Recent observations suggest that RLVR improvements are concentrated within low-rank components, specifically Rank-1 updates. However, a disruptive finding indicates that RL may not be imparting new underlying reasoning logic but rather optimizing sampling strategies to elicit latent correct answers. This raises critical questions: Is RLVR genuinely learning reasoning, or is it overfitting to the reward function? Furthermore, what is the functional role of the non-Rank-1 components in the parameter update matrix?

Methodology

The authors employ a combination of empirical studies and theoretical analysis centered on the spectral properties of weight updates.

Periodic Rank-1 Substitution: To investigate the contribution of non-Rank-1 components, the authors introduce a technique where, at fixed intervals (e.g., every 10 steps), the accumulated weight update is decomposed via Singular Value Decomposition (SVD). Only the Rank-1 component (associated with the largest singular value) is retained and added to the base model parameters, while the residual non-Rank-1 components are discarded. The model then continues training from this "Rank-1 purified" state.
Spectral Analysis: The authors perform SVD on the weight difference matrices ( $\Delta W$ ) of RLVR-trained models to analyze the distribution of singular values (singular spectra) across linear layers.
LoRA Alignment Analysis: Using LoRA with rank $r=1$ , the authors compare the geometric alignment between LoRA adapters and the true Rank-1 subspace of the full weight update. They utilize Frobenius Cosine Similarity and Principal Angles (Left and Right) to quantify alignment in input and output spaces.
Theoretical Gradient Analysis: The paper derives the gradient structure of RL objectives (GRPO, DAPO, GSPO), showing they can be formulated as weighted sums of rank-1 matrices, supporting the empirical observation of Rank-1 dominance.

Key Findings and Results

1. Implicit Reward Overfitting

The study reveals a counterintuitive phenomenon: Implicit Reward Overfitting. Models trained with Periodic Rank-1 Substitution achieve test-set performance comparable to fully fine-tuned models (Full RL), despite exhibiting significantly lower training rewards.

Observation: Standard RLVR training drives training rewards up consistently, but this does not translate to superior generalization.
Implication: Non-Rank-1 components significantly boost training rewards but fail to improve generalization. The model appears to be over-optimizing its sampling strategy to fit the training data's reward distribution rather than learning robust reasoning logic.

2. Functional Role of Non-Rank-1 Components

While Rank-1 components capture reasoning capabilities, non-Rank-1 components are not functionally redundant.

Preservation of General Capabilities: Removing non-Rank-1 components via Rank-1 extraction leads to significant degradation in out-of-domain abilities, including instruction-following (IF-Eval), world knowledge (MMLU, MMLU-Pro), and safety (SafetyBench).
Catastrophic Forgetting: The non-Rank-1 components are crucial for maintaining the model's general knowledge and safety alignment, preventing the catastrophic forgetting often seen in supervised fine-tuning (SFT).

3. Universal Spectral Pattern

The singular spectra of parameter updates in RLVR-trained models follow a consistent pattern:

Leading Spike: A dominant first singular value ( $\sigma_1$ ) corresponding to the Rank-1 component.
Heavy-Tailed Distribution: Following the initial spike, the remaining singular values exhibit a uniform, quasi-linear decay, forming a heavy tail.
Energy Distribution: Although the Rank-1 component drives reasoning, the non-Rank-1 tail contains the majority of the "energy" (Frobenius norm) of the update, encoding safety, robustness, and subtle feature alignment.

4. Geometric Asymmetry in LoRA Dynamics

The analysis of LoRA training dynamics reveals a Geometric Asymmetry between input and output subspaces:

Output Space Alignment: The left singular vectors (output space) of the Rank-1 component show strong alignment with LoRA adapters. This aligns with the finding that RLVR primarily optimizes the sampling strategy (output representation).
Input Space Misalignment: The right singular vectors (input space) show weaker alignment. Input updates involve complex reorganization of contextual features that are difficult to capture with simple low-rank matrices.
Theoretical Basis: Gradient analysis shows that the update direction for the output vector is determined by the backpropagated error (which aligns well), whereas the input vector update is determined by highly context-dependent activations, making alignment difficult.

Significance and Contributions

The paper claims to offer a deeper mechanistic understanding of RLVR, challenging the assumption that RL inherently "learns" reasoning in the traditional sense.

Reframing RLVR: The findings suggest RLVR acts more as a mechanism to optimize sampling efficiency and elicit latent reasoning capabilities rather than fundamentally altering the model's underlying reasoning logic. This explains why increasing $k$ in pass@k metrics can sometimes cause standard RL models to underperform compared to base models.
Decoupling Capabilities: The work successfully decouples reasoning capability (Rank-1) from general knowledge and safety (Non-Rank-1), providing a theoretical basis for why "overfitting" to rewards might occur without harming generalization if the Rank-1 component is preserved.
Spectral Insights: The identification of the "leading spike + heavy tail" spectral pattern provides a new lens for analyzing RL updates, suggesting that RL induces a highly structured spectral shape rather than a strictly low-rank update.
Practical Implications: The findings offer potential insights for improving RL paradigms, such as designing training strategies that mitigate implicit reward overfitting while preserving the non-Rank-1 components necessary for safety and general knowledge.

The authors acknowledge limitations, noting that their observations are based on specific model sizes and architectures, and further research is needed to determine the universality of these findings across different parameter scales and model families.