💬 NLP

The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement

This paper introduces SAVE, a framework that leverages value-anchored on-policy feedback to enable reward models to self-supervise and improve their training by grading policy responses and filtering ambiguous samples, thereby overcoming the reliance on costly human preference data and demonstrating superior performance across diverse benchmarks and RL algorithms.

Original authors: Xiaobo Wang, Tong Wu, Min Tang, Jiaqi Li, Qi Liu, Zilong Zheng

Published 2026-06-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaobo Wang, Tong Wu, Min Tang, Jiaqi Li, Qi Liu, Zilong Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to write great stories. To do this, you need a Teacher (the Policy) who writes the stories, and a Judge (the Reward Model) who grades them.

In the standard way of doing this (called RLHF), the Judge is trained once on a big pile of old human notes, then frozen. The Teacher writes stories, the Judge grades them, and the Teacher gets better.

The Problem:
As the Teacher gets smarter, it starts writing stories that are very different from the old notes the Judge was trained on. The Judge gets confused. It starts giving bad grades to good stories or good grades to bad ones because it hasn't seen this "new style" of writing before. This leads to the Teacher "gaming the system"—writing weird, robotic stories that trick the Judge into giving high scores, even though they aren't actually good. This is called "reward hacking."

Usually, to fix this, you'd have to hire more humans to grade the new stories or ask a super-smart AI to act as a new Judge. Both of these are expensive and slow.

The Solution: SAVE
The paper introduces a new framework called SAVE. Think of it as giving the Judge a self-improving superpower that lets it learn from the Teacher's own work without needing new human help.

Here is how it works, using a simple analogy:

1. The "Value Anchor" (The Baseline)

Imagine the Judge has a special tool: a Value Anchor. Instead of just looking at a story and saying "Good" or "Bad," the Anchor asks: "On average, how good is a story for this specific prompt?"

If the prompt is "Write a poem about a cat," the Anchor knows the average cat poem is okay.

If the Teacher writes a poem that is better than the average, the Judge says, "This is a Positive sample!"
If the Teacher writes a poem that is worse than the average, the Judge says, "This is a Negative sample!"

This turns the Teacher's own output into a self-grading system. The Judge doesn't need to know the "absolute truth"; it just needs to know what's better or worse than the current average.

2. The "Adaptive Filter" (The Quality Control)

Sometimes, the Teacher writes two stories that are almost identical in quality. The Judge might get confused and say, "Hmm, these are both kind of average."

SAVE has a Filter that says: "If we can't clearly tell the difference between the good and the bad, let's ignore that example for now." It only keeps the examples where the difference is clear and obvious. As the training goes on, this filter gets smarter, allowing slightly harder examples to be used later.

3. The "Self-Supervised Loop" (The Cycle)

Here is the magic cycle:

The Teacher writes a batch of stories.
The Judge uses its Value Anchor to compare them. It separates them into "Better than average" and "Worse than average."
The Judge uses these clear differences to update itself. It learns, "Ah, I see! When the Teacher writes like this, it's actually good, even if it looks different from my old training data."
The Teacher then uses this new, smarter Judge to grade its next batch of stories and gets even better.

Why is this a big deal?

No New Humans Needed: The Judge learns from the Teacher's own mistakes and successes, so you don't need to pay humans to grade new data constantly.
Stays Fresh: Because the Judge learns from the Teacher's current writing style, it never gets "out of date." It evolves right alongside the Teacher.
Prevents Cheating: By anchoring the grades to the average performance, it's harder for the Teacher to trick the Judge with weird, low-quality tricks.

The Results

The authors tested this on six different "exam boards" (benchmarks) that test how good a Judge is.

The Score: The initial Judge started with an average score of 76.0. After using SAVE, it improved to 77.3.
The Teacher: When they used this improved Judge to train the Teacher, the Teacher became significantly better at following instructions and writing high-quality content.

In short: SAVE is like giving a teacher's grader a mirror. Instead of relying on old textbooks, the grader looks at the student's current work, compares it to the student's own average, and learns from the differences. This makes the grader smarter and the student better, all without hiring a new proctor.

Technical Summary: The Flip Side of RLHF (SAVE)

Problem Statement

The alignment of Large Language Models (LLMs) via Reinforcement Learning from Human Feedback (RLHF) relies heavily on the quality of the Reward Model (RM). However, constructing robust RMs faces two primary bottlenecks:

Data Acquisition Costs: Obtaining diverse, high-quality preference data is expensive, requiring significant human annotation or distillation from stronger, often inaccessible, judge models.
Distributional Mismatch: Standard RMs are trained on static, offline preference data. As the policy model evolves during RL training, its generation distribution drifts away from the static training set. This leads to miscalibration in the regions most frequently visited by the optimized policy, inducing "reward hacking" and over-optimization (Goodhart's Law).

Conventional remedies, such as collecting fresh human labels or re-querying external judges, merely reproduce the original cost bottlenecks or perpetuate dependence on external supervision. The paper posits a critical question: Can a reward model improve itself using the on-policy responses generated during RL training, without additional human labels or external judges?

Methodology: SAVE Framework

The authors propose SAVE (Self-supervised reward model improvement via Value-Anchored On-policy feEdback), a framework that treats on-policy responses as self-supervised feedback to continuously refine the RM.

Core Components

Value-Anchored Reward Model:
The RM is augmented with a prompt-specific value head ( $V(x)$ ) alongside the standard reward head ( $R(x, y)$ ). The value head estimates the expected reward of responses to a prompt $x$ under the current sampling policy. This is achieved by sharing the RM's backbone ( $f_\omega$ ) and adding a lightweight value head ( $v_\psi$ ).
- RM Advantage: The advantage of a specific response $y$ is defined as $a_{RM}(x, y) = R(x, y) - V(x)$ . This measures whether a response is better or worse than the policy's average expectation for that prompt.
Adaptive Feedback Filtering:
To ensure the self-supervised signal is reliable, SAVE employs a curriculum-driven filtering mechanism:
- For each prompt, a group of $G$ on-policy responses is sampled.
- Responses with RM advantages near zero (ambiguous samples) are filtered out using a dynamic threshold $\mu(t)$ that decays over training steps.
- Only prompts yielding both positive ( $a_{RM} \ge 0$ ) and negative ( $a_{RM} < 0$ ) subsets of responses are retained to provide a contrastive signal.
Self-Supervised Objectives:
The framework optimizes the RM using two distinct losses on the filtered data:
- Contrastive Reward Loss ( $L_R$ ): A value-anchored contrastive objective that pushes positive advantage responses above the value anchor and negative advantage responses below it. This updates the reward head ( $h_\phi$ ) and backbone ( $f_\omega$ ).
- Value Calibration Loss ( $L_V$ ): A Mean Squared Error (MSE) loss that trains the value head to predict the unbiased empirical mean reward of the sampled group. This ensures the anchor remains a calibrated baseline.
Minimax Formulation:
Theoretically, SAVE is framed as a reward-model-centric minimax problem:
$\min_\eta \max_\theta M_t(\eta, \theta)$
Where the policy ( $\theta$ ) acts as an adaptive data generator that naturally seeks out regions where the current RM is miscalibrated (exposing its weaknesses), and the RM ( $\eta$ ) minimizes its ranking and calibration errors on these challenging samples.

Key Contributions

SAVE Framework: A general, self-supervised framework that continuously improves reward models using on-policy feedback from RL training, eliminating the need for additional human annotation or external judges.
Theoretical Justification: The authors formalize the process as a minimax problem, theoretically demonstrating that policy optimization naturally yields informative data for RM improvement by exposing epistemic errors in the reward model.
Empirical Validation: Extensive experiments across six diverse benchmarks and multiple RL algorithms (GRPO, RLOO, GSPO) and policy backbones.

Experimental Results

The paper evaluates SAVE on six reward model benchmarks: RewardBench, RewardBench 2, RM-Bench, PPE Preference, PPE Correctness, and JudgeBench.

Reward Model Performance:
- Starting from a baseline RM (Skywork-Reward-V2-Llama-3.2-3B) with an average score of 76.0, SAVE improved the average to 77.3 when using a stronger policy backbone (Qwen3-4B).
- SAVE achieved the best scores on all six benchmarks, outperforming baselines including continual offline training, Mean Reward baselines, and the HL-BT (Highest-Lowest Bradley-Terry) method.
- Ablation studies confirmed that both the curriculum filtering mechanism and the dynamic policy updates are critical; freezing the policy or removing the curriculum led to performance drops.
Downstream Policy Performance:
- Improved RMs translated to better downstream policy performance. On AlpacaEval 2, the length-controlled win rate increased from 51.68% to 54.24%.
- On Arena-Hard-v2.0, the win rate rose from 30.2% to 33.9%.
- The improved RM proved transferable, enhancing policies trained with it even when not co-trained.

Significance and Claims

The paper claims that SAVE offers a scalable solution to the "bottleneck" of reward model training. By leveraging the policy's own evolution to generate informative, on-policy feedback, SAVE breaks the dependency on static offline data and external judges.

The authors emphasize that:

Self-Improvement is Viable: Reward models can effectively self-improve using only the data generated during the RL process.
Value Anchoring is Crucial: Using a learned, prompt-specific value anchor is superior to naive pairwise ranking (like HL-BT), which can overfit to superficial structural differences between semantically equivalent responses.
Robustness: The method is optimizer-agnostic, showing consistent gains across different critic-free RL algorithms (GRPO, RLOO, GSPO).

The paper concludes that on-policy feedback serves as an effective supervision signal for improving both reward models and the downstream policies they guide, shifting the paradigm from isolated reward modeling toward online joint evolution.

Limitations

The authors modestly acknowledge several limitations:

Model Scale: Experiments were conducted on models in the 3B–4B range; dynamics for substantially larger models remain to be verified.
Evaluation Metrics: Reliance on automatic benchmarks and LLM-based judges; human preference studies would provide stronger evidence, particularly for safety-critical or subjective tasks.
Computational Cost: SAVE introduces additional computation due to sampling multiple responses per prompt and updating the RM during training, though the time overhead is reported as moderate compared to the memory increase.