🤖 machine learning

Verifier-Induced Support Reshaping in On-Policy Optimization

This paper reveals that on-policy reinforcement learning with verifiable rewards (RLVR) can improve immediate task performance while inadvertently causing "verifier-induced support reshaping," where the policy narrows its output distribution to make successful trajectories for future objectives too rare to sample, thereby compromising long-term trainability and joint capability.

Original authors: Shaohang Wei, Zikun Su, Feifan Song, Wen Luo, Wei Li, Guangyue Peng, Houfeng Wang

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Shaohang Wei, Zikun Su, Feifan Song, Wen Luo, Wei Li, Guangyue Peng, Houfeng Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a super-smart robot to be a helpful assistant. You don't just want it to know facts; you want it to follow rules, solve tricky math problems, and write code. To do this, scientists use a method called "reinforcement learning," which is like a game where the robot tries different answers, gets a score from a "verifier" (a strict judge), and learns to repeat the high-scoring moves. The big hope is that we can teach the robot one skill after another—first math, then writing, then coding—without it forgetting the old stuff or getting confused. This is the dream of "continual learning": building a robot that gets better and better at everything, step by step. But what if the very act of teaching the robot to be perfect at one thing accidentally locks the door to learning the next thing? What if the robot learns to be so good at following instructions that it forgets how to think step-by-step, or vice versa?

This paper investigates a sneaky problem called "verifier-induced support reshaping." The researchers found that when you train an AI to be great at one specific task using a strict judge, the AI doesn't just get better at that task; it actually changes its personality in a way that makes it much harder to learn other tasks later. It's like if you trained a dog to only sit when you say "Sit," and in doing so, you accidentally taught it that "Sit" is the only word that matters. Later, when you try to teach it to "Stay," it might be confused because its brain has been reshaped to ignore everything else. The paper shows that this isn't just about the robot forgetting old tricks (a known problem called "catastrophic forgetting"); it's about the robot making the right kind of answers so rare that the training process can't find them anymore. Even if the robot still has the ability to solve a math problem, it might stop trying to solve it the right way because it got used to a different style of answering.

The researchers tested this by training two types of AI models on two very different tasks: solving math problems and following strict writing instructions (like "write exactly 50 words" or "use bullet points"). They trained one group to be math wizards and another to be instruction-following pros. Then, they swapped the judges to see what happened.

Here is what they discovered:

The Math-to-Instruction Trap
When they trained the AI to be a math genius first, it did get better at following instructions on average. However, something strange happened: the AI became very "polarized." It either got the instruction perfectly right or completely wrong, with very few "middle ground" attempts. Before, the AI might have tried many different ways to answer, some of which were close to right. After math training, it stopped trying those middle paths. This meant that if you tried to sample many answers (like asking the AI to try 32 times), you were less likely to find any successful answer for the instruction task. The math training had narrowed the AI's focus so much that it stopped exploring the "support" (the variety of attempts) needed to learn new instruction-following tricks.

The Instruction-to-Math Trap
The reverse was even more interesting. When they trained the AI to be an instruction-follower first, it started changing how it answered math problems. Instead of showing its work step-by-step (like "First, I add 2 and 2..."), the AI started jumping straight to the answer ("The answer is 4"). The researchers call this a shift from "Deliberate Reasoning Initiation" (DRI) to "Direct Answer Initiation" (DAI).
This shift was a big deal. When the AI started skipping the steps, it became much harder to find the correct answer, even if the AI still knew the math. In fact, the more the AI jumped to direct answers, the lower its success rate became when they tried to sample multiple attempts. The "support" for finding the right math answer had shrunk.

The "First Token" Secret
The researchers dug deeper to find out why this was happening. They discovered that the change wasn't about the AI forgetting how to do math later in the sentence. The change happened in the very first word the AI typed!
It turns out that the "judge" (verifier) for math encourages the AI to start with words like "Alright" or "Let's step through," while the judge for instructions encourages starting with "Answer" or "Here." Once the AI learns to start with "Answer," it gets locked into a path where it skips the reasoning steps. The researchers proved this by forcing the AI to start with the "math" words even when it had been trained to be an instruction-follower. When they did this, the AI's ability to find math solutions magically improved. This suggests that the AI didn't lose its math brain; it just got stuck in a bad "opening move" habit.

Can We Fix It?
The team tried to fix this problem. They tried to "pre-train" the AI to always start with the right words (a "DRI prior"), but this only delayed the problem; eventually, the AI still switched to the direct-answer style. They also tried a technique called "on-policy distillation," where a student AI tries to copy a teacher AI. But this only worked if they picked the teacher very carefully; if the teacher had already learned the "bad" habit, the student copied it too.

The Bottom Line
The paper concludes that simply making an AI better at one task doesn't guarantee it will be ready to learn the next one. In fact, the improvements in one area can sometimes make the "search space" for the next area so small that the AI can't find the right answers anymore. The gains in math or instruction following don't always translate into answers that are both correct and follow the rules. It's a warning to AI developers: just because a model scores high on a test today doesn't mean it hasn't locked the door to its future learning. To keep an AI flexible, we need to make sure it keeps a wide variety of "opening moves" available, not just the ones that score high right now.

Technical Summary: Verifier-Induced Support Reshaping in On-Policy Optimization

1. Problem Statement

The paper addresses a critical limitation in Continual On-Policy Reinforcement Learning with Verifiable Rewards (RLVR). While existing literature often focuses on whether a model retains performance on previous tasks (catastrophic forgetting) after adapting to a new objective, this work identifies a forward-looking problem: Verifier-Induced Support Reshaping.

The core issue is that optimizing a policy for a specific verifier (e.g., mathematical correctness) can alter the policy's sampling distribution such that successful trajectories for future objectives (e.g., instruction following) become too rare to be sampled within a fixed rollout budget. Even if the underlying capability is not "forgotten," the policy may no longer generate the necessary "successful" behaviors frequently enough to provide a positive training signal for subsequent stages. The authors define effective rewardable support as the set of successful trajectories that remain likely enough to be sampled under a specified budget.

2. Methodology

The study employs a controlled, bidirectional experimental design using two model families (Qwen3-8B-Base and Qwen2.5-Math-7B) and two distinct post-training objectives:

Mathematical Reasoning (Math-RLVR): Trained on the MATH dataset with a verifier checking final-answer correctness.
Constrained Instruction Following (IF-RLVR): Trained on IFTrain/IFBench with verifiers checking explicit constraints (format, length, content).

Experimental Design

Bidirectional Training: The authors train one branch with Math-RLVR and measure its impact on IF support, and vice versa.
Sequential Training: They test the order of operations (Math $\to$ IF and IF $\to$ Math) to observe how the first stage constrains the second.
Evaluation Metrics:
- Pass@1 (mean@k): Average success rate.
- Best@k: Whether any successful trajectory exists within $k$ samples (measuring searchability).
- Prompt Bucket Analysis: Categorizing prompts into "all-wrong," "mixed," and "all-correct" based on $k=32$ rollouts.
- Joint Benchmarks: Using MathIF and ReasonIF to test if a single response satisfies both correctness and constraints.

Mechanism Analysis

Token-Level Distribution: Analyzing Jensen-Shannon (JS) divergence to locate where policy changes occur.
Causal Interventions: Forcing specific response openings (e.g., "Answer" vs. "Alright") during decoding to test if the initial token choice causally affects downstream searchability.
Mitigation Strategies: Testing Reference Policy Constraints (KL divergence), Routing Priors (SFT on specific opening types), and On-Policy Distillation (OPD).

3. Key Results

3.1 Asymmetric Support Reshaping

Math-RLVR $\to$ IF: Math-RLVR increases average instruction-following success (pass@1) but significantly reduces best@32. Prompts shift from "mixed" (some successes, some failures) to "all-correct" or "all-wrong." This polarization reduces the number of prompts where sampling can recover a success, effectively shrinking the effective support for future IF training.
IF-RLVR $\to$ Math: IF-RLVR reduces math searchability (best@k drops across tested budgets from $k=4$ to $k=32$ ). Crucially, it shifts the response style from Deliberate Reasoning Initiation (DRI) (step-by-step) to Direct Answer Initiation (DAI). This shift correlates strongly with the loss of math searchability.

3.2 Sequential Training Consequences

IF-First: Training on IF first leaves the subsequent Math-RLVR with almost no "mixed" prompts (only all-wrong or all-correct). Without mixed prompts, group-relative updates lack reward variation, preventing the model from recovering DRI-based reasoning strategies.
Math-First: Training on Math first allows IF-RLVR to improve IF scores, but this comes at the cost of a sharp decline in previously gained math performance. Reference policy constraints (KL) can retain math performance but fail to match the IF gains of the unconstrained baseline.

3.3 Mechanism: The Causal Role of Openings

Localization: The largest distributional shifts occur at the first generated token. The ratio of JS divergence at the first token versus interior tokens ranges from $9.8\times$ to $106.7\times$ .
Reranking vs. Erasure: RLVR primarily reranks candidates already present in the base policy's top-3, rather than introducing entirely new tokens. However, it can promote low-probability tokens (e.g., "Answer") to dominance.
Causality: Controlled interventions confirm that forcing a DRI opening (e.g., "Alright") on an IF-RLVR model recovers math searchability, while forcing a DAI opening suppresses it. This proves the opening choice causally dictates the searchability of the reasoning path.

3.4 Limits of Mitigation

Routing Priors: A DRI-biased SFT prior delays the shift to DAI but does not prevent it; by step 100, DAI dominance is reached regardless of the prior.
On-Policy Distillation (OPD): OPD outcomes are highly sensitive to the teacher's state. A converged IF teacher does not preserve math support, and no single teacher state simultaneously optimizes for IF transfer and math retention.

3.5 Joint Capability

On MathIF and ReasonIF benchmarks, marginal gains in one domain (e.g., Math-RLVR improving correctness) do not translate proportionally into joint success (correctness + constraint following). The joint support ( $J = C \land F$ ) increases only slightly, indicating that endpoint improvements in one objective do not guarantee joint trainability or capability.

4. Contributions

Support Reshaping Framework: The paper frames continual RLVR in terms of future trainability, defining "effective rewardable support" and demonstrating that verifiers reshape this support, potentially making future objectives unreachable.
Causal Role of Openings: It localizes cross-task shifts to the first few tokens and establishes a causal link between response openings (DRI vs. DAI) and downstream searchability, suggesting the issue is route selection rather than broad ability erasure.
Limits of Preservation: It empirically demonstrates that standard mitigation techniques (sequential training, KL constraints, routing priors, OPD) only partially preserve support or trade off target-task gains, and that improvements in isolated metrics do not guarantee joint capability.

5. Significance and Claims

The paper claims that endpoint improvements do not guarantee future trainability or joint capability under on-policy optimization. Unlike catastrophic forgetting, which measures retained performance, support reshaping measures whether successful trajectories for future tasks remain discoverable.

The authors argue that the choice of verifier in RLVR fundamentally alters the "initiation space" of the model. If a verifier pushes the model toward a specific opening (e.g., direct answers), it may inadvertently close off the search space required for other tasks (e.g., step-by-step reasoning). Consequently, the paper suggests that future on-policy optimization strategies must explicitly estimate and preserve effective rewardable support across cross-verifier optimizations, rather than relying solely on current task scores or standard forgetting metrics.