🤖 AI

Selective Off-Policy Reference Tuning with Plan Guidance

The paper introduces Selective Off-Policy Reference Tuning (SORT), a method that leverages plan guidance to derive repair updates from reference solutions, thereby transforming failed rollouts into structure-aware learning signals that significantly improve reasoning performance, particularly on weaker models, compared to standard GRPO approaches.

Original authors: Duc Anh Le, Tien-Phat Nguyen, Thien Huu Nguyen, Linh Ngo Van, Trung Le

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Duc Anh Le, Tien-Phat Nguyen, Thien Huu Nguyen, Linh Ngo Van, Trung Le

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Silent" Failure

Imagine you are teaching a student (an AI) to solve difficult math problems. You give them a problem, and they try to solve it.

The Good Scenario: Sometimes, the student gets it right. You say, "Great job!" and they learn from that success.
The Bad Scenario: Sometimes, the problem is so hard that the student tries 8 different ways to solve it, and all 8 attempts are wrong.

In standard AI training methods (called GRPO), when a student fails every single time on a hard problem, the teacher gets confused. The system says, "Well, since they got nothing right, there's no way to tell which part of their thinking was good and which was bad. So, we'll just ignore this problem and move on."

The paper argues this is a huge waste. Even when the answer is wrong, the student often has a "verified reference solution" (the correct answer key). The problem is that simply copying the answer key is bad because it forces the student to memorize the entire solution, including boring parts like "write the number 5" or "add a comma," rather than learning the hard logic steps that actually caused the failure.

The Solution: SORT (The "Plan" Detective)

The authors propose a new method called SORT (Selective Off-Policy Reference Tuning with Plan Guidance). Think of it as a smart tutoring system that fixes the "silent failure" problem without changing how the student tries to solve problems in the first place.

Here is how SORT works, step-by-step:

1. The "Buffer" (Saving the Failures)

When the AI tries a hard problem and gets it wrong every time, instead of throwing the problem away, SORT puts it in a special "waiting room" (a buffer). It waits until it has a few of these hard failures, then it processes them separately.

2. The "Plan" (The Blueprint)

For each failed problem, SORT looks at the correct answer key. But it doesn't just read the answer; it asks the AI to extract a "Plan" or a "Blueprint" from that answer.

Analogy: Imagine the answer key is a long, messy recipe for a cake. The "Plan" is just the list of critical steps: "Preheat oven," "Mix flour," "Fold in eggs." It ignores the boring stuff like "wipe the counter" or "stir slowly."

3. The "Double Check" (The Magic Test)

This is the core innovation. SORT takes the AI and asks it to look at the correct answer key twice:

Test A: "Here is the problem. Now, look at this step in the answer key. How likely are you to guess this step?" (The AI has no help).
Test B: "Here is the problem AND here is the 'Plan' (the blueprint). Now, look at that same step in the answer key. How likely are you to guess it?"

4. The "Lightbulb Moment" (Selectivity)

SORT compares the two results:

If the AI was already good at guessing the step: The "Plan" doesn't change much. These are usually boring, routine steps (like writing numbers). SORT says, "We don't need to teach the AI this; it already knows it."
If the AI was bad at guessing the step, but the "Plan" made it easy: This is the "Lightbulb Moment." The AI realizes, "Oh! If I knew the plan was to 'check for parity,' then I could have guessed this step!"
- These are the critical reasoning steps (the logic gaps).
- SORT gives these specific steps a huge boost in learning. It tells the AI, "Focus all your energy here! This is where you failed, and this is the key to fixing it."

Why This is Better Than Other Methods

Standard Copying (SFT): Like forcing a student to copy a whole textbook page by page. They learn the handwriting and the formatting, but maybe not the logic.
Other "Hint" Methods: Some methods give the AI hints while it is trying to solve the problem. This changes how the AI thinks during the test.
SORT: SORT leaves the AI's "thinking process" (the rollout) completely alone. It only fixes the "after-school tutoring" (the update). It uses the "Plan" only to figure out which specific words in the correct answer are the most important to learn.

The Results

The paper tested this on three different AI models (from small to large) and eight different math and reasoning benchmarks.

The Winner: SORT consistently beat the standard methods.
The Big Win: It helped the weakest models the most. This makes sense because weaker models fail more often, meaning they have more "silent failures" to fix.
The Efficiency: It didn't make the AI write longer, rambling answers (a common side effect of other methods). It just made the answers smarter.

Summary Analogy

Imagine a coach watching a basketball player miss 8 shots in a row.

Old Way: The coach says, "You missed everything. We'll ignore this drill."
Bad Way: The coach says, "Watch this perfect video of a pro player making the shot. Copy every move, even how they tie their shoes."
SORT Way: The coach says, "Let's look at the video of the pro. I'm going to highlight the exact moment they bent their knees and the exact angle they released the ball. Those are the two things you missed. Ignore the rest of the video; let's just practice those two specific moves."

By focusing only on the "structural" moves that the player missed, SORT helps the AI learn faster and smarter, especially when things are really hard.

Technical Summary: Selective Off-Policy Reference Tuning with Plan Guidance (SORT)

1. Problem Statement

Reinforcement Learning from Verifiable Rewards (RLVR), particularly Group Relative Policy Optimization (GRPO), has improved reasoning in language models by reinforcing sampled solutions that yield positive outcomes. However, GRPO faces a critical failure mode on "hard prompts" where every sampled rollout is incorrect. In these scenarios, the group-relative advantage collapses to zero because there is no variance in rewards (all are zero), rendering the prompt invisible to the policy gradient update.

These zero-reward prompts are often the most informative training signals, as they are accompanied by verified reference solutions containing the missing reasoning structure. However, standard approaches to utilizing these references—such as uniform Supervised Fine-Tuning (SFT) or full-trajectory imitation—fail because they mix core reasoning decisions with routine algebra, formatting, and surface wording. This uniform imitation can lead to policy collapse or verbosity without improving reasoning accuracy. Existing guidance methods often alter the rollout distribution or rely on external teachers, which introduces complexity or distributional shifts.

2. Methodology: SORT

The authors propose Selective Off-Policy Reference Tuning with Plan Guidance (SORT), an auxiliary repair mechanism designed specifically for zero-reward GRPO groups. SORT operates without modifying the primary GRPO rollout generation process.

Core Mechanism

Buffering and Deferral: When a prompt yields only incorrect rollouts (zero-reward group), it is not discarded. Instead, the prompt and its verified reference solution are buffered.
Plan Extraction: For each buffered example, the current model extracts a "reasoning plan" from the reference solution. This plan is an ordered outline of key reasoning decisions (e.g., lemmas, case splits, constraint propagation) but is not used as a target sequence for generation.
Dual Evaluation (Diagnostic Context): The model evaluates the reference tokens twice under the same policy:
- Base Condition: The token probability is computed given only the problem prompt.
- Plan-Conditioned (Teacher) Condition: The token probability is computed given the problem prompt plus the extracted reasoning plan.
Token Salience Scoring: The method calculates the log-ratio between the plan-conditioned probability and the base probability. Tokens where the probability significantly increases under plan conditioning are identified as "structurally informative" (i.e., they represent reasoning steps the model missed but can learn if the structure is explicit). Routine tokens (algebra, formatting) show little change.
Geometric Weighting: SORT constructs an auxiliary loss using a geometric-mean weight:
$\omega_{i,t} = \text{sg}\left(\sqrt{p^{\text{base}}_{i,t} \cdot p^{\text{plan}}_{i,t}}\right)$
This weight combines the stability of the base model's support (from Dynamic Fine-Tuning, DFT) with the selectivity of the plan-conditioned teacher. Tokens that become more predictable with the plan are upweighted; those that do not are downweighted or left unchanged.

Theoretical Guarantees

The authors formalize SORT using a ground-truth plan model. They prove that as the extracted plan approximates the true reasoning plan, the SORT weight converges to an "oracle" weight that emphasizes exactly the structurally critical tokens. The regret bound is shown to be proportional to the plan extraction error ( $\gamma$ ) and the base probability, ensuring that the method degrades gracefully to DFT-like behavior if the plan extraction is poor.

3. Key Contributions

Identification of Zero-Advantage Failures: The paper identifies zero-reward prompts as a specific bottleneck in GRPO where the group advantage is zero, and introduces SORT to repair these without altering rollout generation.
Same-Model Plan Conditioning: Unlike methods that inject off-policy traces from stronger models or use external teachers, SORT uses the same model to extract a plan and score reference tokens, avoiding external dependencies.
Geometric-Mean Token Weighting: A novel weighting scheme that integrates DFT-style support awareness with plan-conditioned selectivity, mathematically proven to approximate oracle structural-token weighting.
Empirical Validation: Comprehensive experiments demonstrating consistent improvements over GRPO and guidance-based baselines across multiple model sizes and benchmarks.

4. Experimental Results

The method was evaluated on three instruction-tuned backbones (Llama-3.2-3B, Qwen2.5-7B, and Qwen3-4B) across eight benchmarks, including in-distribution math tasks (AIME, AMC, MATH-500) and out-of-distribution reasoning (GPQA, MMLU-Pro).

Performance Gains: SORT consistently outperformed vanilla GRPO and other baselines (LUFFY, ReLIFT, Scaf-GRPO).
- On the weakest model (Llama-3.2-3B), SORT improved the in-distribution average from 17.8% to 23.9% and out-of-distribution from 22.5% to 33.8%.
- On stronger models, SORT provided marginal but consistent gains, confirming its utility even when zero-reward groups are less frequent.
Comparison to Baselines:
- vs. SFT/ReLIFT: Unlike SFT or ReLIFT, which often increased response length and verbosity without improving accuracy (especially on weaker models), SORT maintained compact generations while boosting reasoning.
- vs. LUFFY: LUFFY, which mixes stronger off-policy traces, showed less selectivity and sometimes degraded performance on out-of-distribution tasks.
Training Dynamics: SORT improved reward scores without the length expansion observed in other methods. It effectively repaired hard prompts by targeting specific reasoning tokens rather than imitating full traces.

5. Significance and Claims

The paper claims that SORT addresses a fundamental limitation in RLVR: the inability to learn from the most difficult prompts where exploration fails. By treating verified references not as trajectories to copy, but as evidence for a reasoning plan, SORT converts "all-wrong" failures into structured, token-selective learning signals.

The authors emphasize that SORT is a conservative, auxiliary update that preserves the original GRPO rollout process. It does not require external teachers, synthesized trajectories, or changes to the inference distribution. The method is particularly significant for weaker models where zero-reward groups are most frequent, offering a pathway to improve reasoning capabilities without the computational cost of generating correct rollouts for every prompt. The approach is framed as a "selective form of self-distillation" that avoids the pitfalls of over-emphasizing easy, routine tokens while focusing on the uncertain, structurally critical steps of reasoning.