🤖 machine learning

Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information

This paper introduces Off-Context GRPO (OC-GRPO), a reinforcement learning algorithm that leverages privileged guidance during training to overcome learning cliffs on difficult reasoning problems while using importance correction to ensure the model optimizes for the original unguided objective, resulting in significant performance gains on mathematical benchmarks.

Original authors: Priyank Agrawal, Ankur Samanta, Shervin Ghasemlou, Jalaj Bhandari, Kavosh Asadi, Daniel Jiang, Aditya Modi

Published 2026-07-22

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Priyank Agrawal, Ankur Samanta, Shervin Ghasemlou, Jalaj Bhandari, Kavosh Asadi, Daniel Jiang, Aditya Modi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Picture a robot being trained to solve tricky math puzzles, alongside a teacher who can instantly check whether any answer it produces is right or wrong. This setup, a learner plus an automatic judge issuing a clean right-or-wrong verdict, is what AI researchers call "Reinforcement Learning with Verifiable Rewards." The robot attempts a puzzle, earns a thumbs up for getting it right and a thumbs down otherwise, and over many attempts learns to collect more thumbs up.

The catch is that some puzzles sit so far beyond the robot that every single attempt fails. When every attempt fails, all the verdicts are identical, there is nothing to compare against, and the teacher has no signal to work from. It resembles trying to learn to ride a bike while falling off the moment the pedals turn: without ever once feeling what balance is, there is nothing to build on. This is called a learning cliff.

The obvious fix is a cheat sheet, a hint or the first few steps of the worked solution, so the robot at least succeeds occasionally. That works, but it raises an awkward question: how much credit does a robot actually deserve for a win it was handed?

This paper answers the question properly. The method, Off-Context GRPO (OC-GRPO), applies a correction that asks how likely the robot was to reach that exact solution without the cheat sheet. Credit is never given in full, some of it always belongs to the hint, and how much survives depends on how capable the robot already was. A robot that was close to solving it alone keeps most of the credit, because the cheat sheet only surfaced an ability that was already there. A robot with essentially no chance keeps almost none, because the cheat sheet substituted for the skill rather than revealing it. A cheat sheet cannot manufacture progress out of a skill that is not there. Mistakes work in reverse: getting a puzzle wrong with half the answer already in front of it draws a heavier penalty than an ordinary miss, since failing with help is worse evidence than failing without it. On the hardest puzzles, where unaided ability is near zero, most of the real learning arrives through that amplified penalty rather than through the assisted wins. The robot is pushed away from what it demonstrably cannot yet do, rather than congratulated for reading.

The upshot is that the robot is always scored on the real puzzle, not the hinted one, despite having practiced with help.

On standard math benchmarks, OC-GRPO lifted average accuracy from 27.8% to 31.7% over the standard training method, a 13.8% relative gain, at essentially no additional cost. The correction matters most for smaller models: there, older hint-based methods actually perform worse than plain training, because a weaker learner cannot absorb the mismatch between the hinted and unhinted puzzle, while OC-GRPO holds its gains at every size tested. The broader lesson is that privileged hints are a perfectly good way to guide exploration. The credit just has to be assigned honestly.

Technical Summary: Off-Context GRPO (OC-GRPO)

1. Problem Statement: The Learning Cliff in RLVR

Reinforcement Learning with Verifiable Rewards (RLVR), particularly using Group Relative Policy Optimization (GRPO), has emerged as a dominant approach for enhancing reasoning in Large Language Models (LLMs). However, the authors identify a fundamental failure mode known as the learning cliff.

In standard GRPO, for a given problem, a group of responses is sampled, scored by a verifier, and advantages are computed based on the relative performance within that group. If a problem is sufficiently difficult such that the model fails to generate any correct solutions in a rollout group, all rewards are zero. Consequently, the within-group reward variance collapses to zero, resulting in zero gradient signals. The model receives no learning signal, regardless of how long training continues.

Existing workarounds often involve privileged guidance (e.g., solution prefixes, hints, or oracle information) injected into the prompt during training to ensure at least some rollouts succeed. While this restores the learning signal, it introduces the off-context problem:

Training Distribution: Rollouts are sampled from a guided prompt $g(x)$ containing privileged information unavailable at deployment.
Target Objective: The goal is to optimize the policy for the original, unguided prompt $x$ .
The Mismatch: Standard guided methods (e.g., POPE, BREAD) optimize an objective $J_{guide}(\theta)$ conditioned on $g(x)$ , assuming the policy will generalize back to $x$ ("back-generalization"). The authors argue this is an implicit optimization of a different objective than the one evaluated at deployment. This mismatch can lead to biased gradients, objective misalignment, and training instability, particularly for smaller models with limited capacity to absorb the distribution shift.

2. Methodology: Off-Context GRPO (OC-GRPO)

The paper proposes OC-GRPO, a minimally modified variant of GRPO that utilizes guided rollouts to overcome the learning cliff while mathematically correcting for the distribution mismatch to ensure the update targets the original unguided objective $J(\theta)$ .

Core Mechanism: Importance Sampling Correction

Instead of treating guided rollouts as if they were generated from the original prompt, OC-GRPO applies an importance sampling (IS) correction to the per-token policy ratio.

Standard GRPO Ratio: $\rho_{i,t}(\theta) = \frac{\pi_\theta(y_{i,t} | x, y_{i,<t})}{\pi_{\theta_{old}}(y_{i,t} | x, y_{i,<t})}$
OC-GRPO Ratio: $\rho^{oc}_{i,t}(\theta) = \frac{\pi_\theta(y_{i,t} | x, y_{i,<t})}{\pi_{\theta_{old}}(y_{i,t} | g(x), y_{i,<t})}$

Here, the numerator represents the current policy's probability of the token given the unguided prompt (the target), while the denominator represents the probability under the guided prompt (the actual sampling distribution).

Theoretical Properties

Unbiasedness: The reweighted expectation under the guided distribution is mathematically proven to be exactly equivalent to the expectation under the original unguided objective. This ensures the algorithm optimizes $J(\theta)$ , not $J_{guide}(\theta)$ .
Behavior-Aware Credit Assignment: The importance ratio acts as a dynamic credit assignment mechanism:
- Successes: If a correct trajectory is highly likely under the guided prompt but unlikely under the unguided prompt (i.e., the model relied heavily on the hint), the importance ratio is $<1$ , dampening the positive credit. This prevents the model from simply learning to "follow hints" rather than solving the problem.
- Failures: If a trajectory fails despite the guidance, the ratio is $>1$ , amplifying the penalty. This reinforces that the model failed to solve the problem even with help.
Variance Control: The variance of the correction term scales with the length of the guidance prefix, not the full rollout. This motivates the design principle of using the shortest possible guidance necessary to break the learning cliff.

Implementation Variants

OC-GRPO-Fixed: Identifies the minimum guidance level required for each hard problem using a base model before training. This augmented dataset is fixed, and the policy never re-selects guidance. This is recommended as the default due to lower inference costs and competitive performance.
OC-GRPO-Adaptive: Dynamically adjusts guidance levels during training based on the current policy's performance. While theoretically flexible, it incurs higher inference costs and showed slightly lower performance in experiments.

3. Key Contributions

Formalization of the Off-Context Problem: The authors formally define the mismatch in existing guided RLVR methods, showing they implicitly optimize a different objective than the deployment target.
OC-GRPO Algorithm: A minimal modification to GRPO using importance sampling to correct for off-context sampling, providing provable unbiasedness for the original objective.
Theoretical Credit Assignment: Proof that the importance correction induces a behavior-aware mechanism that automatically discounts successes heavily reliant on guidance and amplifies penalties for failures that persist despite guidance, without requiring additional reward shaping.
Empirical Validation: Demonstration that OC-GRPO achieves consistent gains over vanilla GRPO and guided baselines across multiple model scales (1.5B, 3B, 7B) and mathematical benchmarks.

4. Experimental Results

The authors evaluated OC-GRPO on the MATH dataset (Levels 3–5) using Qwen2.5 models (1.5B, 3B, 7B) with LoRA adapters.

Performance Gains: On the 7B model, OC-GRPO-Fixed achieved a 13.8% relative gain (3.9% absolute improvement) in average Pass@1 over vanilla GRPO across AIME, Gaokao2023, and OmniMath benchmarks.
Comparison to Baselines:
- OC-GRPO outperformed other guided methods (POPE, PrefixRL, BREAD) which optimize the misaligned $J_{guide}$ objective.
- Scale Dependency: A critical finding is that guided baselines optimizing the misaligned objective often degrade below vanilla GRPO at smaller model scales (e.g., 1.5B and 3B). In contrast, OC-GRPO maintained consistent gains across all scales. This suggests that smaller models cannot absorb the objective mismatch, making the importance correction critical for them.
Stability: Experiments showed that uncorrected guided training (Masked no-IC) leads to reward collapse, gradient norm spikes, and widening log-probability gaps, whereas OC-GRPO remains stable.

5. Significance and Claims

The paper claims that privileged information can guide exploration, but the update must account for the distribution it came from.

Solving the Cliff: OC-GRPO provides a principled way to escape the learning cliff on hard problems without resorting to supervised fine-tuning (SFT) or complex hybrid objectives that distort the model's native reasoning style.
Alignment: It ensures that training on privileged contexts (which do not exist at inference) does not drift the policy away from the true deployment objective.
Scalability: The method is particularly vital for smaller models where the "back-generalization" assumption of other methods fails.
Generality: While demonstrated on mathematical reasoning with solution prefixes, the framework is mechanism-agnostic and applicable to any form of privileged signal (e.g., tool-call results, intermediate states) in agentic RLVR settings.

The authors conclude that OC-GRPO represents a necessary correction for the next generation of RLVR algorithms, enabling robust reasoning on hard problems while maintaining alignment with the original task definition.