🤖 machine learning

Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think

This paper argues that the success of off-policy learning in large-scale reinforcement learning for LLMs stems from an implicit pessimism that optimizes for more conservative target policies, and it proposes a principled modification to stabilize this induced distribution for improved performance.

Original authors: Otmane Sakhi, Aleksei Arzhantsev, Imad Aouali, Flavian Vasile

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Otmane Sakhi, Aleksei Arzhantsev, Imad Aouali, Flavian Vasile

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a student (an AI model) how to solve complex math problems. You give them a set of practice problems and a "score" for how well they did. The goal is to help them learn from these scores to get better at solving future problems.

This paper tackles a specific problem that happens when you try to teach this student on a massive scale: The student is learning from old homework.

The Problem: Learning from "Stale" Homework

In the real world of big AI, generating new practice problems takes a long time. By the time the AI finishes a batch of problems and gets a score, the "teacher" (the AI's current brain) has already changed a bit. So, the AI is being trained on data that was collected by an older, slightly different version of itself.

In the world of AI, this is called Off-Policy Learning. The data comes from the "Past Self," but the learning is happening to the "Current Self."

Most current methods try to fix this mismatch by using a complex mathematical "correction factor" (like a heavy-handed translator) to pretend the old data was actually new. The authors argue this is messy. It introduces high noise, makes the training unstable, and can cause the AI to "panic" and forget how to be creative (a phenomenon called entropy collapse).

The Alternative: Embracing the Mismatch

Instead of trying to force the old data to look like new data, the authors suggest: Just use the old data as it is, but change how we interpret the scores.

They looked at a popular method called OAPL (which skips the complex correction factors) and asked: "What is this method actually doing to the AI's brain?"

The Discovery: The "Pessimistic" Teacher

The authors discovered that when you train without those correction factors, the AI isn't just ignoring the mismatch; it is actually becoming pessimistic.

Here is the analogy:

The "Optimistic" Teacher (Standard Method): If the AI gets a perfect score on one specific math problem, the Optimistic Teacher says, "Wow! That's the only way to solve it! Forget everything else, do exactly that forever!" This is dangerous. If that one perfect score was a fluke or a weird edge case, the AI gets stuck and stops learning anything new.
The "Pessimistic" Teacher (This Paper's Method): The Pessimistic Teacher sees a perfect score and says, "Okay, that's a great solution. Let's try to do that, but let's not bet the farm on it. Let's keep some of our other ideas alive just in case."

Mathematically, the authors found that this "Pessimistic" behavior follows a specific curve called the Lambert function. Instead of the AI's confidence exploding exponentially (like a rocket) when it sees a high score, the Lambert function acts like a shock absorber. It lets the AI improve, but it prevents it from over-committing to a single, potentially flawed, high-scoring example.

Why the Old Method Was "Accidentally" Good

The paper explains a confusing quirk in the existing OAPL method. Theoretically, the math suggested the AI should be in a dangerous, unstable zone. But in practice, engineers had to tweak a setting (changing a "temperature" number) to make it work.

The authors realized that this tweak wasn't just a random fix; it was accidentally pushing the AI into the "Pessimistic" zone. It was manually forcing the AI to be less aggressive and more stable.

The Solution: A Simple, Principled Fix

Instead of relying on engineers to guess the right "temperature" tweak, the authors propose a simple, built-in rule: Shift the average score up by a tiny bit.

Think of it like this:

Old Way: "Here is your score. If it's above the group average, you get a bonus." (This can be unstable).
New Way: "Here is your score. We are going to pretend the average score is slightly higher than it really is, so your bonus is calculated more conservatively."

This simple shift guarantees the AI stays in the "Pessimistic" (safe) zone automatically. It doesn't need complex tuning.

The Results

The authors tested this new approach against the old one:

Stability: When the "homework" was very old (stale data), the old method crashed or the AI stopped being creative. The new method kept working smoothly.
Robustness: When the rules for "how much to change" were made very strict (small regularization), the old method failed, but the new method kept improving.

The Bottom Line

This paper shows that off-policy learning works not because it ignores the past, but because it naturally becomes "pessimistic" and cautious. By understanding this, the authors created a simple rule that makes AI training more stable, less prone to crashing, and better at handling old data, without needing complex mathematical corrections.

Technical Summary: Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think

Problem Statement

Reinforcement learning (RL) has become a central tool for enhancing the reasoning capabilities of Large Language Models (LLMs) in tasks such as mathematics, coding, and multi-step problem solving. However, at the scale required for modern LLMs, the training pipeline is inherently off-policy. Due to the high cost of generating rollouts, training data is often produced by lagged inference workers or older model checkpoints rather than the policy currently being optimized.

Existing methods, such as PPO and GRPO, attempt to handle this mismatch by treating training as approximately on-policy, using importance weights, clipping, or filtering to correct distribution shifts. While effective in near-on-policy settings, these corrections introduce high variance and can destabilize optimization when the policy lag is significant. Conversely, recent ratio-free approaches (e.g., $A^\star$ -PO, OAPL) remove importance weights and train directly on behavior-policy data. While these methods show empirical success, their theoretical underpinnings are not fully understood. Specifically, the nominal derivations of these objectives suggest a target distribution that differs from what practical implementations (which often decouple temperatures for advantage computation and regularization) actually achieve. The paper seeks to explain why these ratio-free methods work and how to stabilize them further.

Methodology and Theoretical Analysis

The authors analyze a class of ratio-free off-policy objectives that regress the learned policy's log-ratio toward a target advantage, without using importance weights. They focus on objectives of the form:
$\mathbb{E} \left[ \sum_i \left( \hat{A}_i \log \frac{\pi_\theta(y_i|x)}{\pi_{old}(y_i|x)} - \frac{\beta}{2} \left( \log \frac{\pi_\theta(y_i|x)}{\pi_{old}(y_i|x)} \right)^2 \right) \right]$
where $\hat{A}_i$ is a group-relative advantage and $\beta$ is a regularization parameter.

Key Theoretical Insight: Lambert-Tempered Targets
By analyzing the stationary point of this objective, the authors derive that the induced target policy is not the standard exponential (KL-regularized) distribution, but rather a Lambert-tempered policy:
$\pi^\star(y|x) = \pi_{old}(y|x) \frac{1}{\tau_s} W_0 \left( \tau_s \exp \left( \frac{A(x,y)}{\beta} \right) \right)$
where $W_0$ is the principal branch of the Lambert W function, and $\tau_s$ is a normalization multiplier determined by the advantage baseline.

The behavior of the target policy depends critically on the sign and magnitude of $\tau_s$ , which is controlled by how the advantage is normalized:

Pessimistic Regime ( $\tau_s > 0$ ): The target is more conservative than the exponential update. The Lambert function grows logarithmically for large advantages, preventing the policy from over-committing exponentially to a few high-reward samples. This mitigates entropy collapse.
Exponential Boundary ( $\tau_s = 0$ ): The target recovers the standard exponential tilting. This regime is fragile; small errors in the advantage baseline can shift the objective into unstable regimes.
Unstable Regime ( $\tau_s < 0$ ): The target becomes more aggressive than the exponential update, amplifying high-advantage responses and increasing susceptibility to entropy collapse.

Analysis of Existing Methods (OAPL/ $A^\star$ -PO)
The paper demonstrates that the nominal advantage estimator used in OAPL (log-sum-exp centered advantage) induces a target in the unstable regime ( $\tau_s \le 0$ ) at the population level due to Jensen's inequality bias in finite group sizes. The empirical stability observed in OAPL implementations is attributed to a heuristic "temperature decoupling" (using a larger temperature for advantage computation than for regularization), which implicitly shifts the advantage to a more conservative regime.

Proposed Modification
To achieve stability without relying on heuristic temperature decoupling, the authors propose a $\beta$ -shifted mean advantage:
$\hat{A}_i = r_i - \bar{r} + \beta$
This simple shift ensures that the induced Lambert multiplier satisfies $\tau_s \ge 1$ , guaranteeing the target policy remains in the pessimistic Lambert regime by construction.

Key Contributions

Characterization of Ratio-Free Objectives: The authors prove that ratio-free off-policy objectives induce Lambert-tempered target policies, where the degree of conservatism is controlled by the advantage normalization.
Explanation of Stability Mechanisms: They explain that the success of methods like OAPL relies on implicit pessimism induced by practical implementation choices (temperature decoupling), while their nominal theoretical derivations actually suggest unstable targets.
Principled Stabilization: They propose a $\beta$ -shifted mean advantage that guarantees the target lies in the stable, pessimistic Lambert regime, removing the need for hyperparameter-heavy temperature decoupling.

Experimental Results

The authors evaluated their method against OAPL using the Qwen3-8B model on the DeepScaleR dataset, with evaluation on math benchmarks (Minerva, AIME2025, AMC23, BRUMO).

Sensitivity to Regularization ( $\beta$ ): Under small regularization strengths (small $\beta$ ), OAPL exhibited rapid entropy collapse and degraded rewards, whereas the proposed Lambert method remained stable and continued to improve.
Robustness to Policy Lag: As the policy lag (stale rollouts) increased (from 32 to 256 steps), OAPL became highly sensitive, showing either entropy collapse or unstable entropy growth. The Lambert method maintained higher entropy and converged to higher rewards across all lag settings.
Conclusion: The proposed modification improves robustness to both aggressive regularization and large policy lags, validating the hypothesis that targeting the pessimistic Lambert regime stabilizes off-policy learning.

Significance and Claims

The paper claims that the effectiveness of off-policy learning in LLM reasoning is not merely due to ignoring distribution mismatch, but because specific advantage normalizations induce a form of implicit pessimism. This pessimism prevents the policy from over-optimizing on stale, high-reward samples, which is crucial when rollouts are generated by older policies.

The authors position their work as providing a principled understanding of why certain implementation choices in recent off-policy algorithms work, moving beyond "trust-region" heuristics to a framework based on Lambert-tempered targets. They argue that explicitly targeting the pessimistic regime via a shifted-mean advantage offers a more stable and reliable alternative to existing methods, particularly in the challenging setting of large-scale, asynchronous LLM training. The paper concludes by suggesting that token-level variants of this theory represent a promising direction for making off-policy reasoning algorithms more principled.