💬 NLP

Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States

This paper introduces POISE, a reinforcement learning method that leverages a lightweight probe trained on a policy model's internal states to estimate value baselines at negligible cost, thereby achieving stable and efficient policy optimization without the computational overhead of separate critics or multiple rollouts.

Original authors: Yunho Choi, Jongwon Lim, Woojin Ahn, Minjae Oh, Jeonghoon Shim, Yohan Jo

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yunho Choi, Jongwon Lim, Woojin Ahn, Minjae Oh, Jeonghoon Shim, Yohan Jo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a brilliant but very expensive student (a Large Language Model) how to solve complex math problems. You want them to get better, so you use a method called Reinforcement Learning. In this method, the student tries to solve a problem, and you give them a score (a reward) if they get the answer right.

To teach them effectively, you need to tell them not just what they got right, but how much better they did compared to their usual performance. This "difference" is called the advantage. To calculate this, you need a baseline—a guess of what the student would typically score on that specific problem.

The Problem: The Cost of Guessing

Currently, there are two main ways to get this baseline, and both are expensive:

The "Huge Tutor" Method (PPO): You hire a second, equally large and expensive AI model just to act as a critic. This critic watches the student's work and guesses the score. This doubles your computing costs because you are running two giant models at once.
The "Group Average" Method (GRPO): You ask the student to solve the same problem 8 times in a row. You then average those 8 scores to get a baseline. This wastes a lot of time and money because you are generating 7 extra answers just to get a baseline, leaving less room to try new problems.

The Solution: POISE (The Student's Own Intuition)

The authors of this paper propose a new method called POISE (Policy Optimization with Internal State Value Estimation).

Here is the core idea: The student already knows how hard the problem is before they even finish solving it.

When a large AI model thinks, it generates a trail of "internal thoughts" (hidden states) as it moves from the question to the answer. The paper argues that these internal thoughts contain a hidden signal about how likely the model is to succeed. It's like the student having a gut feeling: "I'm struggling with this step; I probably won't get it right," or "This feels easy; I'm confident."

POISE works like this:

The Lightweight Probe: Instead of hiring a second giant AI, they attach a tiny, cheap "probe" (like a simple calculator) to the student's brain.
Reading the Signals: This probe looks at the student's internal thoughts (hidden states) and their "uncertainty" (entropy) while they are thinking.
The Prediction: The probe predicts the score based on those internal signals.
The Fairness Trick (Cross-Rollout): To make sure the student doesn't cheat by looking at their own answer to guess the score, the system uses a clever trick. It asks the student to solve the problem twice. To grade the first attempt, the probe looks at the second attempt's internal thoughts, and vice versa. This ensures the baseline is fair and unbiased.

Why This is a Big Deal

It's Cheaper: You don't need a second giant AI model. You just use the tiny probe on the signals the student is already generating.
It's Faster: You don't need to generate 8 answers to get a baseline. You only need 2 (one to solve, one to help grade). This frees up your computer budget to try many different problems, which helps the student learn faster.
It's Stable: Because you can try more different problems, the learning process is less "noisy" and more stable.

The Results

The researchers tested this on math reasoning tasks (like Olympiad math). They found that:

POISE performed just as well as the state-of-the-art methods (like DAPO) that use expensive critics or large groups of answers.
It used less computing power and finished training faster.
The "gut feeling" was accurate: The tiny probe predicted the success rate almost as well as a full-sized AI critic model.
It works elsewhere: They tested it on coding and tool-use tasks, and it worked well there too, proving the student's internal signals are a universal indicator of success.

In Summary

POISE is like realizing that your student doesn't need a second teacher to tell them how well they are doing. By simply listening to their internal "thinking process," you can instantly know if they are on the right track. This saves money, saves time, and makes the learning process smoother, all without sacrificing performance.

Technical Summary: POISE (Policy Optimization with Internal State Value Estimation)

Problem Statement

Reinforcement learning with verifiable rewards (RLVR) is a central driver for improving Large Reasoning Models (LRMs) on complex tasks like mathematical reasoning. However, stable and efficient RLVR relies heavily on accurate baseline estimation to reduce the variance of policy gradients. Existing approaches incur significant computational costs:

PPO (Proximal Policy Optimization): Requires training a separate, LLM-scale critic model that processes the full generated sequence at every update, roughly doubling memory consumption and optimization complexity.
GRPO (Group Relative Policy Optimization): Avoids a separate critic by estimating baselines from the mean reward of a group of sampled rollouts per prompt. This "trades parameters for samples," requiring multiple rollouts per prompt to maintain a stable baseline. Under a fixed compute budget, this reduces the diversity of prompts in a batch, inflating gradient variance. Furthermore, substantial compute is wasted on "degenerate" prompts where all rollouts receive identical rewards (zero advantage).

The core question addressed by this work is: Can an effective baseline be extracted from the computations already performed during policy training, eliminating the need for extra resources?

Methodology: POISE

The authors propose POISE (Policy Optimization with Internal State Value Estimation), a method that leverages the policy model's own internal representations to estimate the value function $V^\pi(x)$ at negligible cost.

1. Internal State Value Estimation

Instead of training a separate critic, POISE trains a lightweight probe to predict the expected verifiable reward directly from the policy model's hidden states and token statistics collected during the forward pass.

Input Features: The probe utilizes three complementary signals:
1. Prompt-level features: Hidden states from the final prompt tokens, capturing the model's representation of the prompt and anticipated difficulty.
2. Trajectory-level features: Hidden states from the end of the reasoning trajectory, exposing value-relevant information not present in the prompt alone.
3. Token-level entropy statistics: Lightweight uncertainty features.
Training Objective: The probe is trained to predict the prompt-level value $V^\pi(x)$ . To prevent the target from leaking information about the specific rollout being evaluated, the authors use a leave-one-out Monte Carlo target. For a prompt $x$ with $K$ sampled rollouts, the target for rollout $i$ is the average reward of the other $K-1$ rollouts.

2. Cross-Rollout Construction for Unbiasedness

A critical theoretical challenge is ensuring the baseline remains conditionally independent of the action (the sampled response) to preserve the unbiasedness of the policy gradient. If the baseline depends on the same rollout it evaluates, it introduces bias.

Solution: POISE employs a cross-rollout construction. For each prompt, two independent rollouts, $y^{(1)}$ $y^{(1)}$ and $y^{(2)}$ $y^{(2)}$ , are sampled.
- The baseline for $y^{(1)}$ is predicted using the internal states of $y^{(2)}$ .
- The baseline for $y^{(2)}$ is predicted using the internal states of $y^{(1)}$ .
This ensures the baseline used to update a specific rollout is derived from an independent sample, satisfying the condition $E[b(x) \nabla \log \pi(y|x)] = 0$ .

3. Integration into Policy Optimization

POISE integrates this estimator into a PPO-style optimization loop:

Online Training: The value estimator is trained jointly with the policy on a sliding buffer of recent trajectories. Because the features are extracted from the forward pass already required for policy updates, the computational overhead is negligible.
Advantage Calculation: Advantages are computed as $A^{(i)}(x) = R(x, y^{(i)}) - b^{(i)}(x)$ , where $b^{(i)}(x)$ is the cross-rollout baseline.
Efficiency: Unlike GRPO, which requires a large group (e.g., 8 rollouts) to estimate a baseline, POISE only requires a pair of rollouts. This allows the fixed compute budget to be allocated to a larger number of distinct prompts per batch, theoretically reducing gradient variance.

Key Contributions

Novel Baseline Estimation: Introduces a method to derive baselines from the policy model's internal states, removing the need for a separate LLM-scale critic or large rollout groups.
Theoretical Unbiasedness: Proposes a cross-rollout architecture that guarantees the conditional independence required for unbiased policy gradients while utilizing trajectory-conditioned features.
Compute Efficiency: Demonstrates that a lightweight probe (linear regression) can achieve value estimation performance comparable to or better than a separate critic, while requiring significantly less compute.
Generalizability: Shows that the internal state signals generalize across different domains (math, coding, tool-calling, instruction-following) and model scales.

Experimental Results

The method was evaluated on Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B across olympiad-level mathematical reasoning benchmarks (AMC, AIME, HMMT, BRUMO).

Performance: POISE achieves performance comparable to DAPO (a state-of-the-art GRPO-based algorithm).
- On Qwen3-4B, POISE achieved an average Avg@32 score of 0.500, closely matching DAPO's 0.508, while outperforming it on specific benchmarks like AMC23 and HMMT25.
- On DeepSeek-R1-Distill-Qwen-1.5B, POISE improved the average score to 0.303 compared to DAPO's 0.296.
Training Efficiency: POISE requires substantially less wall-clock time. On DeepSeek-R1-Distill-Qwen-1.5B, reaching comparable performance took 18 hours with POISE versus 24 hours with DAPO on a single B200 GPU. On Qwen3-4B, the time was 36 hours vs. 49 hours.
Stability: Gradient norms in POISE were consistently lower than in DAPO, indicating reduced gradient variance and more stable optimization.
Value Estimation Quality: The internal state probe achieved a Pearson correlation of 0.870 with empirical rewards, outperforming a separately trained LLM-scale critic (0.676) on the same task.
Generalization: The estimator maintained high performance across coding, tool-calling, and instruction-following tasks, often outperforming separate critics in these domains.

Significance and Claims

The paper claims that POISE provides a compute-efficient path toward stable and scalable RLVR for large reasoning models. By leveraging the model's own internal representations, the method:

Eliminates the memory and compute overhead of training a separate critic model.
Removes the sampling inefficiency of group-relative baselines (GRPO), allowing for higher prompt diversity within a fixed budget.
Avoids the waste of compute on degenerate prompts that yield zero advantage.
Demonstrates that internal states, previously used primarily as diagnostic tools, can serve as practical, high-fidelity optimization signals for reinforcement learning.

The authors conclude that POISE enables more stable and efficient policy optimization without the need for group-relative baselines or auxiliary value models, making it a viable alternative for training large reasoning models under resource constraints.