🤖 machine learning

ESPO: Early-Stopping Proximal Policy Optimization

The paper proposes ESPO (Early-Stopping Proximal Policy Optimization), a reinforcement learning algorithm that dynamically terminates failing reasoning trajectories to eliminate noise and save compute, thereby improving mathematical reasoning performance on benchmarks like AIME and MATH-500 while reducing token usage by over 20%.

Original authors: Zihang Li, Rui Zhou, Yingcheng Shi, Wenhan Yu, Zhewen Tan, Zixiang Liu, Zeming Li, Binhua Li, Yongbin Li, Tong Yang, Jieping Ye

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zihang Li, Rui Zhou, Yingcheng Shi, Wenhan Yu, Zhewen Tan, Zixiang Liu, Zeming Li, Binhua Li, Yongbin Li, Tong Yang, Jieping Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart but sometimes overconfident student (a Large Language Model) how to solve complex math problems.

The Problem: The "Sunk Cost" of Wrong Turns

In the current way of training these models (called PPO), the student is given a problem and asked to write out their entire solution step-by-step.

Here is the catch: If the student makes a tiny mistake in the very first sentence—like misidentifying a number or choosing the wrong formula—the entire rest of the essay is doomed. No matter how beautifully they write the next 500 words, the answer will be wrong.

However, the standard training method forces the student to keep writing until they hit a strict word limit, even after the mistake is made.

The Waste: The computer spends time and energy generating hundreds of useless words that will never get a "good grade."
The Confusion: When the teacher (the algorithm) looks at the whole essay to decide what to teach the student, it gets confused. It sees the huge block of "bad" text at the end and thinks, "Oh, the student was bad at the end of the sentence," rather than realizing the problem started at the very beginning. This "noise" makes learning slower and less efficient.

The Solution: ESPO (The "Early Stop" Coach)

The paper introduces a new method called ESPO (Early-Stopping Proximal Policy Optimization). Think of ESPO as a coach who watches the student in real-time and has the authority to say, "Stop! You've gone off the rails. Let's try again."

Here is how ESPO works, using simple analogies:

1. The "Regret" Radar
Every time the student picks a word, the model calculates a "Regret Score."

The Analogy: Imagine the student has a gut feeling about which word is best (the "greedy" choice). If they pick a word that is very different from their gut feeling, the Regret Score goes up.
The Magic: ESPO doesn't need a new teacher or a human to grade every step. It just looks at the student's own internal "gut feeling" (the math behind the word choice) to see if they are hesitating or guessing wildly.

2. The "Value" Gate
The coach also checks a "Value Meter." This meter predicts how much "good" is left in the current path.

The Logic: If the student is on a path that looks promising (High Value), the coach gives them a little more leeway to recover from a small mistake. But if the path looks hopeless (Low Value) and the student is showing high Regret (confusion), the coach pulls the plug immediately.

3. The "Absorbing Failure" Rule
When ESPO stops the student, it doesn't just say "Try again." It marks that specific moment as a terminal failure.

The Analogy: Instead of letting the student write 500 words of nonsense and then giving the whole essay a zero, ESPO says, "You made a mistake here. The rest of the essay doesn't exist."
The Benefit: This sends a very clear, sharp signal back to the student: "The mistake happened right here, not at the end." This helps the student learn exactly where they went wrong, without the noise of the garbage text that followed.

The Results: Faster, Smarter, and Cheaper

The paper tested this on math problems (like the AIME and AMC competitions) using models of different sizes.

Better Grades: The models trained with ESPO actually got higher scores on these difficult math tests compared to the standard method. They solved more problems correctly.
Saving Energy: Because the models stopped writing when they knew they were wrong, they saved more than 20% of the computer power (tokens) usually wasted on generating useless text.
No Extra Teachers: Unlike other methods that require hiring humans to grade every single step of the reasoning, ESPO does this automatically using the model's own internal signals.

Summary

In short, ESPO is a training technique that stops a language model from wasting time and energy on a train of thought that has already crashed. By cutting off the "bad" parts early and clearly marking where the mistake happened, it helps the model learn faster, solve harder problems, and use less computing power.

Technical Summary: ESPO (Early-Stopping Proximal Policy Optimization)

Problem Statement

In Reinforcement Learning (RL) for Large Language Models (LLMs), particularly in long-horizon reasoning tasks, standard algorithms like Proximal Policy Optimization (PPO) suffer from a "rollout continuation problem." When a model commits an early, irrecoverable reasoning error (e.g., a mathematical miscalculation or a logical fallacy), standard PPO forces the policy to continue generating tokens until a fixed horizon ( $T_{max}$ ) is reached.

This process introduces two critical inefficiencies:

Computational Waste: The model expends significant compute on tokens that will never receive a positive reward.
Gradient Noise: These post-failure tokens are included in advantage estimates alongside tokens from successful trajectories. This pollutes the gradient direction with "post-failure noise," misdirecting the learning process away from the actual failure mode and potentially incentivizing the policy to collapse its logit spread rather than solve the task.

Existing solutions, such as Process Reward Models (PRMs), require costly human annotation of intermediate steps. Other methods like GRPO or DAPO improve credit assignment but still exhaust the full horizon for every trajectory. Learned termination modules (e.g., Option-Critic) add model complexity and require separate optimization objectives.

Methodology: ESPO

The authors propose ESPO (Early-Stopping Proximal Policy Optimization), a lightweight mechanism that detects trajectory failures on-the-fly and terminates rollouts early without requiring additional reward models or human annotation. ESPO modifies the rollout collection phase rather than the PPO objective itself.

Core Components

Per-Step Surrogate Regret ( $g_t$ ):
ESPO computes a regret signal at each timestep using only the logits already generated by the actor. It measures the deviation of the sampled token from the policy's most preferred (greedy) action:
$g_t = \max_{a \in V} \log \pi_\theta(a | s_t) - \log \pi_\theta(a_t | s_t)$
A large $g_t$ indicates the model sampled an action significantly less preferred than its mode, suggesting a potential deviation from a correct reasoning path.
Normalized Cumulative Stopping Statistic ( $z_t$ ):
To handle scale changes during training, $g_t$ is normalized using batch-level Exponential Moving Average (EMA) statistics ( $\mu_g, \sigma_g$ ). This normalized signal is then smoothed over time to form a cumulative regret value:
$z_t = \alpha_s z_{t-1} + (1 - \alpha_s) \tilde{g}_t$
Crucially, the EMA statistics are updated only at batch boundaries to prevent information leakage from future steps in the current rollout.
Value-Gated Early Termination:
The rollout is terminated if the cumulative regret significantly exceeds the estimated value of the current state:
$z_t > \beta \cdot \max(V_\phi(s_t), \epsilon)$
This gating mechanism ensures that states with high predicted future returns (high $V_\phi$ ) are granted more tolerance for deviation, preventing the premature termination of correct but non-mode tokens (high entropy steps).
Absorbing Failure Transition:
Upon termination, the trajectory is treated as an absorbing failure state. The stopping step receives a terminal penalty ( $r_{fail}$ ), and no further decoding occurs. This concentrates the negative Temporal-Difference (TD) error at the detected failure step, allowing the error signal to propagate backward cleanly through Generalized Advantage Estimation (GAE) without the noise of subsequent irrelevant tokens.
Critic Warmup:
To prevent spurious early terminations caused by the high variance of a randomly initialized critic, ESPO employs an adaptive warmup period. The stopping rule is disabled until the critic exhibits stable learning dynamics.

Key Contributions

Efficient Rollout Termination: ESPO introduces a mechanism to detect and truncate failing trajectories using only signals already present in standard actor-critic PPO (logits and value estimates), requiring negligible additional computation.
Noise Reduction: By treating truncation as an absorbing failure state with a terminal penalty, ESPO isolates negative TD errors to the failure point, removing the "post-failure noise" that typically corrupts gradient updates in standard PPO.
Orthogonality: The method is orthogonal to existing credit assignment improvements (like GRPO or DAPO) and can be layered on top of any PPO-style advantage estimator.
No External Supervision: Unlike PRMs, ESPO requires no step-level human annotation or separate reward models.

Experimental Results

The authors evaluated ESPO on DeepSeek-R1-Distill-Qwen-1.5B and 7B models trained for mathematical reasoning across three benchmarks: AIME 2024, AMC 2023, and MATH-500.

Performance Gains

7B Scale: ESPO outperformed both standard PPO and DAPO on all benchmarks:
- AIME 2024: 46.28% (vs. 45.25% PPO, 45.57% DAPO)
- AMC 2023: 85.83% (vs. 82.94% PPO, 83.76% DAPO)
- MATH-500: 87.42% (vs. 85.43% PPO, 85.95% DAPO)
- Average Accuracy: 73.17% (vs. 71.20% PPO, 71.76% DAPO).
1.5B Scale: ESPO achieved an average accuracy of 59.09%, surpassing PPO (57.03%) and DAPO (58.29%).

Efficiency Gains

Token Savings: ESPO reduced cumulative rollout tokens by >20% compared to PPO and ~19% compared to DAPO on the 7B scale.
Ablation Studies:
- Removing the critic warmup reduced performance by 2.1 points, confirming its role in preventing bias from uncalibrated value baselines.
- Removing the terminal failure penalty reduced performance by 2.6 points, highlighting the importance of concentrated negative TD signals.
- Random stopping (Variant F) achieved significantly lower accuracy (42.4% vs. 46.3%) despite similar token savings, proving that the improvement stems from where trajectories are truncated (based on regret/value) rather than simply training on shorter sequences.

Significance and Claims

The paper claims that ESPO provides a scalable, compute-efficient framework for advancing long-horizon reasoning in LLMs. Its primary significance lies in solving the rollout continuation problem by:

Eliminating wasted computation on trajectories that cannot recover.
Cleaning the policy gradient by removing post-failure noise without requiring auxiliary reward models or human annotations.
Maintaining exploration: The method does not cause entropy collapse; in fact, it slows the rate of entropy decay compared to PPO by removing spurious gradient signals that would otherwise force the policy to sharpen its distribution against non-failure tokens.

The authors note limitations regarding models that are "incorrect but highly confident" (where logit gaps are small) and the need for manual hyperparameter tuning of truncation rates, suggesting future work in adaptive mechanisms and extension to multi-turn agentic settings.