Technical Summary: LatentRevise: Learning from Zero-Hit Reasoning

1. Problem Statement: The Zero-Hit Regime in RLVR

Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard approach for enhancing reasoning in language models. However, it faces a fundamental bottleneck known as the zero-hit regime. In this scenario, a prompt is presented where the current policy fails to generate a single correct trajectory within a practical sampling budget (i.e., $pass@N = 0$).

Standard RLVR algorithms, such as Group Relative Policy Optimization (GRPO), rely on group-normalized advantages to update the policy. When all sampled trajectories in a group fail, they receive identical rewards (typically zero or negative). Consequently, the group mean equals the individual rewards, causing the normalized advantage to collapse to zero. This results in a lack of constructive gradient signals, effectively discarding these "hard" prompts as unlearnable noise.

Existing remedies attempt to address this by:

1. Increasing the sampling budget: Raising $N$ or decoding temperature, which is computationally expensive and still outcome-only (lacking intermediate guidance).
2. On-policy distillation: Using a teacher model or privileged information to provide token-level targets. However, these methods rely on external guidance that may be weak, mismatched, or unavailable when the student model is already struggling.

The core problem is that these zero-hit prompts often lie at the "sampling frontier" where new reasoning behaviors are most needed, yet standard pipelines cannot learn from them because they lack a positive signal.

2. Methodology: LatentRevise

The authors propose LatentRevise, a first-order latent revision method designed to recover training signals from failed rollouts without requiring external teacher models or reference traces. Instead of discarding a failed trajectory, LatentRevise treats it as an informative trace and optimizes the model's input embeddings for the reasoning prefix in continuous latent space.

Core Pipeline

1. Input: A failed rollout $y_{fail} = [r; c]$, where $r$ is the reasoning prefix and $c$ is the failed continuation, along with the gold answer $y_{GT}$.
2. Freeze & Optimize: The model parameters $\theta$ are frozen. The input embeddings of the reasoning prefix, $E_r$, are optimized to shift the trajectory away from the failure and toward the solution.
3. Composite Objective: The optimization minimizes a total loss $L_{total}$ composed of three terms:
   • Latent Exploration Loss ($L_{explore}$): A repulsive signal that minimizes the likelihood of reproducing the original failed continuation $c$. This encourages the latent state to explore alternative paths.
   • Latent Guidance Loss ($L_{guide}$): A guiding signal that maximizes the likelihood of generating the gold answer $y_{GT}$ (conditioned on a fixed answer template). This anchors the revision toward the correct solution.
   • Language Fluency Prior ($L_{NLL}$): A regularization term that maintains the negative log-likelihood of the original reasoning tokens $r$. This ensures the optimized latent remains semantically coherent and does not drift into adversarial noise.
4. Manifold Constraint (Frank-Wolfe): To prevent the continuous embeddings from drifting into uninterpretable feature space (where the model might "cheat" by encoding answers in arbitrary vectors), the optimization is constrained to the convex hull of the model's vocabulary embeddings. This is achieved using the Frank-Wolfe (Conditional Gradient) algorithm. At each step, the update is a convex combination of the current embedding and the optimal vertex (vocabulary token) that minimizes the linear approximation of the loss.
5. Discrete Recovery: The optimized soft prefix $E^*_r$ is projected back to discrete tokens via nearest-neighbor search against the vocabulary matrix. A new continuation is autoregressively decoded. If the resulting trajectory passes the verifier, it is retained as supervised data or injected back into RL groups.

3. Key Contributions

The paper outlines three primary contributions:

1. Reframing the Zero-Hit Frontier: The authors identify zero-hit prompts not as unlearnable outliers, but as a recoverable frontier. They demonstrate that valid training signals can be algorithmically recovered from these failures using the model's own latent space.
2. LatentRevise Algorithm: A novel first-order latent revision framework that converts failed rollouts into successful ones. Crucially, it operates without external teacher distributions or privileged traces, relying instead on the gold answer and the model's own failed continuation to generate dense, position-wise correction signals.
3. Empirical Validation: The recovered trajectories are shown to be high-value training data. They improve both $pass@1$ and $pass@k$ on mathematical reasoning benchmarks, particularly in scenarios where standard RL and larger sampling budgets fail to yield improvements.

4. Experimental Results

The method was evaluated on DAPO-Math-14K using Qwen2.5-1.5B-Instruct and Qwen3-8B-Base models across benchmarks including MATH500, AIME 24/25, AMC 23, OlympiadBench, and Minerva Math.

• Zero-Hit Regime (DAPO-Hard): On the subset of queries where the base model achieved $pass@32 = 0$ (approx. 46% of the dataset for the smaller model), standard GRPO provided negligible improvement due to collapsed advantages. LatentRevise, when used to generate SFT data or injected into RL groups, significantly outperformed baselines.
  • For Qwen2.5-1.5B, GRPO+Ours achieved an average $pass@32$ of 53.51%, compared to 44.84% for standard GRPO and 50.69% for GRPO with a doubled rollout group ($N=16$).
  • Notably, the method outperformed OPSD (On-Policy Self-Distillation), which relies on a teacher conditioned on the gold answer, suggesting that LatentRevise's internal latent optimization is more effective than hindsight teacher conditioning in this regime.
• Full Dataset (DAPO-Full): When applied to the full training distribution (mixing easy and hard prompts), LatentRevise continued to outperform GRPO, DAPO, and larger-rollout baselines, with the most significant gains observed at higher $k$ values ($pass@32$).
• Efficiency: LatentRevise incurred a lower computational overhead than simply doubling the rollout group size ($N=16$). It added ~24% training time and ~3% memory over standard GRPO ($N=8$), whereas doubling the group size increased time by ~66% and memory by ~14%.
• Qualitative Analysis: Revised trajectories exhibited increased length and explicit self-reflection markers (e.g., "Wait, let me reconsider"), indicating that the method surfaces error-correction behaviors that discrete sampling rarely reaches.

5. Significance and Claims

The paper claims that LatentRevise offers a complementary axis for reasoning RL. When token-level sampling reaches its finite-budget frontier, first-order revision in the model's own latent space can transform failed rollouts into verified supervision.

The authors assert that:

• Zero-hit prompts are valuable: They are not merely noise but represent a high-value training frontier. Once revised, they provide signals that expand the model's solution coverage.
• Internal recovery is viable: A model can learn from its own failures without needing a stronger teacher or external reference traces, provided the optimization is constrained to the vocabulary manifold.
• Efficiency: Recovering signals from failed rollouts is more computationally efficient than brute-force sampling expansion, especially as model scales and reasoning chains lengthen.

The work suggests that as reasoning tasks become harder, the boundary of sampling-based training can be pushed not just by sampling more, but by revising what has already been sampled, turning the "dead ends" of the current policy into a source of curriculum and training signal.