Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purification

1. Problem Statement

The paper addresses a critical bottleneck in Reinforcement Learning with Verifiable Rewards (RLVR) for Large Language Models (LLMs), specifically in complex reasoning tasks.

• The Core Issue: RLVR methods (like GRPO) rely on sampling correct "rollouts" (reasoning traces) to generate learning signals. However, in complex tasks, reward sparsity is high; correct rollouts are rare due to long-horizon decision-making and binary feedback.
• The Root Cause: The authors identify that many exploration failures are not due to the inherent difficulty of the problem, but rather due to a small number of "Interference Tokens" (typically <5% of the prompt) within the instruction. These tokens introduce noise, causing the model to deviate from correct reasoning paths.
• Limitations of Existing Solutions:
  • Scaling Exploration: Increasing rollout budgets (sampling more) is computationally expensive and inefficient.
  • Prompt Filtering: Discarding difficult prompts (zero-variance) sacrifices the model's ability to learn from challenging samples, limiting capability expansion.

2. Methodology: The LENS Framework

The authors propose LENS (LEss Noise Sampling Framework), an online selective rollout framework consisting of two main stages:

Stage 1: Interference Token Identification and Purification

• Interference Score ($S_I$): The framework measures the deviation between the current policy ($\pi_\theta$) and a reference policy ($\pi_{ref}$) at the token level.
  $$S_I(s, a) \triangleq |\log \pi_\theta(a | s) - \log \pi_{ref}(a | s)|$$
  Tokens with high scores indicate significant deviation, suggesting they are likely "interference" causing over-optimization or spurious behavior.
• Purification: For prompts with low sampling success, the top $k$ tokens with the highest interference scores are identified and removed (pruned), creating a denoised prompt ($x'_i$). This is done with a small deletion ratio ($\gamma \approx 1\%-5\%$) to preserve original semantics.

Stage 2: Calibrated Rollout Policy Optimization (CRPO)

Instead of simply training on the clean prompts, LENS uses a transfer learning approach to teach the model to ignore interference in the original noisy setting.

• Rollout Replacement: If the denoised prompt ($x'_i$) yields a higher success rate than the original ($x_i$), successful rollouts generated from $x'_i$ are used as "high-reward supervision."
• Sample Reweighting: These successful denoised rollouts replace a subset of failed rollouts from the original prompt in the training batch.
• Objective Function: A unified PPO-style objective is optimized. It uses importance weighting to correct for the distribution mismatch between the original and denoised prompts, ensuring the model learns to produce correct reasoning even when the input contains interference tokens.
  • The loss function includes a KL regularization term to prevent the policy from drifting too far from the reference.

3. Key Contributions

1. Novel Insight: The discovery that a small fraction of interference tokens in prompts is a primary cause of RLVR training collapse and low sampling success, rather than just task complexity.
2. LENS Framework: A plug-and-play framework that combines token-level interference purification with a calibrated policy optimization algorithm (CRPO) to transfer learning signals from clean to noisy inputs.
3. Efficiency & Robustness: Unlike methods that discard hard samples or simply sample more, LENS actively teaches the model to filter out noise, leading to more robust reasoning capabilities.

4. Experimental Results

The authors evaluated LENS across multiple model families (Llama-3.2, Qwen-2.5, Qwen-3) and benchmarks.

• Math Reasoning Performance:
  • LENS achieved an average performance gain of 3.88% over standard GRPO across seven math benchmarks (MATH500, AIME, AMC, etc.).
  • It demonstrated a 1.6× speedup in convergence (requiring fewer gradient steps to reach peak accuracy).
  • LENS outperformed baselines that doubled the rollout budget (GRPOextended) and those that filtered prompts (DAPO, GRESO), despite using strictly fewer computational resources.
• Out-of-Domain Generalization:
  • On scientific (SuperGPQA, GPQA-Diamond) and general reasoning (BBEH, MMLU-Pro) tasks, LENS showed a 1.83% average gain, proving it improves general reasoning rather than overfitting to math patterns.
• Training Dynamics:
  • Reduced Zero-Reward Prompts: LENS significantly lowered the ratio of prompts yielding zero rewards compared to GRPO.
  • Stability: Training curves showed more stable accuracy progression and the earlier emergence of "long-form reasoning" (aha moments) compared to the fluctuations seen in GRPO.

5. Significance

• New Perspective for RLVR: The paper shifts the focus from "sampling more" or "filtering harder" to "cleaning the signal." It highlights that pruning interference tokens is a critical, under-explored lever for improving rollout efficiency.
• Resource Efficiency: LENS offers a Pareto improvement, achieving higher accuracy with faster convergence and lower computational costs compared to scaling-based approaches.
• Robustness: By explicitly training the model to ignore interference, LENS enhances the model's ability to reason in real-world, noisy prompting environments, addressing a key gap in current RLVR research.

In summary, LENS demonstrates that by identifying and purifying a few noisy tokens in instructions, and then calibrating the policy to learn from these purified signals, one can significantly boost the efficiency and performance of reinforcement learning for reasoning tasks.