🤖 AI

DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes

DenoiseRL is a scalable reinforcement learning framework that enhances reasoning in large language models by learning directly from incorrect traces of weak models to recover from noisy prefixes, thereby eliminating the need for expensive teacher supervision or curated datasets while outperforming strong on-policy baselines.

Original authors: Caijun Xu, Changyi Xiao, Zhongyuan Peng, Yixin Cao

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Caijun Xu, Changyi Xiao, Zhongyuan Peng, Yixin Cao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a student how to solve complex math problems. Usually, you'd hire a genius tutor to show them the perfect steps. But what if you don't have a genius tutor? What if you only have a student who is bad at math?

Most current AI training methods say, "If we don't have a genius, we can't get better." This new paper, DenoiseRL, says, "Actually, we can use the bad student's mistakes to teach the smart student how to be even smarter."

Here is how it works, using simple analogies:

1. The Problem: The "Perfect Tutor" Bottleneck

Right now, to make AI smarter at reasoning (like solving math), we usually need a "stronger" AI to act as a teacher. It's like trying to learn advanced calculus from a professor. But what if you don't have a professor? You're stuck.

2. The Solution: The "Detour" Training

DenoiseRL changes the game. Instead of looking for a perfect teacher, it takes a "weak" AI (the bad student) and asks it to solve a problem. The weak AI will likely get lost, make wrong turns, and hit a dead end.

Instead of throwing those wrong answers away, DenoiseRL uses them as a training obstacle course.

The Analogy: Imagine a hiker (the AI) trying to reach a mountain peak (the correct answer).
- Normal Training: The hiker starts at the bottom and tries to find the path.
- DenoiseRL Training: The hiker is magically teleported to a spot halfway up the mountain, but they are standing in a swamp of mud (the wrong reasoning steps generated by the weak AI).
- The Goal: The hiker isn't allowed to just walk forward. They have to figure out, "Wait, I'm stuck in mud. How do I climb out of this mess and get back on the right trail to the peak?"

3. How It Works: "Denoising" the Path

The paper calls this "Denoising." Think of the weak AI's wrong steps as "noise" or static on a radio signal.

The system takes a chunk of the weak AI's wrong answer (the "noisy prefix").
It forces the smart AI to start its answer from that wrong place.
The smart AI has to realize, "Oh, this first part is wrong. I need to correct it, switch tracks, and find the right way to the solution."

This teaches the AI a superpower: Recovery. It learns that even if it starts down the wrong path, it can spot the error, fix it, and still get the right answer.

4. The Sweet Spot: Don't Make the Mud Too Deep

The researchers found that the "mud" (the wrong steps) needs to be the right amount of deep.

Too shallow: If the wrong steps are tiny, the AI doesn't learn much.
Too deep: If the wrong steps are huge, the AI gets confused and starts "overthinking." It gets stuck in a loop of doubting itself, checking its work over and over, and never finishing.
Just right: A moderate amount of wrong steps forces the AI to practice fixing errors without getting paralyzed.

5. The Results

When they tested this on math and logic puzzles:

The AI trained with DenoiseRL got better scores than AI trained with standard methods.
It didn't need a "genius" teacher; it learned by fixing the mistakes of a "weaker" version of itself.
It became better at spotting its own errors and correcting them on the fly.

Summary

DenoiseRL is like a gym for AI brains. Instead of just lifting weights (solving problems from scratch), it puts the AI in a situation where it has to climb out of a hole it didn't dig. By practicing how to recover from mistakes, the AI becomes more robust, smarter, and less dependent on having a perfect teacher to guide it.

Technical Summary: DenoiseRL

Problem Statement

Reinforcement learning (RL) has become a dominant paradigm for enhancing reasoning in large language models (LLMs). However, current state-of-the-art approaches face a structural bottleneck: they often rely on supervision from stronger teacher models or heavily curated, difficult datasets to drive improvement. This dependence creates a scalability issue; when sufficiently capable off-the-shelf teachers are unavailable, further capability gains become difficult. Furthermore, standard on-policy RL methods often suffer from an exploration bottleneck, where the policy saturates by generating mostly correct rollouts or narrowly confined failure modes, leaving "informative failures" too scarce for meaningful gradient updates. Existing attempts to address this, such as Weak-to-Strong Generalization (W2SG), are limited by the noise and capacity of the weak teacher, while data synthesis approaches require complex, manual engineering.

Methodology: DenoiseRL

The paper proposes DenoiseRL, a reinforcement learning framework that reframes reasoning improvement as a denoising and recovery problem. Instead of using weak models to generate correct pseudo-labels or hard data, DenoiseRL utilizes weak models to generate structured perturbations (incorrect reasoning prefixes) that serve as training signals.

Core Mechanism

Noisy Prefix Injection: The framework samples incorrect reasoning trajectories from a weak model (e.g., Qwen2.5-1.5B). It selects a prefix of this incorrect trajectory (determined by a prefix ratio $\rho$ ) and prepends it to the policy model's input.
Recovery Objective: The policy model is trained not to imitate the weak model, but to recover from the erroneous state. It must recognize the corruption, revise its reasoning, switch to a correct path, and reach the verified answer.
Training Distribution: The training process mixes two types of rollouts for each problem:
- Main Rollouts ( $N$ ): Standard on-policy rollouts starting from the question.
- Denoise Rollouts ( $K$ ): Rollouts starting from the noisy prefix generated by the weak model.
Output Budget Fairness: To ensure fair comparison, the total response length is capped. The noisy prefix consumes part of the token budget, and the model's continuation is truncated accordingly. Only the tokens generated by the policy (the continuation) are updated during backpropagation; the noisy prefix is masked in the loss calculation to prevent instability.
Optimization: The framework employs Group-Relative Policy Optimization (GRPO). A single advantage baseline is shared across all $N+K$ trajectories for a given problem. This allows the denoise rollouts (which often result in failure if the model cannot recover) to serve as negative samples, while successful recoveries provide positive learning signals relative to the group.

Key Contributions

Novel RL Paradigm: The introduction of DenoiseRL, which unifies weak-to-strong learning with difficulty-driven data synthesis. It repurposes weak model errors as structured noise rather than learning signals, enabling scalable improvement without external supervision.
Enhanced Self-Correction: The method explicitly trains the model to recover from corrupted intermediate states, elevating self-correction from an emergent behavior to a direct training target.
Empirical Validation: The paper demonstrates that DenoiseRL consistently outperforms strong on-policy RL baselines (GRPO and DAPO) across competitive mathematical and general reasoning benchmarks (MATH500, AMC23, AIME2024/2025, BBEH) on both 4B and 8B parameter models.
Design Analysis: The authors provide a detailed analysis of key hyperparameters, identifying that:
- Prefix Ratio ( $\rho$ ): Moderate noise ( $\rho=0.2$ ) is optimal; excessive noise ( $\rho=0.8$ ) induces "overthinking" and endless self-doubt loops.
- Recovery Intensity ( $K$ ): A balanced mix of denoise rollouts ( $K=4$ out of 16 total) yields the best trade-off. Too few ( $K=1$ ) provides insufficient signal, while too many ( $K=8$ ) distracts from the core goal of solving problems from scratch.
- Off-Policy Updates: Updating the policy on the noisy prefix tokens leads to training collapse, confirming that gradients should only flow through the model's own continuation.

Results

Experiments conducted on Qwen3-4B and Qwen3-8B models show consistent improvements:

Performance Gains: DenoiseRL-GRPO improved the average score on Qwen3-4B from 39.6% (GRPO baseline) to 42.0%. DenoiseRL-DAPO achieved the best overall results on the 8B model, reaching an average of 44.8% compared to the DAPO baseline of 42.8%.
Benchmark Specifics: Notable gains were observed on harder benchmarks like AIME2024 and AIME2025, where DenoiseRL-DAPO on the 8B model achieved 27.0% and 24.8% respectively, outperforming all other configurations.
Behavioral Changes: Case studies reveal that the model learns to preserve useful structural information from the prefix while discarding flawed logic, effectively switching strategies (e.g., from explicit root searching to Vieta's formulas) to recover the correct answer.

Significance and Claims

The paper claims that DenoiseRL offers a scalable and effective alternative pathway for improving reasoning in LLMs. Its primary significance lies in:

Reducing Dependency: It eliminates the need for expensive data curation or stronger teacher models, relying instead on the model's own ability to learn from its (and weaker models') failures.
Scalability: By converting weak model errors into structured perturbations, the method automatically increases training difficulty without generating new data.
Robustness: The framework promotes stronger self-corrective behavior, particularly as training difficulty increases, suggesting that recovery training influences deeper reasoning dynamics beyond simple performance metrics.

The authors conclude that model mistakes themselves can serve as a powerful source of learning signal, providing a new perspective on post-training for reasoning models. They acknowledge limitations, noting that the quality of the recovery signal depends on the nature of the weak model's errors and that excessive corruption can lead to inefficient, overly long reasoning trajectories.