🤖 machine learning

CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning

The paper proposes CLEANER, a method that leverages a model's intrinsic self-correction capabilities through a Similarity-Aware Adaptive Rollback mechanism to generate error-free training trajectories, thereby resolving credit assignment issues and significantly boosting the performance and efficiency of parameter-constrained Agentic Reinforcement Learning.

Original authors: Tianshi Xu, Yuteng Chen, Meng Li

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tianshi Xu, Yuteng Chen, Meng Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a young apprentice (a small AI model) how to solve complex puzzles using a powerful but tricky tool: a computer code interpreter. The goal is for the apprentice to write code, run it, and get the right answer.

However, there's a problem. Because the apprentice is still learning, they make a lot of mistakes. They write code that crashes, causing the computer to spit out long, confusing error messages. In a standard training setup, the apprentice keeps these messy error messages in their memory, trying to fix them step-by-step. Eventually, they might get the right answer, but the path they took was full of noise, confusion, and dead ends.

The paper argues that this "messy path" is actually hurting the learning process. When the apprentice finally succeeds, the teacher (the training algorithm) gives them a "good job" reward for the entire journey. This is unfair because it rewards the mistakes just as much as the correct thinking. It's like praising a chef for burning the soup but eventually fixing it with a spoon; the chef learns that burning things is okay as long as they fix it later.

The Solution: CLEANER (The "Self-Purifying" Chef)

The authors propose a new method called CLEANER. Instead of letting the apprentice keep the messy history of their mistakes, CLEANER acts like a smart editor that cleans up the story while the apprentice is learning.

Here is how it works, using a creative analogy:

1. The "Oops" Moment (The Trigger)
The apprentice writes a line of code, runs it, and the computer screams "ERROR!" In a normal class, the apprentice would just add this error to their notebook and try again.

2. The "Rewind" Button (SAAR Mechanism)
CLEANER steps in immediately. It pauses the process and asks the apprentice: "Okay, you made a mistake. Can you fix it right now?" The apprentice tries again and succeeds.

3. The "Edit" (Similarity-Aware Adaptive Rollback)
This is the magic part. CLEANER looks at the mistake and the fix to decide how to clean the notebook:

Scenario A (The Typos): If the fix is just a small change (like fixing a missing comma), CLEANER assumes the apprentice's thinking was actually good, they just had a typo. It quietly erases the error and the fix, replacing the messy line with the clean, correct code. The notebook now shows a smooth, successful thought process.
Scenario B (The Logic Flaw): If the fix is completely different from the original attempt (like realizing the whole approach was wrong), CLEANER knows the original thinking was flawed. It erases the entire wrong attempt and the error message, replacing it with the new, correct line of reasoning.

4. The Result: A "Purified" Trajectory
By the time the training is done, the apprentice has never "seen" the messy, error-filled versions of their work. They have only learned from "self-purified" stories where they solved the problem correctly the first time (or fixed it instantly without leaving a trace of the struggle).

Why This Matters

Less Noise, More Learning: Because the apprentice isn't distracted by a history of errors, they learn the correct logic much faster.
Efficiency: The paper claims this method is incredibly efficient. They managed to train a small AI model to perform as well as much larger, more expensive models, but they did it using only one-third of the training steps. It's like getting a master chef's skills in three months instead of a year.
Better Results: On difficult math and coding tests (like AIME and LiveCodeBench), this method improved accuracy by about 3% to 6% compared to standard methods.

In a Nutshell

Think of standard training as letting a student practice on a whiteboard covered in erased, smudged, and confusing scribbles. CLEANER is like a magical eraser that instantly cleans up the smudges, leaving only the perfect, clear steps of the solution. This allows the student to internalize the right way to think, rather than getting confused by the wrong way they tried first.

Technical Summary: CLEANER – Self-Purified Trajectories Boost Agentic Reinforcement Learning

1. Problem Statement

The paper addresses a critical bottleneck in Agentic Reinforcement Learning (RL) for Large Language Models (LLMs), particularly for parameter-constrained models (e.g., 4B–7B parameters) that utilize tools like Python interpreters. While tool use enables complex problem-solving, the exploration phase is plagued by frequent execution failures. These failures generate "noisy trajectories" containing invalid code and verbose tracebacks that contaminate the context window.

Under standard outcome-based reward settings (e.g., Group Relative Policy Optimization or GRPO), the entire trajectory receives a uniform positive reward upon final success, regardless of intermediate errors. This creates a credit assignment dilemma:

Noise Pollution: Erroneous actions and the subsequent "Error → Feedback → Retry" loops are inadvertently reinforced alongside successful reasoning.
Semantic Interference: The model learns to rationalize incorrect execution paths rather than re-grounding decisions, leading to policy instability and performance degradation.
Existing Limitations: Current mitigation strategies face a trade-off. Assigning dense rewards often triggers "reward hacking" (optimizing intermediate metrics over final outcomes), while supersampling (generating multiple candidates to filter for quality) incurs prohibitive computational costs, as the rollout phase dominates RL training time.

2. Methodology: CLEANER and SAAR

The authors propose CLEANER, a framework that boosts agentic RL by eliminating error-contaminated context directly during data collection, rather than relying on external filtering or increased sampling. The core mechanism is Similarity-Aware Adaptive Rollback (SAAR).

The SAAR Mechanism

SAAR operates during the rollout phase when an execution error ( $o^-$ ) occurs. Instead of permanently recording the failure in the history, the system initiates a two-phase process to construct a "self-purified" trajectory:

Phase I: Error Trigger and Lookahead Correction
- Upon an error, the current state is frozen.
- The model is prompted with the augmented context (including the error trace) to generate a correction ( $c'_t$ ) and an auxiliary reasoning trace ( $r'_{aux}$ ).
- If the correction succeeds, the process moves to Phase II.
Phase II: Similarity-Aware Adaptive Replacement
- SAAR calculates the semantic similarity between the erroneous code ( $c_t$ ) and the corrected code ( $c'_t$ ) using a similarity function (e.g., difflib.SequenceMatcher).
- Case A: Implementation-Level Repair (High Similarity): If the similarity is above a threshold $\gamma$ , the error is deemed superficial (e.g., syntax typos). The system performs a shallow replacement, discarding the error tuple and grafting the corrected code directly onto the original reasoning.
- Case B: Reasoning-Level Repair (Low Similarity): If similarity is low, the error indicates a deep logical flaw. The system performs a deep replacement, removing the entire failed turn (reasoning + code + error) and substituting it with the auxiliary reasoning and corrected code to maintain semantic coherence.

Implementation Details

Logit Recomputation: To ensure the policy update is grounded in the correct causal path, the system recomputes log-probabilities for the corrected actions under the purified context (not the error-augmented context). This is achieved efficiently using RadixAttention (via SGLang) to reuse KV caches for invariant history prefixes.
Curriculum Mixing: To balance error avoidance with the model's intrinsic ability to self-correct, the training pipeline stochastically applies SAAR to 70% of trajectories while retaining 30% in their original state.

3. Key Contributions

The paper outlines four primary contributions:

CLEANER Framework: A novel approach to resolve the credit assignment dilemma in agentic RL by training on self-purified trajectories, allowing models to internalize correct reasoning patterns while filtering execution noise.
SAAR Mechanism: An autonomous method to construct clean signals by adaptively repairing failures (from syntax typos to logical flaws) without the computational overhead of supersampling.
Efficiency and Performance: Demonstrating that CLEANER achieves state-of-the-art (SOTA) performance with significantly higher efficiency, matching SOTA models while requiring only one-third of the training steps.
Reproducibility: Provision of a fully reproducible training pipeline, including code, environment configurations, and processed datasets.

4. Experimental Results

The authors evaluated CLEANER on AIME24/25, GPQA, and LiveCodeBench using Qwen3-4B-Instruct and Qwen2.5-7B-Instruct as base models.

Performance Gains: Compared to baselines, CLEANER achieved average accuracy gains of:
- 6% on AIME (Pass@1).
- 3% on GPQA.
- 5% on LiveCodeBench.
Efficiency: CLEANER matched the performance of the SOTA baseline DemyAgent-4B while utilizing only one-third of the training steps (250 steps vs. 750 steps).
Error Suppression: Training metrics showed a consistent suppression of erroneous tool calls, with a clear correlation between reduced tool failure rates and improved pass rates.
Ablation Studies:
- Necessity of Tools: Integrating a code interpreter significantly boosted performance (e.g., +20% on Qwen2.5-7B for math/coding tasks).
- Trajectory Purification: The SAAR mechanism provided consistent gains across all benchmarks and model scales.
- Internalization: When SAAR was deactivated during inference, performance degradation was negligible (e.g., <1% drop on AIME24 Pass@1), confirming that the model had internalized the error-avoidance logic rather than relying on the mechanism as a scaffold.
- Recovery: Introducing CLEANER to a suboptimal, unstable baseline (at step 200) stabilized training and improved accuracy, though it did not fully reach the performance of models trained with SAAR from scratch.

5. Significance and Claims

The paper claims that CLEANER offers a scalable solution for efficient agentic RL by shifting the focus from "more data" (supersampling) to "cleaner data" (trajectory purification).

Scalability: By eliminating the need for expensive supersampling, the method makes high-quality agentic RL feasible for resource-constrained settings.
Learning Signal Alignment: The method aligns training signals with correct behavior, preventing the reinforcement of error-recovery loops and enabling models to internalize robust tool usage patterns.
Generalizability: The results suggest that small models (4B), when subjected to tailored post-training with trajectory purification, can achieve capabilities comparable to significantly larger models (e.g., 72B or 671B) in agentic reasoning tasks.

The authors conclude that trajectory purification is a critical, scalable alternative for advancing the efficiency and performance of tool-augmented LLM agents.