🤖 machine learning

Do Not Waste Your Rollouts: Recycling Search Experience for Efficient Test-Time Scaling

This paper introduces Recycling Search Experience (RSE), a training-free strategy that enhances test-time scaling by distilling and reusing intermediate insights from failed and successful rollouts to eliminate computational redundancy and improve reasoning efficiency on complex tasks.

Original authors: Xinglin Wang, Jiayi Shi, Shaoxiong Feng, Peiwen Yuan, Yiwei Li, Yueqi Zhang, Chuyi Tan, Ji Zhang, Boyuan Pan, Yao Hu, Kan Li

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinglin Wang, Jiayi Shi, Shaoxiong Feng, Peiwen Yuan, Yiwei Li, Yueqi Zhang, Chuyi Tan, Ji Zhang, Boyuan Pan, Yao Hu, Kan Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Amnesia" of AI

Imagine you are trying to solve a very difficult maze. You send out 100 different explorers (these are the AI's "rollouts" or attempts) to find the exit.

The Old Way (Current AI Search): Each explorer starts from the beginning, runs into a wall, turns around, and tries a different path. When they hit a dead end, they give up. The next explorer starts fresh, runs into the same wall, and hits the same dead end. They might even find a shortcut that the first explorer found, but they have to rediscover it from scratch.
The Waste: The AI is wasting massive amounts of energy (computing power) re-deriving facts it already knows and re-walking paths it already knows lead nowhere. It's like a student taking a test, failing, and then taking the exact same test again without looking at their previous mistakes or notes.

The Solution: "Recycling Search Experience" (RSE)

The authors propose a new strategy called Recycling Search Experience (RSE). Think of this as giving the explorers a shared, living map that gets updated after every round of exploration.

Instead of treating every attempt as a disposable trial, RSE treats the search as a cumulative team effort. Here is how it works in three simple steps:

1. The "Distillation" (Making Notes)

After a batch of explorers finishes their run, the AI doesn't just throw away their raw, messy logs. Instead, it acts like a smart editor who reads through the chaos and writes two specific lists:

The "Good News" List (Positive Experience): "Hey, we found out that turning left at the fountain is a dead end, but going straight leads to a bridge." These are verified facts and shortcuts.
The "Bad News" List (Negative Experience): "Do not go down the dark tunnel; it leads to a pit." These are known dead ends and logical traps to avoid.

2. The "Shared Bank" (The Memory)

These lists are stored in a Shared Experience Bank. This isn't a giant, messy pile of papers; it's a curated, organized database. The AI uses a "deduplication" filter to make sure it doesn't write the same note twice (e.g., it won't list "Don't go left" ten times; it just keeps one clear warning).

3. The "Guided Search" (Using the Map)

When the next batch of explorers starts, they don't start from zero. They are handed this Shared Bank.

If they see a "Good News" note, they can skip the long walk to the bridge and go straight there (shortcutting the work).
If they see a "Bad News" note, they immediately turn away from the dark tunnel (pruning the dead ends).

Why This is a Game Changer

The paper claims this method is much more efficient than the old ways.

Analogy: Imagine trying to find a specific needle in a haystack.
- Old Way: You send 100 people to dig blindly. They all dig the same spots, miss the needle, and get tired.
- RSE Way: The first 10 people dig, find some dirt that isn't the needle, and mark those spots. The next 10 people are told, "Don't dig here." They also find a spot that might be the needle and mark it. The next group skips the bad spots and focuses on the promising ones.
The Result: You find the answer faster and with less energy because you aren't wasting time on mistakes you've already made.

What the Paper Actually Found

The researchers tested this on very hard math problems (like those found in high-level competitions like the IMO or HMMT), coding challenges, and complex planning tasks (like planning a multi-day trip).

Better Results: The AI using RSE got higher scores than other methods that just tried to "think harder" or "try more times" without sharing notes.
Works on Hard Problems: It was especially good at solving the hardest problems where other methods got stuck or gave up.
No Extra Training: The best part is that the AI didn't need to be retrained. It just changed how it searched for answers during the test, using its own internal ability to critique its work.

The Bottom Line

The paper argues that intelligence isn't just about trying harder; it's about remembering what you've learned. By turning "disposable" attempts into a "cumulative" memory, the AI stops wasting energy on dead ends and redundant discoveries, making it smarter and more efficient without needing bigger hardware.

Technical Summary: Do Not Waste Your Rollouts: Recycling Search Experience for Efficient Test-Time Scaling

Problem Statement

Current Test-Time Scaling (TTS) strategies, which enhance Large Language Model (LLM) reasoning by allocating additional inference compute, suffer from a fundamental inefficiency: information waste. Existing paradigms—whether parallel sampling, sequential refinement, or tree-based search—typically treat individual rollouts (reasoning trajectories) as disposable samples. Once a trial is completed or discarded, the valuable intermediate insights, verified propositions, and identified failure patterns contained within that trajectory are effectively lost.

This "memoryless" approach leads to substantial computational redundancy. Models repeatedly re-derive conclusions that have already been discovered in other rollouts and revisit known dead ends, failing to leverage the cumulative knowledge generated during the search process. The paper argues that maximizing the potential of test-time compute requires not just increasing the volume of rollouts, but optimizing the quality of exploration by transforming disposable trials into a cumulative, experience-guided process.

Methodology: Recycling Search Experience (RSE)

To address this inefficiency, the authors propose Recycling Search Experience (RSE), a self-guided, training-free inference strategy. RSE reframes test-time search from a series of isolated trials into a cumulative process where valuable insights from prior trajectories are distilled and reused to guide subsequent exploration.

The methodology operates through three coordinated components:

Batched Experience-Guided Search:
Instead of real-time synchronization (which is unstable) or purely sequential refinement (which is limited by context windows), RSE partitions the rollout budget into $R$ rounds. In each round $r$ , the model generates a batch of $K_r$ trajectories in parallel. Before each round, a global Experience Bank is serialized into the prompt, ensuring all rollouts in the batch start from a synchronized state containing accumulated knowledge from previous rounds.
Self-Guided Experience Distillation:
Rather than concatenating raw, verbose reasoning chains (which risks context overflow and noise), RSE employs a lightweight prompting step to distill each completed trajectory into structured, discrete items. This process leverages the model's intrinsic self-assessment capability without external supervision:
- Positive Experience ( $E^{pos}$ ): Extracts verified propositions, lemmas, or intermediate conclusions that serve as "truth anchors" to shortcut redundant derivations.
- Negative Experience ( $E^{neg}$ ): Identifies critical pitfalls, strategic dead ends, or logical fallacies to act as "negative constraints" for pruning the search space.
Semantic Experience Deduplication:
To prevent context explosion and maintain high information density, RSE employs a semantic deduplication strategy. New distilled experiences are compared against the existing bank using a similarity threshold ( $\tau$ ). Only semantically distinct items are added, filtering out repetitive insights and ensuring the bank remains a diverse set of actionable guidance rather than a redundant list.

Key Contributions

The paper outlines three primary contributions:

RSE Framework: The proposal of a training-free strategy that turns test-time search into a cumulative, experience-guided process by recycling rollout-level experience.
Theoretical Analysis: A formal analysis demonstrating that RSE offers a sample-efficiency advantage over independent sampling. The authors prove that under sound experience extraction, the probability of finding a correct solution with RSE is no worse than independent sampling, with the potential for exponential gains in complex tasks requiring multiple intermediate conclusions. They also provide a bounded-degradation result for scenarios involving imperfect experience extraction.
Empirical Validation: Extensive experiments across diverse benchmarks and model families showing consistent performance gains over strong baselines.

Experimental Results

The authors evaluated RSE on challenging mathematical reasoning benchmarks (HMMT24, HMMT25, IMO-Bench, HLE-Math-text) and extended the evaluation to coding (LiveCodeBench-v6), expert-level QA (GPQA-Diamond), and long-horizon planning (TravelPlanner).

Performance Superiority: RSE consistently outperformed baselines including Standard Sampling, Majority Voting, Self-Refine, and the concurrent PaCoRe method across all tested models (Qwen3-30B, Qwen3-4B, Phi-4, DeepSeek-V3.2). For instance, on HMMT25 with the Qwen3-30B model, RSE achieved 83.9% Pass@1 by Iteration 3, surpassing PaCoRe (80.2%) and Self-Refine (72.6%).
Scaling Dynamics: Unlike baselines that exhibit early saturation (e.g., PaCoRe hitting a ceiling by Iteration 3 due to a "verification-centric bottleneck"), RSE demonstrated delayed convergence and continued performance gains through later iterations, indicating a higher empirical ceiling.
Compute Efficiency: RSE established a superior Pareto frontier in terms of computational cost (FLOPs) versus accuracy. It achieved higher accuracy gains per unit of compute compared to simple aggregation or verification-based concatenation.
Generalizability: The method proved effective beyond mathematics. Notably, on the TravelPlanner task, PaCoRe collapsed to 0.00% pass rate due to context overload, whereas RSE achieved a 34.44% pass rate, demonstrating its ability to handle open-ended planning tasks where consensus voting is inapplicable.

Significance and Claims

The paper claims that RSE represents a paradigm shift in test-time scaling. By treating rollouts not as disposable samples but as a source of reusable experience, RSE addresses the systemic inefficiency of current "memoryless" search.

Efficiency: The authors argue that the primary bottleneck in current TTS is not the lack of compute, but the lack of systematic mechanisms to accumulate and reuse rollout-level experience. RSE bridges this gap, allowing models to concentrate compute on promising, unexplored regions of the solution space rather than re-deriving known facts or revisiting dead ends.
Self-Guided Nature: A key significance is that RSE requires no external reward models, ground-truth labels, or architectural changes. It relies solely on the model's ability to critique and distill its own reasoning, making it applicable to general search environments.
Theoretical Grounding: The work provides a theoretical foundation for the intuition that "checkpointing" valid intermediate conclusions allows later rollouts to build on fragmented partial successes, significantly reducing the probability decay associated with complex reasoning chains.

In conclusion, the paper posits that maximizing the potential of test-time compute requires optimizing the quality of exploration through experience recycling, transforming isolated trials into a cumulative, intelligent search process.