💻 computer science

IdleSpec: Exploiting Idle Time via Speculative Planning for LLM Agents

IdleSpec is a scalable inference approach that improves LLM agent performance and reduces latency by leveraging idle waiting periods to speculatively generate and aggregate plan candidates using adaptive drafting strategies that account for observation uncertainty.

Original authors: Daewon Choi, Kyunghyun Park, Woomin Song, Saket Dingliwal, Sai Muralidhar Jayanthi, Jinwoo Shin, Aram Galstyan

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daewon Choi, Kyunghyun Park, Woomin Song, Saket Dingliwal, Sai Muralidhar Jayanthi, Jinwoo Shin, Aram Galstyan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a complex mystery. You have a team of specialists (like a forensic lab, a database expert, or a field agent) who need to run tests, search records, or interview witnesses.

Here's the problem: While your specialists are working, you sit around doing nothing. You wait for the lab to finish the DNA test, you wait for the database to return the search results, and you wait for the field agent to come back with a report. In the world of AI agents, this waiting time is called "idle time."

For a long time, researchers thought this waiting time was just wasted energy. They thought, "Oh well, the computer has to wait, so let's just sit tight."

The paper "IdleSpec" introduces a clever new way to use this waiting time. It's like giving the detective a notepad and saying, "While the lab is working, don't just stare at the wall. Write down three different theories about what the results might be and how you'll solve the case next."

The Core Idea: Speculative Planning

The authors call their method IdleSpec. Here is how it works in simple terms:

1. The "Waiting Room" Strategy
When the AI agent sends a tool to do something (like searching the web or running code), it usually just waits. IdleSpec says, "No, let's use this time!" While the tool is running, the AI starts drafting plans. It doesn't just wait; it actively thinks ahead.

2. The Two Types of Guesses (Progressive vs. Recovery)
The tricky part is that the AI doesn't know what the tool will find yet. The tool might find the perfect answer, or it might return a dead end. To handle this, IdleSpec uses two different "personas" to write its plans:

The Optimist (Progressive): This persona assumes the tool will work perfectly. It writes a plan for "What do we do next if we get the good news?" It keeps moving forward.
The Realist (Recovery): This persona assumes the tool might fail or give a weird result. It writes a plan for "What do we do if this goes wrong?" It prepares a backup route.

By writing both types of plans while waiting, the AI is ready for anything that happens next.

3. The "Post-Game" Review
Once the tool finally finishes and the results arrive, the AI stops writing new plans. It looks at the actual results and the plans it wrote earlier. It then picks the best parts of those plans to decide its next move.

4. Learning from Experience
The AI also keeps a scorecard. If the "Optimist" plan usually works, it writes more of those next time. If the "Realist" plan saves the day often, it writes more of those. It learns which type of thinking is best for the current situation.

Why This is a Big Deal

The paper tested this on three different types of "detective work":

General Puzzles (GAIA): Tasks requiring web searches and file reading.
Multi-Step Searches (FRAMES): Tasks where you have to search, find a clue, search again, and find another clue.
Coding Challenges (MLE-Bench): Tasks where the AI has to write code and wait for it to run (which can take a long time).

The Results:

Better Accuracy: In all these tests, the AI using IdleSpec got more questions right than the AI that just sat and waited. For example, on the coding challenges, it improved the rate of winning "medals" (top scores) by nearly 10%.
No Extra Waiting: The best part is that this didn't make the AI slower. Because the AI was thinking while the tools were working, the total time to finish the task stayed the same. It's like getting a head start without running faster.
Better than "Sleeping": The paper compared this to an older method called "Sleep-Time Compute," which tried to guess the future but often got it wrong and confused the AI. IdleSpec's method of preparing for both success and failure worked much better.

The Analogy Summary

Think of a chef cooking a complex meal.

The Old Way: The chef puts a pot on the stove to boil water and then stands there staring at it for 10 minutes, doing nothing.
The IdleSpec Way: The chef puts the pot on the stove. While the water is heating up, the chef chops vegetables, preps the spices, and writes down two different recipes: one if the water boils fast, and one if it takes longer. When the water is ready, the chef is already halfway through the cooking process.

In short: IdleSpec turns "dead time" into "thinking time," allowing AI agents to solve harder problems faster without actually waiting any longer.

Technical Summary: IdleSpec

Problem Statement

Large Language Model (LLM)-based agents solve complex tasks through iterative reasoning loops involving tool calls (e.g., web search, code execution) and environment interactions. A critical inefficiency in this paradigm is idle time: the waiting period incurred while the agent awaits observations from external tools. While prevalent, this time is typically treated as unavoidable overhead.

Existing approaches to utilizing idle time suffer from significant limitations:

System Efficiency Focus: Prior works often focus on asynchronous execution or system-level scheduling to overlap independent computations, rather than using the time for additional reasoning.
Suboptimal Agentic Strategies: Recent attempts like Sleep-Time Compute treat idle periods as opportunities for pre-computation but rely on restrictive assumptions. They often assume future queries are predictable and utilize only a small fraction of available idle time (e.g., 13.7% on GAIA). Furthermore, they tend to use a single "forward-progress" mode, which fails when tool observations are inherently uncertain, leading to hallucinations or degraded performance when the pre-computed context does not match the actual observation.
Variable Budgets: The duration of idle time varies significantly across different tool calls and benchmarks, making static allocation strategies ineffective.

The core challenge is how to exploit this substantial but highly variable idle time to improve agent performance without introducing latency overhead, specifically under conditions of observation uncertainty.

Methodology: IdleSpec

The authors propose IdleSpec (Exploiting Idle Time via Speculative Planning), a scalable inference framework designed to utilize idle time via speculative planning. The framework operates on a two-phase strategy:

1. Idle-Time Drafting (Speculative Generation)

During the execution of a tool call, the agent iteratively generates candidate plans. To handle the uncertainty of the yet-to-be-observed result, IdleSpec employs two complementary drafting strategies:

Progressive Drafting: Assumes the forthcoming observation will be successful and contributes to task success. The agent drafts the next logical step to maintain forward momentum.
Recovery Drafting: Assumes the current action may fail or yield unproductive results (e.g., empty search results). The agent drafts an alternative plan to pursue the same sub-goal from a different angle.

The agent samples between these strategies using a learned distribution. Drafting is iterative and terminates immediately upon the arrival of the observation, ensuring high utilization of the available time window without exceeding the tool's execution duration.

2. Draft Aggregation and Strategy Selection

Once the observation arrives:

Aggregation: The agent aggregates the set of generated candidate plans (both progressive and recovery) with the actual observation. Crucially, these plans are treated as reference points rather than strict constraints. The agent synthesizes a refined next step, using the drafts to guide reasoning while remaining free to deviate if the drafts do not fit the new context.
Posterior Updates (Adaptive Sampling): To determine which strategy (Progressive vs. Recovery) to prioritize in future idle periods, IdleSpec uses a Thompson sampling mechanism. After each step, the agent produces a binary forecast signal ( $\ell \in \{PROG, REC\}$ ) evaluating whether the current trajectory is on track. This signal updates a Beta posterior distribution over strategy probabilities, adaptively biasing future drafting toward the strategy that has proven effective in the current context.

Key Contributions

Systematic Analysis of Idle Time: The paper establishes that idle time constitutes a substantial reasoning budget across diverse benchmarks (GAIA, FRAMES, MLE-Bench) but is highly variable in duration. It further demonstrates that planning is the most effective strategy for utilizing this time compared to summarization or reflection, as planning can condition on plausible future observations rather than committing to incomplete interpretations.
IdleSpec Framework: Introduction of a novel inference-time method that combines speculative progressive and recovery planning with adaptive strategy sampling. This approach effectively manages observation uncertainty and maximizes idle-time utilization.
Empirical Validation: Comprehensive experiments across three distinct agentic benchmarks demonstrating consistent performance gains over vanilla baselines and existing idle-time methods.

Experimental Results

The authors evaluated IdleSpec on three benchmarks using various LLM backbones (including Gemini-2.5-Flash, Gemma4-E4B, and Qwen3.5-4B):

GAIA (General AI Assistant): IdleSpec achieved an average accuracy of 55.6% with Gemini-2.5-Flash, surpassing the vanilla baseline by 5.1%. Improvements were consistent across all difficulty levels and model sizes (e.g., +6.8% on Qwen3.5-4B). In contrast, Sleep-Time Compute showed limited gains and occasionally degraded performance, particularly on smaller models.
FRAMES (Multi-hop Search): IdleSpec consistently outperformed baselines, achieving an average accuracy of 55.6% (Gemini-2.5-Flash) compared to 50.5% for the vanilla baseline.
MLE-Bench (Machine Learning Engineering): In this execution-heavy environment with long code-execution delays, IdleSpec improved the Any Medal rate from 36.4% to 45.5% (a 9.1% gain) and increased the valid submission rate from 77.3% to 86.4%.
Efficiency and Latency: Unlike test-time scaling methods (e.g., Sequential Revision, Planning) that increase end-to-end latency by serializing extra computation, IdleSpec achieves Pareto improvement. It matches the accuracy of test-time scaling methods while maintaining latency close to the vanilla baseline because the additional computation occurs during otherwise wasted idle periods.
Ablation Studies: The results confirm that using both progressive and recovery drafts is superior to using either alone. Furthermore, treating drafts as references (aggregation) rather than mandatory constraints is critical for performance.

Significance and Claims

The paper claims that IdleSpec offers a robust, generalizable solution for exploiting idle time in agentic interactions. Its significance lies in:

Reframing Idle Time: It shifts the perspective of idle time from a delay to a usable computational resource that can improve agent performance with minimal latency overhead.
Handling Uncertainty: By explicitly modeling observation uncertainty through dual drafting strategies (progressive and recovery) and adaptive sampling, IdleSpec avoids the failure modes of prior methods that assume predictable futures.
Scalability: The framework is compatible with existing test-time scaling methods and can be applied across diverse agent architectures and tool suites without modifying the underlying model.

The authors conclude that IdleSpec demonstrates that effective utilization of idle time is a promising direction for building more capable LLM agents, particularly for complex, long-horizon tasks where tool execution delays are significant.