💻 computer science

Speculative Actions: A Lossless Framework for Faster Agentic Systems

This paper introduces "Speculative Actions," a lossless framework that accelerates agentic systems by using faster models to predict and parallelize likely future actions, achieving up to 20% latency reductions across diverse domains while providing a formal cost-latency analysis to optimize the tradeoff between speculation breadth and efficiency.

Original authors: Naimeng Ye, Arnav Ahuja, Georgios Liargkovas, Yunan Lu, Kostis Kaffes, Tianyi Peng

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Naimeng Ye, Arnav Ahuja, Georgios Liargkovas, Yunan Lu, Kostis Kaffes, Tianyi Peng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a game of chess against a grandmaster who thinks incredibly deeply but very slowly. Every time it's your turn, you have to wait 10 minutes for them to make a move. While you wait, you just sit there, staring at the board, doing absolutely nothing.

Now, imagine you have a clairvoyant assistant sitting next to you. This assistant is much faster than the grandmaster, though not quite as perfect. While the grandmaster is still thinking, your assistant whispers, "I bet they will move their Knight here, or maybe their Bishop there."

Instead of just waiting, you immediately start preparing your counter-move for both of those possibilities. If the grandmaster actually moves the Knight, you are already ready to strike instantly. If they move the Bishop, you switch to your other plan. If they do something totally unexpected, you just discard your preparation and start fresh, losing only a tiny bit of time.

This is exactly what the paper "Speculative Actions" is about.

Here is the breakdown of their idea using simple analogies:

1. The Problem: The "Waiting Room" Bottleneck

Currently, AI agents (like chatbots that can book flights or play games) work in a strict line.

Step 1: The AI thinks.
Step 2: The AI asks a tool (like a search engine) for info.
Step 3: The AI waits for the tool to reply.
Step 4: The AI thinks again.

The problem is Step 3. The AI is often faster than the tools it uses. It spends a huge amount of time just sitting in a "waiting room" for the internet or a database to reply. This makes the whole process painfully slow.

2. The Solution: The "Guess-and-Go" Strategy

The authors propose a framework called Speculative Actions. They borrow an idea from computer chips (processors) that have been doing this for decades.

They split the AI's job into two roles:

The Actor (The Boss): This is the slow, super-smart, expensive AI. It makes the final, perfect decisions.
The Speculator (The Intern): This is a fast, cheap, slightly less smart AI. Its job is to guess what the Boss is going to do next.

3. How It Works (The Analogy)

Imagine you are ordering a complex meal at a fancy restaurant.

Without Speculation: You tell the waiter your order. The waiter walks to the kitchen, waits for the chef to cook, brings it back, and then you ask for dessert. You are stuck waiting at the table the whole time.
With Speculation:
1. You tell the waiter your main course.
2. The Speculator (Intern) immediately guesses, "The customer will probably want the chocolate cake for dessert."
3. The Intern runs to the kitchen and starts prepping the cake while the main course is still cooking.
4. Scenario A (The Guess was Right): The main course arrives, and you say "Chocolate cake!" The Intern hands it to you instantly. You saved 5 minutes!
5. Scenario B (The Guess was Wrong): You actually wanted the fruit salad. No problem! The kitchen just throws away the prepped cake (or saves it for later) and starts the fruit salad. You didn't lose any time; you just wasted a tiny bit of prep effort.

4. Where They Tested It

The team tested this "Intern vs. Boss" system in four different worlds:

Chess: While the slow AI thinks about its move, the fast AI guesses what the opponent will do and starts planning the counter-move.
Shopping: While the AI waits for a database to check if an item is in stock, the fast AI guesses the customer wants to return an item and starts checking the return policy in advance.
Web Search: While waiting for a slow Wikipedia search, the fast AI guesses the answer and prepares the next search query.
Operating Systems: Here, they took a riskier approach (called "lossy"). The fast AI makes quick, temporary changes to the computer's settings to keep it running smoothly, and the slow AI fixes them later if they were wrong. This made the computer feel much snappier.

5. The Results

Speed: They reduced the waiting time by up to 20%. In some cases, the AI was able to guess the next step correctly 55% of the time.
Safety: The system is "lossless," meaning if the guess is wrong, the final result is exactly the same as if they had waited. It's like having a safety net.
Cost: They figured out the math to ensure they don't guess too much. If you guess 10 things at once, it might cost too much money. They found the "sweet spot" where you guess enough to save time but not so much that it costs a fortune.

The Big Takeaway

This paper teaches us that waiting is optional.

By using a fast, cheap "guessing" AI to do the heavy lifting while we wait for the slow, smart AI to finish its work, we can make AI agents feel instant and responsive, even when they are doing complex tasks that usually take hours. It turns "dead time" into "productive time."

1. Problem Statement

Large Language Model (LLM) driven agents are increasingly deployed in complex, interactive environments (e.g., web browsers, operating systems, game engines). However, their runtime efficiency is severely bottlenecked by sequential API latency.

Sequential Dependency: Current agents operate in a strict "think-act-observe" loop. Each step requires an LLM call or external tool invocation (e.g., web search, database query) that blocks execution until a response is received.
Impact: This serialization leads to excessive wall-clock times (ranging from minutes to hours for complex tasks like chess games or deep research), making agents impractical for real-time interaction and inefficient for reinforcement learning or prompt optimization loops requiring thousands of iterations.
Core Question: Must an agent interact with its environment in a strictly sequential manner? The authors argue no.

2. Methodology: Speculative Actions

Inspired by speculative execution in microprocessors and speculative decoding in LLM inference, the authors propose a general framework called Speculative Actions.

Core Architecture

The framework introduces two distinct roles within the agent loop:

Actor (The Authority): A slow, high-capability executor (e.g., SOTA LLM, external API, human) that generates the ground-truth action.
Speculator (The Predictor): A fast, low-latency model (e.g., smaller LLM, reduced prompt, heuristic) that predicts the Actor's next move and the subsequent environment state.

The Workflow

Parallel Execution: While the Actor is processing the current step $t$ , the Speculator predicts the likely next action $\hat{a}_t$ and the subsequent state $\hat{s}_{t+1}$ .
Pre-launch: Based on the prediction, the system speculatively launches the API calls required for step $t+1$ (and potentially further steps) in parallel. These calls are cached as "pending actions."
Validation & Commit:
- Once the Actor returns the true action $a_t$ , the system checks if it matches any of the Speculator's predictions.
- Hit: If a match is found, the system immediately commits to the pre-launched branch, skipping the latency of generating the next request.
- Miss: If no match is found, the speculative branches are discarded (or rolled back), and the system proceeds with the standard sequential flow.

Safety and Losslessness

To ensure the framework is lossless (i.e., it does not degrade the final outcome compared to a sequential agent), the authors implement:

Semantic Guards: Actors confirm state transitions before committing.
Safety Envelopes: Speculative side effects are restricted to idempotent, reversible, or sandboxed operations (e.g., pre-fetching data, sandboxed OS changes).
Repair Paths: Mechanisms to rollback or compensate if a guess is rejected.

3. Key Contributions

A. Theoretical Framework & Analysis

Formal Modeling: The problem is modeled as a Markov Decision Process (MDP) where actions are API calls.
Cost-Latency Tradeoff: The authors derive closed-form expressions for the expected runtime and token cost.
- They prove that with $k$ parallel speculative branches and prediction accuracy $p$ , the latency reduction has an upper bound of 50% (in the limit of infinite speculation time) for breadth-focused strategies.
- They introduce a confidence-aware selective speculation strategy, showing that dynamically selecting branches based on predicted confidence minimizes cost while maximizing latency savings.
Depth vs. Breadth: The paper analyzes "breadth-focused" (parallel single-step) vs. "depth-focused" (multi-step lookahead) strategies, showing depth speculation can theoretically achieve up to 100% speedup but requires more complex pruning.

B. Empirical Validation Across Domains

The framework was tested in four distinct environments:

Turn-based Gameplay (Chess):
- Setup: Speculator predicts opponent moves while the Actor thinks.
- Result: Achieved 54.7% prediction accuracy with 3 branches, reducing game time by 19.5%.
E-Commerce Dialogue:
- Setup: Speculator predicts user intent and necessary API calls (e.g., checking return eligibility) during user typing time.
- Result: 22–38% API call accuracy. Since speculation takes ~2-3 seconds (vs. 30s user typing), agents can respond instantly in ~1/3 of turns.
Multi-hop Web Search (HotpotQA):
- Setup: Speculator predicts the next Wikipedia search query while the current API call executes.
- Result: 46% accuracy with top-3 predictions, significantly reducing idle time during network latency.
Operating Systems (Lossy Extension):
- Setup: OS parameter tuning (CFS min-granularity). Here, "lossless" is relaxed; speculative changes are applied immediately and overwritten by the Actor if wrong.
- Result: The joint system converged to optimal settings in 13 seconds (vs. 200s for Actor-only) with lower total cost due to faster convergence.

4. Key Results

Latency Reduction: Across domains, the framework achieved up to 20% end-to-end latency reduction.
Prediction Accuracy: Next-action prediction accuracy reached up to 55% in controlled environments.
Cost Efficiency:
- In lossless settings, the cost increase is sub-linear relative to the number of branches due to the "kill" mechanism for duplicate or incorrect branches.
- In the OS tuning (lossy) setting, the total cost decreased because the system converged to the optimal solution much faster, avoiding the waste of exploring poor configurations.
Scalability: The framework works with heterogeneous model ensembles (e.g., combining GPT-5-nano with Gemini-2.5-flash) to optimize the speed/accuracy tradeoff.

5. Significance

Paradigm Shift: Moves agentic systems from strictly sequential processing to parallel, opportunistic execution, fundamentally changing how agents interact with environments.
Generalizability: Unlike previous work focused solely on LLM token generation or planning, this framework applies to any API call, including tool usage, MCP servers, and human-in-the-loop interactions.
Practical Impact: Provides a principled way to tune the "speculative breadth" (number of parallel guesses) based on the specific cost/latency constraints of the deployment environment.
Future of Agents: Demonstrates that "waiting" time in agent loops can be transformed into productive computation, making real-time, high-throughput agentic automation feasible.

In summary, Speculative Actions offers a robust, theoretically grounded method to accelerate agentic systems by decoupling the decision-making latency from the execution latency, achieving significant speedups without compromising the correctness of the final agent trajectory.