🤖 AI

Executing as You Generate: Hiding Execution Latency in LLM Code Generation

The paper introduces "Eager," a parallel execution framework that overlaps code generation and execution to eliminate idle time, achieving up to 99.9% reduction in non-overlapped latency and up to 55% faster end-to-end performance across multiple benchmarks and models.

Original authors: Zhensu Sun, Zhihao Lin, Zhi Chen, Chengran Yang, Mingyi Zhou, Li Li, David Lo

Published 2026-04-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhensu Sun, Zhihao Lin, Zhi Chen, Chengran Yang, Mingyi Zhou, Li Li, David Lo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef (the AI) trying to cook a complex meal for a customer (the User).

The Old Way: The "Wait-and-See" Kitchen (Serial Execution)

Currently, most AI coding systems work like a very strict, old-school kitchen:

The chef writes the entire recipe on a piece of paper, from the first ingredient to the final garnish.
Only after the chef finishes writing the whole thing does the kitchen staff (the Executor) start cooking.
The customer sits at the table, staring at an empty plate, waiting for the chef to finish writing and then waiting for the food to be cooked.

The Problem: The kitchen staff sits idle while the chef writes. The chef sits idle while the kitchen cooks. The customer waits for the sum of both times. It's inefficient and feels slow.

The New Way: The "Assembly Line" Kitchen (Parallel Execution)

The paper introduces a new system called Eager. Instead of waiting for the whole recipe, the chef starts handing ingredients to the kitchen staff as soon as they are written down.

The chef writes "Chop the onions."
Immediately, the kitchen staff starts chopping the onions while the chef is still writing "Sauté the garlic."
The chef writes "Add the tomatoes."
The kitchen staff adds the tomatoes while the chef is writing "Simmer for 10 minutes."

The Result: By the time the chef finishes writing the last sentence, the kitchen has already been cooking for a while. The customer gets their food much faster because the "cooking time" is hidden inside the "writing time."

How Does It Actually Work? (The Magic Tricks)

The paper explains that for this to work without making a mess, the system uses three clever tricks:

1. The "Smart Sniffer" (AST-Based Chunking)

You can't just hand the kitchen staff a single word like "Chop" and expect them to know what to do. They need a complete instruction like "Chop the onions."

The Analogy: The system acts like a smart sniffer dog. It watches the chef write. As soon as it sees a complete, logical sentence (a "chunk"), it grabs it and hands it to the kitchen. If the sentence isn't finished yet (e.g., the chef wrote "Chop the o..." and stopped), the sniffer waits for the next word before acting. This ensures the kitchen never tries to cook a half-finished recipe.

2. The "Batching Chef" (Dynamic Batching)

Sometimes the kitchen staff is slower than the chef. If the chef writes 10 instructions in a second, the kitchen can't run 10 separate setups.

The Analogy: Instead of running to the stove 10 times, the kitchen staff waits a split second, grabs all 10 instructions, and does them all in one big, efficient trip. This saves time on "setting up" the stove for every single tiny task.

3. The "Emergency Brake" (Early Error Interruption)

This is the most exciting part. In the old way, if the chef wrote a recipe that said "Add 100 cups of salt," the kitchen wouldn't know until the very end of the cooking process. The customer would wait 10 minutes, only to get a ruined dish.

The Analogy: With Eager, as soon as the kitchen staff tries to add the salt and realizes, "Whoa, that's too much!" they hit the Emergency Brake.
- They immediately stop the chef from writing the rest of the recipe.
- They tell the chef, "Stop! You made a mistake here."
- The chef can then immediately fix just that part, rather than writing 50 more lines of a broken recipe.
- Bonus: Because the chef didn't waste time writing the rest of the broken recipe, the customer gets the fix much faster.

Why Does This Matter?

The researchers tested this with 7 different AI models and 4 different types of coding tasks (like analyzing data or making charts).

Speed: They found that Eager can hide up to 99.9% of the waiting time. In the best cases, the total time to get a result dropped by 55%. It's like cutting your wait time for a pizza in half.
Smarter Fixes: Because the AI gets feedback immediately when it makes a mistake (instead of waiting until the end), it fixes its errors much better. It's like a student getting a red pen mark on a math problem while they are still solving it, rather than waiting until the end of the exam to see they got it wrong.

The Bottom Line

This paper is about stopping the "stop-and-go" traffic of AI coding. Instead of the AI writing a whole novel before anyone reads a single chapter, it's now writing and reading chapter-by-chapter simultaneously. It makes AI coding feel instant, responsive, and much less frustrating for everyone involved.

1. Problem Statement

Current Large Language Model (LLM) coding agents operate under a serial execution paradigm: the model generates the entire code block, and only then invokes an interpreter to execute it.

Inefficiency: This creates a sequential workflow where the executor sits idle during generation, and the generator sits idle during execution.
Latency: The total end-to-end latency is the sum of generation time ( $T_{gen}$ ) and execution time ( $T_{exec}$ ), resulting in unnecessary user wait times.
Missed Opportunity: Unlike human developers who revise code, modern token-based LLMs commit tokens sequentially without revision. This property suggests that code could be executed incrementally as it is generated, provided the language supports it (e.g., interpreted languages like Python).

2. Methodology: The "Parallel Execution" Paradigm

The authors propose a Parallel Execution paradigm where code generation and execution overlap in time. This is implemented through a framework called Eager (Executing As you GEneRate).

A. Theoretical Framework

The system is modeled as a three-stage pipeline: Generation $\to$ Detection $\to$ Execution.

Latency Analysis: The authors derive closed-form latency bounds.
- Serial Latency: $T_{serial} \approx T_{gen} + T_{exec}$ .
- Parallel Latency: $T_{parallel} \approx T_{gen} + T_{tail}$ , where $T_{tail}$ is the execution time of only the final chunk.
- Regimes: The speedup depends on the ratio of generation speed to execution speed. In the "generation-dominated" regime (common in LLMs), execution time can be almost entirely hidden behind generation.

B. System Implementation: Eager

Eager acts as a producer-consumer pipeline:

Producer (AST-Based Chunker):
- Consumes the LLM's token stream.
- Uses AST (Abstract Syntax Tree) parsing to identify syntactically complete and semantically executable units (chunks).
- Employs lookahead disambiguation: If a statement is syntactically valid but could be part of a larger structure (e.g., inside a function definition), the chunker waits for one more token to confirm the boundary before dispatching.
Consumer (Gated Executor with Dynamic Batching):
- Maintains a persistent execution session (e.g., a persistent Python REPL) to preserve variable bindings and imports across chunks.
- Dynamic Batching: Merges multiple pending chunks into a single batch to amortize per-invocation setup overhead ( $T_{setup}$ ).
- Gating: Defers execution of "fast" chunks (e.g., function/class declarations) that have negligible execution time but incur setup costs, merging them with subsequent logic chunks.
Early Error Interruption:
- If a runtime error occurs in a chunk, Eager immediately terminates the LLM's generation process.
- It returns the error and the code generated up to the failure point, preventing the model from wasting time generating code that depends on a failed state.

3. Key Contributions

Formalization of Parallel Execution: The first theoretical framework characterizing the latency bounds, speedup potential, and operating regimes of overlapping code generation and execution.
Eager Framework: A concrete implementation featuring AST-based chunking, dynamic batching, and early error interruption mechanisms.
Comprehensive Empirical Evaluation: Extensive testing across 4 benchmarks, 7 LLMs (including open-weight and proprietary models), and 3 execution environments (Local, Docker, Open Interpreter).

4. Experimental Results

The evaluation used four benchmarks (DSBench, DABench, PandasPlotBench, GitChameleon) and simulated token streams as well as real LLM outputs.

Latency Reduction (RQ1 & RQ2):
- Non-Overlapped Execution Latency (NEL): Eager reduced NEL by 83–100% across all configurations, effectively hiding execution time behind generation.
- End-to-End Latency (E2EL):
  - For error-free executions: Reduced latency by up to 37.3%.
  - For error-encountered executions: Reduced latency by up to 55.3% due to early interruption.
- Generalization: Results held true across different generation speeds (20–200 tokens/sec) and environments (Local, Docker, Open Interpreter).
Code Repair Improvement (RQ3):
- Contrary to intuition, providing the LLM with partial code (stopped at the error) rather than the full code (which includes the error and subsequent flawed logic) improved repair success rates.
- On data-centric benchmarks, repairing from partial code improved error resolution rates by 2.1 to 44.3 percentage points.
- Reasoning: Full code "anchors" the model to the flawed logic generated after the error. Partial code forces the model to regenerate a corrected solution from the failure point.
- Exception: GitChameleon (version-specific tasks) saw slight degradation because the truncated suffix often contained necessary API context for version diagnosis.

5. Significance and Implications

User Experience: Significantly reduces the perceived wait time for AI coding agents, making them feel more responsive.
System Design: Challenges the traditional "write-then-execute" assumption in programming language design. The authors suggest future languages could be designed with "streamability" as a first-class goal (e.g., explicit delimiters) to better support incremental execution.
Agent Frameworks: Encourages a shift from file-based execution (write full file $\to$ run) to stream-based interfaces where executors consume code incrementally.
Error Handling: Demonstrates that early error feedback not only saves time but actively improves the quality of self-correction in LLMs by preventing the model from hallucinating further context based on a failed state.

In summary, Eager proves that parallel execution is not only theoretically viable but practically superior, offering substantial latency reductions and improved code repair capabilities for LLM-based coding agents.