🤖 AI

Real-Time Execution with Autoregressive Policies

This paper demonstrates that autoregressive policies can achieve real-time execution with strict latency bounds by adjusting tokenization horizons and applying constrained decoding, thereby outperforming flow-matching counterparts in both task completion speed and generalization.

Original authors: Sangkyu Lee, Seohyeon Park, Tackgeun You, Avi Caciularu, Idan Szpektor, Hwasup Lim, Youngjae Yu

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sangkyu Lee, Seohyeon Park, Tackgeun You, Avi Caciularu, Idan Szpektor, Hwasup Lim, Youngjae Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to do a task, like stacking cups or picking up a toy. To do this, the robot uses a "brain" (a large AI model) that looks at the world, thinks about what to do, and then sends commands to its arms.

The problem with these big AI brains is that they are slow thinkers. They need time to process information before they can speak. In the old way of doing things (Synchronous Inference), the robot had to stop moving completely, wait for the brain to finish thinking, and then move again. It's like a driver who has to stop the car at every single red light, wait for the light to turn green, and then start driving again. This makes the robot clumsy, slow, and unresponsive to sudden changes.

To fix this, researchers usually tried to use a different type of "brain" (called Diffusion Policies) that could think faster. But the authors of this paper asked: What if we could make the original, slower "Autoregressive" brain work just as well in real-time?

Here is how they did it, using a simple analogy:

The "Chef and the Waiter" Analogy

Imagine the robot's brain is a Chef (the AI model) and the robot's arm is a Waiter.

The Old Problem (Synchronous):
The Waiter asks the Chef, "What should I do next?" The Chef stops cooking, thinks for a long time, writes a full 10-step recipe on a piece of paper, and hands it to the Waiter. The Waiter reads the whole thing and then starts doing the steps. While the Waiter is busy doing the first 5 steps, the Chef is still sitting there doing nothing, waiting for the Waiter to finish before thinking of the next recipe. This causes the robot to "pause" and lag behind.

The New Solution (Real-Time Execution with Autoregressive Policies):
The authors realized they could change how the Chef and Waiter talk to each other without changing the Chef's brain.

Smaller Batches (The Tokenization Horizon):
Instead of asking the Chef to write a whole 10-step recipe at once, they ask for just 2 steps at a time. This is much faster to write.
- Analogy: The Chef writes a tiny note: "Step 1: Pick up cup. Step 2: Move to plate." The Waiter grabs this note immediately and starts moving.
The Action Queue (The Conveyor Belt):
The Waiter keeps a small tray (a queue) of these tiny notes. As soon as the Waiter finishes the first note, they grab the next one from the tray.
- Analogy: The Waiter never stops moving because there is always a note ready on the tray. The Chef is busy writing the next note while the Waiter is already doing the current one. This is Asynchronous Inference (thinking while moving).
The Safety Guard (Constrained Decoding):
Since the Chef is writing faster, there's a risk they might write a note that is too long or makes no sense (like a recipe that says "jump to the moon"). The authors added a "Safety Guard" that checks the Chef's work.
- Analogy: The Safety Guard ensures the Chef only writes notes that are short enough to be read before the Waiter runs out of time. If the Chef tries to write a too-long sentence, the Guard cuts it off or forces a simpler version. This guarantees the Waiter never has to stop and wait.
The "Best Guess" Strategy (Multi-Trajectory Decoding):
While the Waiter is busy, the Chef has a little bit of extra time. Instead of just writing one note, the Chef quickly drafts four different versions of the next step and picks the best one.
- Analogy: The Chef thinks, "Option A: Move left. Option B: Move right. Option C: Lift high. Option D: Lift low." The Safety Guard checks all four quickly, and the Waiter picks the best one to execute. This makes the robot smarter and more accurate without slowing it down.

What Did They Find?

The researchers tested this new method on robots in both computer simulations and the real world.

Faster than the "Slow" Brain: Even though the original "Autoregressive" brain was known for being slow, this new method made it run so smoothly that it actually outperformed the newer, "faster" Diffusion brains in many tasks.
Better at Following Instructions: Because the Autoregressive brain is naturally good at understanding complex language (like a human reading a recipe), it followed instructions better than the other types of robots, even when moving quickly.
No More Pausing: The robot never stopped moving. It reacted to changes instantly because the "Chef" was always one step ahead, writing the next note while the "Waiter" was still moving.

The Big Takeaway

The paper proves that you don't need to switch to a completely different type of AI brain to get a robot to move in real-time. You just need to change how you ask the brain for instructions. By breaking instructions into tiny chunks, checking them for speed, and letting the brain think while the robot moves, you can make even the "slow" thinkers become fast, smooth, and highly reactive.

In short: Don't change the engine; just change the transmission.

Technical Summary: Real-Time Execution with Autoregressive Policies

1. Problem Statement

The deployment of large-scale Vision-Language-Action (VLA) models in robotics faces a critical bottleneck: inference latency. Large parameter sizes inherent to foundation models cause the "think-then-move" (synchronous inference) paradigm to induce significant robot idle time. During this pause, the robot cannot react to environmental changes, sacrificing both reactivity and task completion speed.

While recent work has addressed real-time execution (asynchronous inference or "think-while-moving"), it has primarily focused on diffusion-based policies (parallel decoding). Autoregressive policies (sequential decoding), which offer advantages in convergence speed and instruction-following generalizability, have been largely excluded from real-time execution research. This is because their sequential nature typically results in higher latency ( $\delta_{t,H}$ ), making them unsuitable for the strict timing requirements of asynchronous control where the robot must continuously receive actions without pausing.

The core challenge is to enable autoregressive policies to operate in real-time without sacrificing their inherent benefits, ensuring that the robot receives a continuous stream of actions despite the variable and often high latency of token-by-token generation.

2. Methodology

The authors propose $\pi_0$ -REALFAST, a framework enabling real-time execution for autoregressive policies through four key technical adjustments:

A. Decoupling Action Horizons and Tokenization

The authors redefine the action chunking strategy. Instead of tokenizing the entire action horizon $H$ at once, they set $H = 2m$ and tokenize in two stages:

Conditioning Prefix: The first $m$ actions (already executed or in the queue) are tokenized as a prefix ( $T_m(A_{0:m})$ ).
Prediction: The model predicts the next $m$ actions ( $T_m(A_{m:2m})$ ) conditioned on the prefix and the current observation.
This reduces the number of tokens to be decoded in a single inference step, directly lowering the latency $\delta_{t,m}$ required to generate the next actionable chunk.

B. Constrained Decoding for Latency Bounds

To guarantee that the robot never pauses, the system must ensure that the time to decode the next chunk ( $\delta_{t,m}$ ) does not exceed the time the controller consumes the current chunk ( $d_m \cdot \Delta t$ ).

Problem: Token lengths in autoregressive models vary, making latency unpredictable.
Solution: The authors implement constrained decoding. They model the decoding process as a bounded "coin change problem" using dynamic programming (DP).
- A DP table is precomputed to determine if a specific token sequence length $L$ can be validly detokenized within a maximum step count $B$ .
- During inference, logits for tokens that would violate the latency bound (i.e., lead to a sequence that cannot be detokenized within $B$ steps) are masked out.
- This guarantees that the decoding process completes within a strict latency bound $\delta'$ , ensuring $d_m \le m$ .

C. Multi-Trajectory Decoding

To maximize performance during the "idle" time created by the latency gap between inference and action consumption, the authors employ multi-trajectory decoding.

The model generates $N$ candidate trajectories (e.g., $N=4$ ) in parallel.
Since the prefix (KV cache) is shared, the computational overhead is marginal.
The best trajectory is selected based on likelihood, improving task success rates without increasing the worst-case latency bound.

D. Asynchronous Inference Loop

The system operates asynchronously:

The client (robot controller) maintains an Action Queue (AQ).
The server (policy) decodes the next $m$ tokens conditioned on the AQ prefix.
Once decoding is complete (within the bound), the new chunk is appended to the AQ.
The robot executes actions from the AQ continuously, ensuring no pauses occur even if the policy is still computing the next chunk.

3. Key Contributions

Feasibility of Autoregressive Real-Time Execution: The paper demonstrates that autoregressive policies can achieve real-time execution, challenging the assumption that they are too slow for asynchronous control.
Latency-Guaranteed Decoding: The introduction of constrained decoding via dynamic programming ensures strict latency bounds, a prerequisite for preventing robot pausing in asynchronous settings.
Performance Optimization: The combination of reduced token horizons ( $m$ ) and multi-trajectory decoding allows autoregressive policies to outperform equivalent-level flow-matching (diffusion) policies in real-time scenarios.
Robustness to External Latency: The method maintains high success rates even when external latencies (network/sensor) vary, provided the inference frequency ( $d_m$ ) is managed correctly.

4. Experimental Results

The authors evaluated $\pi_0$ -REALFAST (fine-tuned from $\pi_0$ -FAST) against baselines including $\pi_0$ (flow-matching) and $\pi_0.5$ (flow-matching with FAST tokenizer) in both simulated and real-world environments.

Simulated Environment (LIBERO):
- $\pi_0$ -REALFAST consistently outperformed $\pi_0$ with Real-Time Action Chunking (RTC) across all task categories (Spatial, Object, Goal, Long).
- It achieved performance comparable to the more advanced $\pi_0.5$ , despite $\pi_0.5$ being trained on larger datasets.
- The study highlighted that enforcing $d_m \le m$ (allowing the inference frequency to be lower than the action consumption rate) is more robust than strict $d_m = m$ synchronization, which caused performance degradation in $\pi_0$ .
Real-World Environment (DROID):
- In zero-shot tabletop manipulation tasks, $\pi_0$ -REALFAST significantly surpassed the synchronous $\pi_0$ -FAST baseline.
- It outperformed $\pi_0$ and showed performance comparable to $\pi_0.5$ .
- Qualitative Analysis: Unlike synchronous inference, which often led to collisions due to overshooting (inability to adjust mid-trajectory), $\pi_0$ -REALFAST successfully adjusted trajectories in real-time to avoid collisions, demonstrating the critical value of frequent action updates.
Ablation Studies:
- Constrained Decoding: Removing constrained decoding led to performance degradation, confirming its necessity for reliability.
- Training Steps: The model reached peak performance quickly (within 6k steps), consistent with the fast convergence properties of autoregressive policies.
- Multi-Trajectory: Increasing $N$ (number of trajectories) improved success rates, validating the utility of utilizing idle compute time.

5. Significance and Claims

The paper claims that autoregressive policies are a competitive and viable choice for real-time robotic control, provided that inference latency is managed through specific architectural and decoding strategies.

Trade-off Re-evaluation: The authors argue that the field has over-prioritized reducing inference latency (to secure reactivity) at the expense of policy robustness and generalizability. Their results suggest that a slightly higher but bounded latency, when paired with autoregressive generalization capabilities, yields better deployment outcomes than lower-latency policies that lack these capabilities.
Native Support: The findings motivate the pretraining of autoregressive policies to natively support real-time execution, rather than treating them as secondary to diffusion-based approaches.
Limitations: The authors modestly note that their experiments are currently limited to single-arm, relatively static tabletop environments. Extending to bimanual or highly dynamic tasks would require decoding more tokens, imposing stricter latency constraints that may necessitate further architectural innovations (e.g., speculative decoding).

In conclusion, the work establishes that with asynchronous inference, tokenization horizon adjustment, and constrained decoding, autoregressive policies can match or exceed the performance of diffusion-based counterparts in real-time robotic deployment, leveraging their superior instruction-following and convergence properties.