Technical Summary: Premover – Fast Vision-Language-Action Control by Acting Before Instructions Are Complete

Problem Statement

Current Vision-Language-Action (VLA) policies are typically evaluated under the assumption that the user's full instruction is available before the robot begins computation. However, in real-world deployment, users require several seconds to type or speak a request. During this "streaming prefix" interval, the VLA policy remains idle, creating a substantial latency gap. On benchmarks like LIBERO, this input interval accounts for approximately 39% of the total interaction time (averaging ~17.7 seconds of idle time per episode).

Existing efficiency research focuses on reducing inference latency after the instruction is received (e.g., via early-exit gates or token pruning). These methods do not address the idle time during user input. The core challenge is to utilize this interval for computation without compromising safety or success: acting too early on an incomplete instruction risks committing the robot to the wrong target, while waiting for the full instruction forfeits the latency benefit.

Methodology: Premover

Premover is a lightweight module designed to operate on top of a frozen VLA backbone (specifically $\pi0.5$ in the experiments). It converts the idle input interval into useful precomputation by introducing two coupled components: a Focus Map and an Action Readiness Gate.

1. Vision-Language Focus Map

The first challenge is determining where to attend given a partial instruction (e.g., "pick up the ketchup..."). A frozen backbone, trained via action imitation, often disperses attention across background regions rather than localizing the referent of a partial prompt.

• Mechanism: Premover attaches two small, trainable projection heads (one for image patches, one for language tokens) to an intermediate layer of the frozen backbone. These heads map hidden states into a shared latent space.
• Computation: It computes the cosine similarity between image patch embeddings and the current streaming language tokens. The maximum similarity over the token sequence generates a per-patch probability score, forming a Focus Map.
• Supervision: This map is supervised using simulator-rendered target-object segmentation masks via a class-balanced Binary Cross-Entropy (BCE) loss.
• Injection: To influence the frozen backbone without retraining it, the Focus Map is used to reweight image tokens for the next inference step ($t+1$). A floor scale parameter $\alpha$ interpolates between the focus map and a uniform weight, preserving scene context (e.g., obstacle avoidance) while amplifying instruction-relevant regions.

2. Action Readiness Gate

The second challenge is determining when the partial instruction is informative enough to act.

• Mechanism: The system calculates an Action Readiness Score ($r_t$) based on the concentration of the Focus Map. It measures the difference between the mean probability of the top-$K$ patches and the global mean probability. A high score indicates the prefix has localized a specific target; a low score indicates dispersed attention.
• Decision Logic: A learnable scalar threshold $\tau$ acts as a gate. The policy executes actions only when $r_t \geq \tau$.
• Supervision: The threshold is trained using a binary label indicating whether the target object has appeared in the current prefix. A temperature-scaled BCE loss is applied to make the decision differentiable.

Training

Both components are jointly trained in a single forward pass. The backbone parameters remain frozen; gradients flow only through the two projection heads and the readiness threshold. This constitutes less than 1% of the backbone's total parameters, allowing for efficient adaptation without large-scale retraining.

Key Contributions

1. Reframing Idle Time: The paper argues that the user input interval is a usable resource for computation rather than dead time, proposing a regime where policies ground visual attention to streaming prefixes.
2. Lightweight Architecture: Premover achieves this by adding minimal trainable parameters (two projection heads and a scalar) to a frozen VLA, preserving pretrained weights exactly.
3. Dual-Component Design: It explicitly separates the problem of where to focus (Focus Map) from when to act (Readiness Gate), addressing the trade-off between early execution and safety.
4. Streaming Evaluation Protocol: The authors introduce a streaming evaluation protocol that simulates real-time token arrival synchronized with a standard typing rate (52.24 WPM), providing a more realistic latency metric than standard full-prompt baselines.

Experimental Results

The method was evaluated on the LIBERO suite (Spatial, Object, Goal, LIBERO-10) and VLA-arena (Level-1 suites) using the $\pi0.5$ backbone.

• Success Rate: Premover matches the full-prompt baseline's success rate on LIBERO (95.1% vs. 95.0%) and significantly outperforms "Naive Premoving" (which acts immediately on every token and collapses to 66.4%). On VLA-arena, it preserves 30.9% success compared to the full-prompt 33.0%, whereas Naive Premoving drops to 27.0%.
• Latency Reduction:
  • On LIBERO, Premover reduces the mean end-to-end wall-clock time from 34.0s to 29.4s (a 13.6% reduction).
  • On VLA-arena, it reduces wall-clock time by approximately 10.3%.
• Ablation Studies:
  • The Readiness Gate alone recovers 21.9 percentage points of success rate over naive streaming, demonstrating its critical role in preventing premature action.
  • The Focus Map adds an additional 6.7 percentage points, improving distractor disambiguation.
  • The floor scale $\alpha$ is robust across a range of values, with $\alpha=0.2$ selected as optimal to balance target amplification and scene context retention.

Significance and Claims

The paper claims that Premover represents an important step toward efficient and responsive real-world VLA applications. Its primary significance lies in demonstrating that:

1. User input time is a resource: Latency can be reduced by performing computation during the typing interval rather than waiting for completion.
2. Lightweight modulation is sufficient: Complex retraining is not required to enable early grounding; a small module can effectively modulate a frozen policy to handle streaming instructions.
3. Safety through gating: A learned readiness criterion allows the system to act early without the catastrophic failure modes associated with naive early execution.

The authors acknowledge limitations, noting that Focus Map supervision currently relies on simulator-rendered masks (not directly available on real robots) and that the projection heads are trained per benchmark rather than transferred zero-shot. Future work is suggested to address real-robot grounding via open-vocabulary detectors and to handle non-monotone prefix dynamics (e.g., user revisions).