Technical Summary: Real-Time Robot Execution with Masked Action Chunking (REMAC)

Problem Statement

Real-time execution is critical for cyber-physical systems like robots operating in dynamic environments, where even minor delays can lead to task failure. While Vision-Language-Action (VLA) models have advanced robotic manipulation, ensuring real-time responsiveness remains challenging.

Existing approaches face a trade-off between speed and accuracy:

• Model-level acceleration often sacrifices accuracy for speed.
• Action Chunking (predicting a sequence of actions per step) improves efficiency but reduces reactivity to sudden state changes and introduces discontinuities at chunk boundaries.
• Asynchronous Inference (predicting the next chunk while executing the current one) ensures continuous action availability but exacerbates two specific failure modes:
  1. Inter-chunk Discontinuity: Incoherent transitions between consecutive action chunks due to diverging latent modes.
  2. Intra-chunk Inconsistency: A critical, often overlooked issue where the actions executed within a chunk partially misalign with the current perception. Specifically, the first $d$ actions of a chunk (where $d$ is the inference delay) are inherited from the previous chunk's prediction (conditioned on an older observation $o_{t-h}$), creating a mismatch with the current state $o_t$.

Previous methods primarily addressed inter-chunk discontinuity using test-time refinements (e.g., smoothing, inpainting, or gradient corrections). However, these often rely on heuristics prone to failure or introduce additional latency, and they largely ignore intra-chunk inconsistency.

Methodology: REMAC

The authors propose Real-time Execution with Masked Action Chunking (REMAC), a training-time adaptation strategy that learns corrective adjustments on a pretrained flow-matching policy without incurring additional inference latency. The method targets both intra-chunk consistency and inter-chunk continuity.

1. Masked Action Chunking

To address intra-chunk inconsistency, REMAC modifies the training objective to account for the delay between observation and execution.

• Prefix Masking: The training loss is restricted to the "executable" portion of the action chunk. A delay-conditioned prefix mask $m_d$ excludes the first $d$ actions (which are inherited from the previous chunk and thus mismatched with the current observation) from the supervision signal. This forces the policy to focus on learning robust predictions for the remaining actions despite the uncertainty of the inherited prefix.
• Self-Conditioned Curriculum: To simulate test-time conditions during training (where executed actions are unavailable), the authors introduce a curriculum that gradually shifts the training input from ground-truth actions to the policy's own predictions. This is achieved by mixing the ground-truth chunk $A_t$ with the pretrained policy's prediction $\tilde{A}_t$ using a scheduled mixing parameter $\sigma$. This mitigates exposure bias and stabilizes training.
• Residual Alignment ($\Delta$-matching): An additional loss term encourages the model to learn the residual correction relative to the pretrained policy, explicitly aligning the induced adjustment with the difference between the pretrained prediction and the ground truth.

2. Prefix-Preserved Sampling

At inference time, the sampling pipeline is adjusted to align with the training procedure:

• Initialization: The initial latent action state is not drawn from a Gaussian prior but is initialized using the actions that will be executed during the delay period (the "prefix" from the previous chunk).
• Preserved Execution: During the integration steps of the flow-matching sampler, the overlapping segment between the current and upcoming chunks (the actions already committed for execution) is preserved and not regenerated. Only the remaining portion of the chunk is synthesized.

3. Implementation

• Parameter Efficiency: The method uses Low-Rank Adaptation (LoRA) to fine-tune the pretrained policy, adding at most 1.5% parameters. This allows the model to learn delay-aware adjustments while preserving the original model's predictive capacity.
• Zero Latency Overhead: The LoRA weights are merged into the backbone, and the sampling procedure does not require extra computation, ensuring no additional inference delay compared to the base policy.

Key Contributions

1. Identification of Intra-chunk Inconsistency: The paper highlights that the misalignment between inherited actions and current perception is a critical failure mode in asynchronous inference, distinct from inter-chunk discontinuity.
2. REMAC Framework: A novel training strategy combining masked action chunking, self-conditioned curriculum scheduling, and residual alignment to learn delay-aware policies.
3. Latency-Free Adaptation: Unlike test-time correction methods (e.g., RTC, BID) that add computational overhead, REMAC introduces no additional inference latency.
4. Compatibility: The method is designed to be a plug-and-play enhancement that can be combined with existing test-time algorithms.

Experimental Results

The authors evaluated REMAC across 12 simulated tasks (Kinetix) and three real-world grasping tasks using a Franka Research 3 robot.

• Simulation (Kinetix):

  • REMAC consistently outperformed baselines (Naive Async, Bidirectional Decoding, RTC) across all inference delays ($d=0$ to $d=4$).
  • It demonstrated higher success rates and faster task completion times.
  • Performance degradation was significantly lower as delay increased compared to baselines.
  • Ablation studies confirmed that each component (Prefix Masking, Curriculum, $\Delta$-matching) contributed to the final performance, with the full method achieving the highest success rates.

• Real-World Deployment:

  • In tasks involving grasping objects of varying difficulty (e.g., a Rubik's cube with tight clearance), REMAC achieved higher completion progress than Synchronous inference, Naive Async, Temporal Ensembling, and RTC.
  • Under injected delays of up to 150ms (simulating network/hardware latency), REMAC maintained robustness, whereas RTC's performance degraded significantly due to its added latency.
  • Kinematic analysis showed REMAC produced smoother trajectories with fewer abrupt changes in velocity and acceleration compared to synchronous methods.

Significance

The paper claims that REMAC provides a robust solution for real-time robot execution under asynchronous inference by addressing the root cause of intra-chunk inconsistency through training-time adaptation rather than test-time heuristics. By enabling policies to remain resilient to mismatches between intended actions and actual execution without incurring extra latency, REMAC facilitates faster and more reliable task completion in dynamic environments. The authors position the method as a versatile backbone improvement that can be seamlessly integrated with existing VLA frameworks and combined with other test-time strategies for further gains.