🤖 AI

Learning Native Continuation for Action Chunking Flow Policies

The paper introduces Legato, a training-time continuation method for action-chunked Vision Language Action models that reshapes flow dynamics and utilizes randomized schedule conditioning to produce smoother, more consistent trajectories, thereby outperforming existing Real-Time Chunking approaches in real-world manipulation tasks.

Original authors: Yufeng Liu, Hang Yu, Juntu Zhao, Bocheng Li, Di Zhang, Mingzhu Li, Wenxuan Wu, Yingdong Hu, Junyuan Xie, Junliang Guo, Dequan Wang, Yang Gao

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yufeng Liu, Hang Yu, Juntu Zhao, Bocheng Li, Di Zhang, Mingzhu Li, Wenxuan Wu, Yingdong Hu, Junyuan Xie, Junliang Guo, Dequan Wang, Yang Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to perform a complex task, like pouring water from one cup to another. To do this, the robot uses a "brain" (a Vision-Language-Action model) that looks at the scene and decides what to do next.

However, this brain is slow. It can't think fast enough to give the robot a new instruction every single millisecond. So, engineers use a trick called "Action Chunking." Instead of giving one instruction at a time, the brain predicts a whole "chunk" of future movements (say, the next 2 seconds of motion) all at once.

The Problem: The "Stutter"

The paper identifies a major flaw in how these chunks are currently stitched together.

Imagine you are walking down a hallway. You take a step, then another. If you plan your next three steps all at once, you might walk smoothly. But when you finish those three steps and need to plan the next three, your brain has to pause, re-evaluate, and start a new plan.

In current robots, this pause causes a "stutter" or a "hiccup" at the boundary where one chunk ends and the next begins. The robot might:

Hesitate (freeze for a split second).
Suddenly jerk in a different direction.
Change its mind about which hand to use or which object to grab.

The paper calls this "spurious multimodal switching." In plain English: The robot gets confused at the transition points and starts wavering between different possible actions, making the movement look jerky and unnatural.

The Old Fix: "Real-Time Chunking" (RTC)

Previous attempts to fix this, called Real-Time Chunking (RTC), were like a post-it note reminder.

When the robot finishes a chunk, the system looks at the last few moves it made.
It forces the new chunk to start exactly where the old one left off.
The Flaw: This is an external rule applied after the robot has already thought. It's like telling a painter, "Make sure your next brushstroke connects to the last one," but the painter didn't learn how to connect them naturally. The robot still gets confused internally, leading to hesitation and jerky movements.

The New Solution: "Legato"

The authors propose a new method called Legato (named after a musical term meaning "smoothly connected").

Instead of just adding a rule at the end, Legato teaches the robot's brain how to connect the dots while it is learning.

Here is how it works, using a creative analogy:

1. The "Training with Training Wheels" Analogy
Imagine teaching a child to ride a bike.

Old Way (RTC): You let them ride, and if they wobble at the end of a turn, you grab the handlebars and force them straight. They learn to rely on you to fix the wobble.
Legato Way: You teach them to feel the balance during the turn. You give them a "schedule" of how much support they need at every second of the turn. As they get further into the turn, you slowly let go, but you guide them smoothly the whole time.

2. The "Smooth Ramp" vs. The "Hard Wall"
Legato uses a "schedule-shaped" guidance.

When the robot starts a new chunk, it knows exactly what the previous chunk ended with.
Legato tells the robot: "For the first part of this new chunk, you must follow the previous path exactly."
Then, it slowly says: "Okay, you can start to drift a little bit and make your own choices."
Finally: "Now you are free to plan the rest of the movement."

This creates a smooth ramp of freedom rather than a hard wall where the robot has to suddenly switch gears.

3. The "Native Skill"
The most important part of Legato is that it changes the robot's internal "flow." It reshapes the math inside the robot's brain so that connecting chunks is a natural part of how it thinks.

It doesn't just force the robot to match the past; it teaches the robot that staying consistent with the past is the easiest and most logical path to take.
This stops the robot from hesitating or flipping back and forth between different ideas (multimodal switching).

The Results

The researchers tested this on real robots doing five different tasks: stacking bowls, pouring things, picking up objects, opening drawers, and folding towels.

Smoother Moves: Legato's robots moved like a fluid stream of water, whereas the old method moved like a series of jerky steps.
Faster Completion: Because the robots didn't hesitate or freeze at the "chunk boundaries," they finished tasks about 10% faster.
Less Confusion: The robots stuck to their plan (e.g., "I will use my left hand") instead of wavering back and forth.

Summary

Think of Action Chunking as a robot taking a series of photos to plan its movement.

The Problem: The photos don't line up perfectly, so the movie looks jittery.
The Old Fix: You try to edit the photos together after taking them (RTC), but the jitter remains.
Legato: You teach the camera how to take the photos so they naturally line up perfectly while you are shooting. The result is a smooth, continuous movie where the robot moves with confidence and grace.

Technical Summary: Legato – Learning Native Continuation for Action Chunking Flow Policies

1. Problem Statement

Vision Language Action (VLA) models are increasingly deployed in real-world robotics using action chunking, a strategy where the model predicts a sequence of future action vectors in a single inference step. While chunking amortizes inference costs and enables high-frequency control, it introduces a fundamental drawback: discontinuities at chunk boundaries.

These discontinuities arise from two primary factors:

Inference Delay: The time required to compute the next chunk causes a gap between the execution of the current chunk and the generation of the next.
Intrinsic Multimodality: Flow-based policies often exhibit multimodal distributions, leading to "spurious multimodal switching" where the policy abruptly changes its intended action mode between chunks.

Existing solutions like Real-Time Chunking (RTC) attempt to mitigate this by applying inference-time inpainting, partially constraining new chunks to overlap with previous actions. However, RTC is an external mechanism applied only at inference time. Because the policy itself is not trained to handle this continuation, it remains prone to hesitation, abrupt direction changes, and trajectories that are not intrinsically smooth. The paper argues that stable chunked execution requires chunk continuation to be a native, learned property of the policy.

2. Methodology: Legato

The authors propose Legato, a training-time continuation framework for action-chunked flow-based VLA policies. Legato internalizes the continuation mechanism by reshaping the learned flow dynamics to ensure consistency between training and inference under per-step guidance.

Core Components

Legato addresses two critical requirements for effective continuation:

Per-Step Guidance: Guidance must be applied repeatedly across all denoising steps, not just at initialization.
Training-Inference Consistency: The dynamics learned during training must match the dynamics used during inference.

A. Schedule-Shaped Guidance

Legato introduces a guidance schedule $\omega \in [0, 1]^H$ over the action chunk horizon $H$ . This schedule is defined by three parameters:

$d$ : The inference delay (full guidance prefix).
$s$ : The executed length per cycle.
$r$ : The ramp-down length where guidance strength decreases.
These satisfy $r + s + d = H$ .

Instead of hard-clamping the prefix (as in Training-time RTC), Legato uses a smooth schedule: it anchors the beginning of the chunk to known actions (ground truth during training, previous chunk prefix during inference) and gradually ramps the guidance strength down to zero. This is achieved via an action-noise mixture:
$\epsilon_{eff} = (1 - \omega) \odot \epsilon + \omega \odot A$
where $\epsilon$ is noise and $A$ is the reference action.

B. Reshaping Flow Dynamics

Standard flow matching trains a velocity field $u_{FM}$ to transport noise to actions. However, applying per-step guidance at inference time induces a different dynamic system. To eliminate the train-test mismatch, Legato derives the exact continuous-time dynamics induced by repeated per-step guidance and reshapes the target velocity field during training.

The network is trained to predict a modified target velocity $v_{target}$ :
$v_{target}(t, A, \epsilon, \omega) = (1 - \kappa \odot (1 - t)) \odot (A - \epsilon)$
where $\kappa = \omega / \Delta t$ . This allows the policy to natively support the schedule-shaped guidance without external intervention at inference time.

C. Randomized Schedule Conditioning

To handle varying inference latencies across hardware and runtime optimizations, Legato randomizes the schedule parameters ( $d, r$ ) during training. The policy is explicitly conditioned on the schedule $\omega$ (appended as a feature). This enables a single model to adapt to different inference delays and control the trade-off between trajectory smoothness and overlap consistency at inference time without retraining.

3. Key Contributions

Legato Framework: A training-time continuation framework that enables per-step, schedule-shaped guidance while maintaining strict training-inference consistency by reshaping flow dynamics.
Randomized Schedule Conditioning: A mechanism to support varying inference delays and provide flexible control over trajectory smoothness, allowing the model to generalize across different hardware and latency conditions.
Empirical Validation: Extensive real-world experiments demonstrating that learning continuation as a native policy property outperforms inference-time correction methods.

4. Experimental Results

The authors evaluated Legato on five real-world manipulation tasks (stacking bowls, pouring, pick-and-place, drawer opening, towel folding) using a dual-arm robot.

Performance Metrics

Task Completion Time: Legato consistently achieved shorter completion times (approx. 10% improvement) compared to RTC.
Trajectory Smoothness: Legato produced significantly smoother trajectories, evidenced by lower NSPARC (Negative Spectral Arc Length) and Overlap RMSE values.
Spurious Multimodal Switching: Legato reduced hesitation and abrupt direction changes, leading to more consistent grasp goals and arm selections.

Comparative Analysis

vs. RTC: Legato outperformed RTC across all five tasks in terms of speed and smoothness.
vs. Training-Time RTC: Legato outperformed a baseline that applies hard prefix constraints during training. The authors attribute this to Legato's reshaping of flow dynamics, which better aligns with the effective denoising behavior of repeated guidance.
Generalization: The method demonstrated robustness across different VLA backbones ( $\pi_0$ , $\pi_0.5$ , Gr00t N1.6) and varying inference delays.

5. Significance and Claims

The paper claims that Legato represents a shift from treating action chunking continuity as an external, inference-time correction to a native, learned property of the policy.

Intrinsic Smoothness: By reshaping the flow dynamics, Legato ensures that the policy inherently understands how to transition between chunks, reducing the "jitter" and hesitation associated with multimodal switching.
Efficiency: The reduction in hesitation and direction changes directly translates to faster task completion times in real-world scenarios.
Adaptability: The use of randomized schedule conditioning allows a single model to be deployed across diverse hardware environments with different inference latencies, a critical requirement for practical robotic deployment.

The authors conclude that while current formulations limit denoising steps to training-time specification, the proposed method successfully demonstrates that learning the continuation dynamics is superior to enforcing hard constraints, offering a robust path toward smoother and more efficient real-world robotic control.