💻 computer science

LaST $_{0}$ : Latent Spatio-Temporal Chain-of-Thought for Robotic Vision-Language-Action Model

LaST $_0$ is a robotic Vision-Language-Action framework that overcomes the latency and representational limitations of explicit linguistic reasoning by employing a token-efficient, latent Spatio-Temporal Chain-of-Thought within a dual-system Mixture-of-Transformers architecture, thereby achieving significant success rate improvements across diverse real-world manipulation tasks.

Original authors: Zhuoyang Liu, Jiaming Liu, Hao Chen, Jiale Yu, Ziyu Guo, Chengkai Hou, Chenyang Gu, Xiangju Mi, Renrui Zhang, Kun Wu, Zhengping Che, Jian Tang, Pheng-Ann Heng, Shanghang Zhang

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhuoyang Liu, Jiaming Liu, Hao Chen, Jiale Yu, Ziyu Guo, Chengkai Hou, Chenyang Gu, Xiangju Mi, Renrui Zhang, Kun Wu, Zhengping Che, Jian Tang, Pheng-Ann Heng, Shanghang Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to cook an egg. You tell it, "Scoop the egg out of the pan and put it on the bread."

The Old Way (Explicit Reasoning):
In the past, to do this, the robot would have to stop, think out loud, and write a step-by-step essay before moving its arm. It might say: "First, I need to lower my hand. Then, I need to tilt the pan. Then, I need to scoop..."
While the robot is busy writing this essay, the egg is getting cold, or worse, the pan is tipping over. This "thinking out loud" takes too much time (latency), making the robot slow and clumsy. Also, some things are hard to describe with words, like the exact feeling of the egg sliding or the 3D shape of the pan.

The New Way (LaST0):
The paper introduces LaST0, a robot brain that thinks differently. Instead of writing an essay, it uses a "Silent, Invisible Thought Process" (called Latent Spatio-Temporal Chain-of-Thought).

Here is how LaST0 works, using simple analogies:

1. The "Silent Movie" vs. The "Script"

Instead of generating words to explain what it's going to do, LaST0 generates a silent, internal movie of the future.

What it sees: It doesn't just look at the picture; it imagines the 3D shape of the egg, the texture of the pan, and how the robot's own arm feels (proprioception).
The "Latent" part: These thoughts aren't words you can read; they are compact, high-speed data packets. It's like the difference between reading a long recipe book (slow) and having a chef's muscle memory (fast and intuitive). This allows the robot to "feel" the physics of the situation without getting stuck on vocabulary.

2. The "Brain and Body" Team (Dual-System)

LaST0 splits its brain into two specialized experts working together, like a Strategic Commander and a Reflexive Athlete.

The Commander (Slow Reasoning Expert): This part works at a slower speed. It looks at the big picture, imagines the next few seconds of the "silent movie," and figures out the plan. It asks, "If I move my arm this way, where will the egg be in 0.5 seconds?" It does this only occasionally (like checking a map every few seconds).
The Athlete (Fast Acting Expert): This part is super fast. It doesn't stop to think about the big plan. It just watches the Commander's latest update and reacts instantly to the robot's immediate surroundings. It handles the high-speed movements needed to catch the egg before it falls.

The Magic Connection: They share a "telepathic link" (Shared Attention). The Commander updates the Athlete with a new plan, and the Athlete executes it immediately. This means the robot can think deeply about the future without slowing down its current movements.

3. Why It's Better

Speed: Because it doesn't waste time writing sentences, the robot is 14 times faster than previous methods that "thought out loud." It can react in real-time, just like a human.
Understanding Physics: It captures things words can't describe. It understands the "weight" and "slipperiness" of the egg through its internal 3D model, not just through text.
Consistency: Because it imagines a continuous "movie" of the future, its actions are smooth and connected. It doesn't jerk around; it flows.

The Results

The researchers tested this robot on 10 different real-world tasks, from simple tabletop tricks to complex tasks involving mobile robots and dexterous hands (like human hands).

Success Rate: LaST0 succeeded 13% to 14% more often than the best previous robots.
Long Tasks: It was especially good at long, complicated tasks (like scooping an egg, putting it on bread, and repeating the process), where other robots would get confused and fail.

In Summary:
LaST0 is a robot that stops "talking" to itself and starts "feeling" the future. By using a silent, internal simulation of 3D space and time, and splitting its brain into a slow planner and a fast doer, it moves with the speed and grace of a human, rather than the hesitation of a computer reading a manual.

Technical Summary: LaST0 – Latent Spatio-Temporal Chain-of-Thought for Robotic Vision-Language-Action Models

1. Problem Statement

Vision-Language-Action (VLA) models have recently demonstrated strong generalization by integrating the semantic understanding of Vision-Language Models (VLMs) with robotic control policies. However, existing approaches that attempt to improve reasoning capabilities face two fundamental limitations in robotic manipulation:

Inference Latency: Methods that explicitly generate linguistic reasoning traces (Chain-of-Thought) or predict future visual observations prior to acting incur significant autoregressive generation overhead. This latency constrains the temporal resolution required for real-time, closed-loop robotic control.
Representational Bottleneck: Explicit reasoning is confined to the linguistic space, which struggles to faithfully capture "ineffable" physical attributes and fine-grained robotic dynamics that are difficult to verbalize. Robotic agents require reasoning about physical and geometric states that often exceed the expressive capacity of natural language.

2. Methodology: LaST0 Framework

The authors propose LaST0, a unified VLA model that enables efficient "reason-before-act" behavior through a Latent Spatio-Temporal Chain-of-Thought (LaST CoT). Instead of generating text or images, the model performs reasoning within a compact, continuous latent space.

Core Architecture

LaST0 is built upon a Mixture-of-Transformers (MoT) design, initialized from the Janus-Pro (DeepSeek-LLM 1.5B) backbone. It decouples reasoning and control into two specialized experts that share a global self-attention context but operate at different frequencies:

Slow Reasoning Expert: Operates at a low frequency (e.g., every $k$ -th timestep). It performs autoregressive inference to generate the LaST CoT, a sequence of latent tokens representing future physical dynamics.
Fast Acting Expert: Operates at the native high-frequency control rate. It generates continuous action commands conditioned on high-frequency observations and the periodically updated latent representations from the reasoning expert.

Latent Spatio-Temporal CoT

The LaST CoT is a token-efficient latent space that autoregressively predicts future states across three modalities over a time horizon $H$ :

2D Visual Latents: Encoded from future RGB frames using a frozen SigLIP-Large encoder.
3D Geometric Latents: Encoded from future point clouds using a pretrained Uni3D encoder (used only during training to provide ground-truth targets).
Robot Proprioception Latents: Encoded from future robot states (joint angles, velocities, gripper state).

These modalities are interleaved chronologically ( $z^v_1, z^p_1, z^s_1, \dots$ ) to model coupled spatio-temporal dependencies. During training, the reasoning expert is supervised via cosine similarity loss against ground-truth latent sequences derived from future observations, rather than discrete token prediction.

Training Strategy

Pretraining: LaST0 is pretrained on a diverse corpus of over 400K trajectories from Open-X-Embodiment, DROID, and RoboMIND.
Supervised Fine-Tuning (SFT): The model undergoes joint optimization where the reasoning expert minimizes latent regression loss, and the action expert minimizes Flow Matching loss.
Heterogeneous Frequency Training: The action expert is trained with randomly mixed fast-slow operating ratios (e.g., 1:1, 1:2, 1:4). This enables the model to adaptively select appropriate execution frequencies during deployment, ensuring robustness when the reasoning update rate varies.

3. Key Contributions

Latent Spatio-Temporal CoT: A novel framework that performs reasoning in a compact latent space, capturing fine-grained physical and robotic dynamics (2D, 3D, and proprioceptive) that are difficult to verbalize, while supporting temporally consistent modeling.
Dual-System MoT Architecture: A design that coordinates low-frequency latent reasoning with high-frequency action generation via a Mixture-of-Transformers scheme. This allows for deliberative reasoning without compromising real-time control responsiveness.
Efficient Training and Inference: By replacing explicit text/image generation with latent regression and utilizing asynchronous frequency coordination, the method achieves significant speedups compared to explicit CoT approaches.

4. Experimental Results

The authors evaluated LaST0 across simulation and real-world benchmarks, comparing it against state-of-the-art VLA models (e.g., OpenVLA, $\pi_0.5$ , CoT-VLA, SpatialVLA).

Simulation Benchmarks

RLBench: On 10 diverse manipulation tasks, LaST0 achieved a mean success rate of 82%, outperforming the strongest baseline (HybridVLA-7B at 74%) by 8%. It also demonstrated superior inference speed (15.4 Hz) compared to explicit CoT methods (1.1 Hz).
LIBERO: LaST0 achieved a state-of-the-art mean success rate of 98.1%, surpassing $\pi_0.5$ (96.9%) and OpenVLA-OFT (97.1%). Notably, it excelled in the long-horizon LIBERO-Long suite (95.6%), demonstrating effective handling of extended tasks.

Real-World Experiments

Evaluated on 10 complex tasks spanning tabletop (single/dual-arm), mobile manipulation, and dexterous hand control:

Performance: LaST0 achieved a mean success rate of 72% on the Franka platform, significantly outperforming SpatialVLA (41%), $\pi_0.5$ (59%), and CoT-VLA (50%).
Long-Horizon Robustness: In a long-horizon task requiring three consecutive successful executions (placing an egg on bread repeatedly), LaST0 maintained success rates of 66% $\to$ 47% $\to$ 33% across steps, whereas $\pi_0.5$ dropped to 47% $\to$ 20% $\to$ 7%.
Generalization: The model successfully generalized to mobile platforms (AgileX) and dexterous humanoid hands (TienKung 2.0), handling complex articulated object manipulation.

5. Significance and Claims

The paper claims that LaST0 addresses the critical trade-off between reasoning depth and inference speed in robotic VLA models. By shifting reasoning from the linguistic space to a physically grounded latent space, the framework captures the "ineffable" physical dynamics essential for robust manipulation without the latency penalty of autoregressive text generation.

The authors emphasize that the dual-system architecture allows the model to maintain the benefits of "reason-before-act" (temporal consistency, physical understanding) while operating at frequencies suitable for real-time control. The results suggest that latent reasoning is a more effective and efficient paradigm for embodied AI than explicit textual or visual prediction, particularly for tasks requiring fine-grained physical interaction and long-horizon planning.