Technical Summary: µ0: A Scalable 3D Interaction-Trace World Model

1. Problem Statement

Robot learning faces a fundamental "data paradox": while videos provide a scalable, abundant source of physical behavior data, the most effective supervision for control—action-labeled robot data—is scarce, expensive, hardware-specific, and incompatible across different robot embodiments.

Existing approaches to world models struggle to bridge this gap:

• Pixel-space video models: While scalable, they expend model capacity on dense appearance and background reconstruction, often failing to capture the metric geometry, contact structure, and occlusion patterns necessary for manipulation.
• Direct action models (e.g., Vision-Language-Action models): These remain limited by the scarcity and embodiment-specificity of labeled demonstrations.
• Existing motion-centric methods: Prior 3D flow or trajectory models often under-sample critical small regions (like tool tips), conflate object motion with camera motion by operating in local/2D coordinates, or pair long demonstrations with coarse episode-level captions rather than event-level intent.

The paper posits that a scalable solution requires a representation that describes what must move (interaction points) independent of how a specific robot moves it, while preserving global 3D structure and aligning motion with language.

2. Methodology

The proposed solution, µ0, is a query-conditioned 3D trace-space world model. It does not predict dense pixels or robot-specific actions directly. Instead, it forecasts smooth 3D trajectories for semantically selected interaction points (objects, tools, hands, contact regions). The system consists of two primary components: TraceExtract (the data engine) and µ0 (the world model).

2.1 TraceExtract: A Scalable Data Pipeline

TraceExtract converts heterogeneous human and robot videos into event-captioned 3D trace supervision. It addresses the limitations of prior fixed-grid methods through three stages:

1. Semantic Keypoint Sampling: Instead of uniform grids, it uses DINOv2 features to cluster patches into entity-level groups. It allocates a fixed keypoint budget per entity, selecting spatially diverse points on high-visibility frames. A movement filter removes static or background-dominated tracks to focus on interaction signals.
2. 3D Trace Construction: To maintain global 3D consistency across long videos with camera motion, the system employs global-local reconstruction. It uses sparse anchor frames to establish a shared global coordinate frame, reconstructs dense local chunks, and aligns them back to the global frame. Tracks are propagated across chunk boundaries using the last valid world-space position. The resulting traces are reprojected into a per-chunk reference camera to remove camera motion while retaining image alignment.
3. Event-Centric Captioning: Long demonstrations are segmented into motion-centric events based on trace acceleration. Prominent acceleration peaks define action anchors, and valleys define chunk boundaries. A Vision-Language Model (VLM) generates captions for these chunks (start, midpoint, end), which are then merged by a text-only LLM to create fine-grained and coarse task summaries.

2.2 The µ0 World Model

µ0 is a modular architecture designed to predict future 3D traces given an observation, language instruction, and optional keypoint history.

• Multi-Modal Conditioning Backbone: It utilizes a pretrained SmolVLM2-2.2B to encode RGB observations and language instructions. Depth (optional) is routed through a separate trainable patch stem before sharing deeper SigLIP layers with RGB tokens to exploit geometric cues without disrupting pretrained statistics.
• Permutation-Equivariant Trace Expert: This component treats each keypoint as an exchangeable query. It represents future motion as cubic B-spline control points rather than dense waypoints, ensuring compactness and smoothness. Each query token is grounded by local DINO features, Fourier embeddings for pixel location, and segment embeddings for history vs. future.
• Semantic Flow Matching: The model is trained as a conditional flow model over B-spline control points. It learns to denoise noisy control points into clean future traces. The objective includes:
  • Flow loss: Matching the velocity field of control points.
  • Validity prediction: Identifying when a trajectory should terminate (e.g., due to occlusion).
  • Semantic rigidity: Encouraging keypoints within the same DINO cluster to preserve local geometry.
• Trace-Conditioned Action Expert: For downstream robot control, the pretrained µ0 (VLM backbone + Trace Expert) is frozen. An Action Expert is trained on top of µ0's trace-denoising features. The policy reads features from a single partial-denoising step, injects them into VLM features via gated cross-attention, and predicts continuous action chunks conditioned on proprioception and gripper images.

3. Key Contributions

1. TraceExtract: A scalable data engine that extracts event-captioned 3D trace supervision from heterogeneous videos via semantic keypoint selection, globally aligned 3D lifting, and hierarchical language captioning. It scales trace curation by roughly 8× over prior 3D trace datasets.
2. µ0: A query-conditioned 3D trace-space world model featuring a VLM backbone, a permutation-equivariant Trace Expert, B-spline trace targets, and semantic flow-matching training.
3. Trace-Conditioned Action Adaptation: A framework where the frozen µ0 serves as a reusable motion prior. This enables action-free video pretraining to transfer effectively to robot policies, requiring action supervision only for the target robot interface.

4. Experimental Results

4.1 Trace Prediction Performance

µ0 was evaluated on 2D and 3D trace prediction tasks against baselines including tokenized VLMs (e.g., Gemini, GPT), 2D flow models, and 3D trace models (e.g., TraceGen, Dream2Flow).

• Accuracy: µ0 achieved the best Top-5 Average Displacement Error (ADE), Final Displacement Error (FDE), and Dynamic Time Warping (DTW) scores across all time horizons in both 2D and 3D settings.
• Efficiency: µ0 demonstrated a prediction latency of 0.29s, which is 2.9× faster than the next-fastest 2D baseline (Track2Act at 0.85s) and significantly faster than large VLM APIs.
• Qualitative: µ0 produced coherent, goal-directed traces, avoiding the noise and misalignment observed in prior methods.

4.2 Downstream Robot Control

The authors evaluated the frozen µ0 paired with an action expert in simulation (RoboCasa365) and real-world (UR3 robot) scenarios.

• Simulation (RoboCasa365): On 8 manipulation tasks, µ0 + action expert achieved a 30.25% average success rate. This outperformed the action-labeled VLA π0 (25.25%) despite µ0 using no action supervision during pretraining. It also significantly outperformed the previous video-only baseline, TraceGen (23%).
• Real-World (UR3): On three tasks (Pick & Place, Pour, Unfold Towel), µ0 + action expert achieved a 91.7% average success rate. This surpassed:
  • VLM + action expert (without trace features) by 18.4 percentage points.
  • Action-labeled VLAs π0 and π0.5 by 20.0 and 11.7 percentage points, respectively.
  • TraceGen + action expert by 10.0 percentage points.
• Scaling: Performance improved with larger model sizes and more pretraining data. Scaling analysis indicated that the trace representation provides crucial motion structure that limited policy capacities cannot recover on their own.

5. Significance and Claims

The paper claims that 3D interaction traces serve as a scalable and transferable representation for cross-embodiment manipulation. By decoupling the prediction of what moves (the trace) from how it moves (the action), µ0 establishes a path for robot learning that leverages abundant video data without relying on expensive, embodiment-specific action labels.

The authors emphasize that µ0 acts as a "reusable motion prior." Once pretrained on video, the model can be frozen and paired with different action experts for various robot embodiments. The results suggest that this approach can achieve performance competitive with, or superior to, state-of-the-art models pretrained on large-scale action-labeled datasets, while maintaining the scalability of video-only pretraining.

Limitations noted by the authors:

• µ0 inherits errors from the perception stack (clustering, 3D reconstruction, tracking, captioning).
• The representation captures geometry and motion but does not explicitly model forces, tactile feedback, or contact modes.
• Current evaluations are limited to tabletop manipulation with specific embodiments; broader validation on mobile manipulators and dexterous hands is future work.