⚡ electrical engineering

VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

This paper introduces VLK, a framework that synthesizes vision-language-kinematics supervision from 3D Gaussian Splatting-reconstructed scenes to train humanoid robots for effective sim-to-real perception-based loco-manipulation without human intervention.

Original authors: Yen-Jen Wang, Jiaman Li, Sirui Chen, Takara E. Truong, Pei Xu, Pieter Abbeel, Rocky Duan, Koushil Sreenath, Angjoo Kanazawa, Carmelo Sferrazza, Guanya Shi, Karen Liu

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yen-Jen Wang, Jiaman Li, Sirui Chen, Takara E. Truong, Pei Xu, Pieter Abbeel, Rocky Duan, Koushil Sreenath, Angjoo Kanazawa, Carmelo Sferrazza, Guanya Shi, Karen Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot butler how to walk around a house, pick up a box, and put it on a table. The hardest part isn't just telling the robot what to do (like "pick up the box"); it's teaching its body how to move to do it, all while looking at the world through its own eyes.

This paper presents a new way to teach robots this skill without needing humans to physically guide them thousands of times. Here is the breakdown using simple analogies:

The Big Problem: The "Missing Link"

To teach a robot, you usually need three things happening at the exact same time:

What it sees: A video from the robot's eyes.
What it hears: A voice command (e.g., "Walk to the sofa").
What it does: A perfect map of how its entire body moved to make that happen.

Existing data sources are like a broken puzzle. You might have videos of people walking, or you might have robot movements, but you rarely have all three pieces perfectly matched together for every possible room and task. Collecting this data in the real world is slow, expensive, and dangerous for the robot.

The Solution: The "Digital Twin" Factory

The authors built a factory that creates this missing data entirely inside a computer. Think of it as a video game engine that never stops playing.

Rebuilding the World (The Set): They take real photos and scans of actual rooms (like a lab or an apartment) and turn them into a hyper-realistic 3D digital world. It's like creating a "digital twin" of a real room where the robot can walk around.
The Invisible Director (The Script): Instead of hiring a human to walk around and record themselves, they use a computer program to "direct" the robot. The computer knows exactly where the walls and furniture are (privileged information). It tells the robot: "Walk to the chair," "Pick up the box," and "Put it down."
The Magic Camera (The View): Once the computer figures out the perfect body movements, it "replays" the scene from the robot's perspective. It takes a picture of what the robot would have seen, creating a perfect match between the command, the view, and the movement.

The Result: 48,000 Lessons in a Flash

Using this system, they generated 48,000 unique training examples in just a few days.

The Teacher: A computer program (the VLK policy) that learns from these 48,000 examples. It learns to look at an image, hear a command, and predict exactly how the robot's joints should move next.
The Student: A real robot (the Unitree G1) that takes these predictions and tries to execute them.

How It Works in Real Life

When they tested the real robot:

The Brain: The robot looks at the room and hears, "Pick up the box from the floor."
The Prediction: Based on its training, it predicts a short path for its whole body to move.
The Tracker: A separate, fast system takes that prediction and makes the robot's motors actually move, ensuring it doesn't fall over or drop the box.

What They Found

It Works: The robot successfully walked around real rooms, picked up boxes from the floor, and placed them on tables.
More Data = Better Skills: The more synthetic examples they generated, the better the robot got. Simple walking was easy to learn, but picking up and placing objects required more practice (more data).
Lighting Matters: They found that if they didn't train the robot with different lighting conditions in the simulation (like turning lights on/off or changing shadows), the robot got confused in the real world. Adding "visual chaos" to the training made the robot much tougher and more adaptable.

The Catch (Limitations)

The robot is currently very good at moving large, box-like objects that it can hold with both hands. It is not yet good at picking up tiny, delicate things like a coffee cup or a screwdriver. The training data focused on big interactions, so the robot hasn't learned the fine motor skills needed for small objects yet.

In short: The paper shows that you can build a "digital playground" where a robot practices millions of times against a perfect computer director, and then step out into the real world and actually do the job, all without a human holding its hand.

Technical Summary: VLK – Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

Problem Statement

Perception-based humanoid loco-manipulation requires mapping high-dimensional egocentric visual observations and natural language instructions to coordinated whole-body kinematic behaviors. A fundamental bottleneck in scaling this capability is the lack of large-scale, synchronized datasets containing the complete tuple of: (1) egocentric visual observations, (2) task instructions, and (3) robot-compatible whole-body kinematic trajectories.

Real-world teleoperation provides high-quality data but is expensive and difficult to scale across diverse scenes and objects.
Human motion capture offers rich kinematic references but lacks corresponding robot egocentric views.
Egocentric video datasets provide visual experience but lack paired robot kinematics or actions.

Consequently, there is no existing source that provides the complete vision-language-kinematics (VLK) supervision necessary for scalable policy learning.

Methodology

The authors propose a pipeline that generates paired VLK supervision synthetically within metric-scale reconstructed indoor scenes. The system decouples the generation of world assets, behavior synthesis, and visual rendering to overcome data scarcity.

1. Synthetic Data Generation Pipeline

The pipeline operates in three stages to produce 48,000 paired trajectories without human intervention:

Scene Reconstruction and Annotation: Indoor environments are reconstructed from iPhone 14 Pro scans using 3D Gaussian Splatting (3DGS) to preserve photorealistic detail and metric scale. Semantic 3D bounding boxes and walkable regions are annotated to define constraints for motion synthesis.
Robot Motion Synthesis: Using the annotated scene constraints, the system synthesizes Unitree G1 trajectories for two task families:
- Object-directed navigation: Trained on motions from BONES-SEED.
- Object interaction: Trained on OMOMO sequences retargeted to the G1 morphology using OmniRetarget.
- A conditional diffusion model generates coupled G1-object trajectories, conditioning on task instructions, initial states, object geometry, and desired wrist contact labels.
Egocentric View Rendering: The synthesized G1 trajectories are replayed in Isaac Sim within the reconstructed 3DGS scenes. A virtual ZED 2i camera (calibrated to the physical robot) renders paired RGB observations. Domain randomization (lighting, camera extrinsics, focal length) is applied to improve robustness to sim-to-real visual gaps.

2. Vision-Language-Kinematics (VLK) Policy

The core learning component is a policy that predicts short-horizon whole-body kinematic trajectories.

Input: Current egocentric RGB image ( $o_t$ ), task instruction ( $\ell$ ), and current robot kinematic state ( $x_t$ ).
Output: A future trajectory ( $\hat{x}_{t+1:t+H}$ ) of length $H=30$ (1 second at 30 Hz) and binary wrist-object contact labels.
Architecture & Training: The policy is initialized from a pretrained $\pi_0.5$ model and fine-tuned using a flow-matching formulation with an $x_0$ -prediction objective. Auxiliary losses are applied for foot-floor contact, trajectory consistency, forward kinematics accuracy, and foot-sliding regularization.
State Representation: The kinematic state includes root planar displacement, yaw, height, 6D root orientation, joint angles (encoded via sine/cosine to avoid discontinuities), and contact labels.

3. Whole-Body Tracking and Deployment

Execution: The VLK policy's predictions serve as high-level references for a contact-aware whole-body tracker (based on SceneBot). The tracker converts the predicted kinematics into joint-level PD targets for the physical Unitree G1.
Real-Time System: Deployment utilizes a real-time VLA system with Triton as the inference backend. On an RTX 5090, end-to-end latency is stable at 31 ms. To handle motion blur and ensure continuity, the system employs sharpness-based frame selection and overlapping action chunks.

Key Contributions

VLK Supervision Generation: A novel pipeline that synthesizes 48,000 paired trajectories (egocentric images, language, and kinematics) in reconstructed 3DGS scenes, addressing the data bottleneck for humanoid loco-manipulation.
Decoupled Data Synthesis: A method that leverages privileged scene information (e.g., object poses, collision geometry) to ease behavior generation while rendering observations after the fact, enabling massive scaling without human demonstration.
Sim-to-Real Transfer: Demonstration that a policy trained entirely on synthetic, reconstructed scenes can successfully perform perception-based loco-manipulation on a physical Unitree G1 in real-world environments.
Contact-Aware Tracking: Integration of predicted binary contact labels into the whole-body tracking loop to stabilize object transport during manipulation.

Experimental Results

The system was evaluated on the Unitree G1 in two environments (lab and apartment) with varying furniture layouts.

Success Rates (Real-World):
- Navigation: Achieved near-perfect success (19/20 to 20/20) for "Walk To" and "Turn Around" tasks.
- Floor Manipulation: High success for picking and placing boxes from the floor (16/20 to 20/20).
- Surface Manipulation: Lower success rates for placing boxes on surfaces (8/20 to 15/20), attributed to limited coverage of support-surface heights in the retargeted interaction data.
Simulation vs. Reality: The system achieved high success rates in MuJoCo simulation (with IsaacSim rendering), validating the closed-loop performance before real-world deployment.
Data Volume Ablation: Performance scales with the amount of synthesized data. While navigation is robust even with limited data, manipulation tasks (specifically "Pick Surface") showed significant improvement, rising from 0% success at 10% data volume to 46% with the full dataset.
Domain Randomization: Visual domain randomization was critical for sim-to-real transfer. Removing it reduced walking success from 90% to 41% under visual perturbations.

Significance and Claims

The paper claims that synthesizing robot-compatible interactions within reconstructed scenes provides effective supervision for sim-to-real perception-based humanoid loco-manipulation.

The authors position this work as a solution to the data bottleneck that has hindered scalable policy learning. By connecting partial solutions—3DGS for realistic world assets and whole-body tracking for behavior execution—they demonstrate a viable "Real-to-Sim-to-Real" pipeline. The results suggest that generating synthetic interactions in reconstructed environments is a promising path toward scalable, perception-conditioned humanoid control, particularly for tasks involving navigation and the transport of large objects.

Limitations Acknowledged:
The current system is limited by the coverage of the OMOMO dataset, making it better suited for bimanual transport of box-like objects rather than grasping small items (e.g., cups, tools). Additionally, the contact-aware tracker stabilizes large-object manipulation but does not address precise object grasping. Extending this to small-object manipulation would require richer interaction data and specialized low-level controllers.