💻 computer science

VAIC: Vision-Guided Humanoid Agile Object Interaction Control via Decoupled Commands

This paper introduces VAIC, a vision-guided control framework that enables humanoid robots to perform agile, diverse object interactions in unstructured environments by distilling a privileged teacher policy into a deployable student policy that relies solely on onboard depth, proprioception, and decoupled velocity commands.

Original authors: Dongting Li, Qianyang Wu, Xingyu Chen, Liang Li, Yuhang Lin, Sikai Wu, Guoyao Zhang, Mingliang Zhou, Diyun Xiang, Qiang Zhang, Renjing Xu, Jianzhu Ma

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dongting Li, Qianyang Wu, Xingyu Chen, Liang Li, Yuhang Lin, Sikai Wu, Guoyao Zhang, Mingliang Zhou, Diyun Xiang, Qiang Zhang, Renjing Xu, Jianzhu Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a humanoid robot trying to learn how to carry a heavy box up a flight of stairs, push a wobbly shopping cart, or ride a skateboard. In the past, teaching a robot these skills was like trying to teach a human by forcing them to follow a rigid, second-by-second script of every muscle movement. If the script was even slightly off, or if the robot couldn't see exactly where the object was (because the box was blocking its view), the robot would fall over.

The paper introduces VAIC (Vision-Guided Agile Interaction Control), a new "brain" for robots that changes the game. Instead of following a rigid script, VAIC teaches the robot to understand intent and feel its way through the world, even when it can't see everything clearly.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Scripted" Robot vs. The Real World

Current robots are like actors who have memorized a play perfectly but can't improvise.

The Script: They need a perfect, detailed map of every joint movement (like a dance routine).
The Blind Spot: If a robot is carrying a big box, the box blocks its camera. It can't see the ground or the object it's holding. Without a perfect map, it panics and falls.
The Gap: In the real world, humans don't give robots a full dance routine. We just say, "Walk forward" or "Push that cart." Old robots couldn't handle this vague instruction.

2. The Solution: The "Decoupled Command" (The GPS and the Switch)

VAIC introduces a new way for humans to talk to robots. Instead of a full script, the human gives two simple things:

The GPS (Velocity): "Go forward at this speed."
The Switch (Interaction State): A simple on/off switch that says, "I am just walking" or "I am now touching/pushing/pulling."

This separates "Where to go" from "How to touch." It's like telling a driver, "Drive to the store," rather than telling them exactly how many degrees to turn the steering wheel every second.

3. The Training: The "Teacher" and the "Student"

The paper uses a clever two-step training method, like a master chef training an apprentice.

Step 1: The Privileged Teacher (The Master Chef)
First, they train a "Teacher" robot in a perfect video game simulation. This Teacher has "superpowers": it can see through walls, knows the exact weight of every object, and knows the perfect physics of the world. It learns how to carry boxes and ride skateboards perfectly.
- Analogy: Imagine a chess grandmaster playing against a computer that knows every possible future move. The grandmaster learns the perfect strategy.
Step 2: The Student (The Apprentice)
Next, they train a "Student" robot. This Student is the one that will actually go into the real world. It does not have the superpowers. It can't see through the box, and it doesn't know the exact weight.
- The Magic Trick: The Student watches the Teacher and tries to guess what the Teacher is thinking. It uses a special "Object Adapter" (a smart memory bank) that looks at the blurry camera images and the robot's own body feelings (proprioception) to guess where the object is and how it's moving.
- Analogy: The Student is like a blindfolded person learning to juggle by listening to the sound of the balls hitting the floor and feeling the air currents. They have to infer the ball's path without seeing it.

4. The Results: Walking the Walk

The researchers tested this on a real robot with three very different, difficult tasks:

Carrying a Box: The robot had to walk up stairs and slopes while holding a box that blocked its view. VAIC succeeded where others failed because it could "feel" the terrain through the box.
Pushing a Cart: The cart was wobbly and hard to control. VAIC learned to anticipate the cart's wobble and adjust its balance instantly.
Skateboarding: The robot had to balance on a skateboard, which involves sudden, jerky movements. VAIC kept its balance where other robots fell over.

Why This Matters

The paper claims that VAIC is a "unified" system. This means the same brain (policy) can handle carrying a box, pushing a cart, and riding a skateboard without needing to be retrained for each specific task. It bridges the gap between the perfect world of computer simulations and the messy, unpredictable real world.

In short: VAIC teaches robots to stop following a rigid script and start using their "sixth sense" (combining camera data and body feelings) to figure out what's happening around them, allowing them to interact with objects agilely even when they can't see them perfectly.

Technical Summary: VAIC (Vision-Guided Humanoid Agile Object Interaction Control)

Problem Statement

Humanoid robots face a critical "deployment gap" when transitioning from simulation to real-world unstructured environments, particularly regarding agile interaction with objects. While recent advances in deep reinforcement learning and motion imitation have improved whole-body control, existing controllers suffer from two fundamental limitations:

Reliance on Dense Reference Trajectories: Current methods often require precise, joint-level reference motions at runtime. This "kinematic hand-holding" is impractical for real-world autonomy or teleoperation, where human operators can only provide high-level intent (e.g., velocity commands).
Dependence on Perfect State Observability: Many approaches assume access to ground-truth 6D object poses and precise terrain geometry. In physical deployment, downward-facing vision is frequently occluded by carried objects, and sensor noise creates blind spots, rendering state estimation unreliable.

These limitations hinder the ability of robots to perform tightly coupled whole-body coordination tasks, such as carrying bulky objects over stairs, manipulating underactuated carts, or skateboarding, where the object's dynamic state fundamentally alters the robot's balance.

Methodology: VAIC Framework

The authors propose VAIC (Vision-Guided Agile Interaction Control), a unified framework that bridges the gap between oracle-level simulation tracking and physical, perceptive deployment. The core insight is to decouple high-level user intent from low-level interaction execution.

1. Decoupled Command Interface

VAIC replaces dense reference motions with a deployment-friendly, intent-driven command interface defined as $u_t = (v_t, c_t)$ :

$v_t$ (Velocity Target): A multi-axis velocity target ( $\in \mathbb{R}^3$ ) representing navigational intent ( $[v_x, v_y, v_{yaw}]$ ).
$c_t$ (Interaction Indicator): A per-frame binary indicator specifying the manipulation phase ($0$ for navigating/approaching, $1$ for active contact/interaction).

This formulation separates "where to go" from "how to interact," enabling intuitive teleoperation and autonomous navigation without requiring the policy to track specific joint trajectories.

2. Two-Stage Distillation Paradigm

VAIC employs a specialized two-stage online distillation process to transfer skills from a privileged teacher to a deployable student:

Stage I: Teacher Training & Student Warm-up:
- A Privileged Teacher Policy is trained via Reinforcement Learning (PPO) in an oracle simulation setting. It has access to exact object kinematics ( $s^{obj}_t$ ), terrain geometry, and dense reference motions.
- Concurrently, a Student Policy is warmed up. The student's input is strictly limited to proprioception and the decoupled commands ( $v_t, c_t$ ), while the visual input (depth) is zero-injected.
- Distillation: The student learns to mimic the teacher's actions via Behavior Cloning (BC) while auxiliary modules (Object Adapter and Privilege Adapter) are trained via supervised learning to implicitly infer the missing privileged states (object dynamics and geometric features) from the teacher's latent representations.
Stage II: Vision-Guided Student Exploration:
- The dense reference motions and perfect object states are completely removed from the student's inputs.
- The Student Actor takes full control, using PPO to optimize for task completion and command tracking in an environment with realistic sensor noise.
- Object Adaptation Module: A recurrent module (CNN-GRU) processes temporal depth streams and proprioceptive history to implicitly infer unobservable object dynamics ( $\hat{s}^{obj}_t$ ). The binary indicator $c_t$ acts as a mode-switching prior, helping the network anticipate changes in contact forces.
- Privilege Adapter: Projects the inferred object state and geometric templates into the robot's frame to reconstruct the policy latent space.
- Online Adaptation: The adapter modules continue to update online using the student's own rollout data, preventing representation drift and grounding visual reconstruction in actual physical interactions.

Key Contributions

Decoupled Command Interface: A control formulation that replaces dense reference motion requirements with high-level velocity targets and phase indicators, enabling intuitive teleoperation and generalization across different dynamic profiles without task-specific retraining.
Occlusion-Resistant Distillation: A method utilizing a recurrent Object Adaptation module to distill privileged geometric and dynamic states from a teacher. This allows the robot to implicitly reconstruct unobservable object dynamics and execute precise interactions even under severe visual occlusion (e.g., carrying a box that blocks the camera view) and sensor noise.
Unified Hardware Validation: Demonstration that a single policy can generalize across diverse terrain sequences (flat, slopes, stairs) and adapt to out-of-distribution object attributes (varying weights, sizes, and positions) in real-world deployments.

Experimental Results

The framework was evaluated on a humanoid robot equipped with an onboard Intel RealSense D435i camera across three challenging categories: Box Carrying (over varied terrain), Cart Manipulation (pulling/pushing underactuated payloads), and Skateboarding (balancing under impulsive dynamics).

Quantitative Performance: In MuJoCo simulations, VAIC significantly outperformed baselines (PPO, AMP, PhysHSI, VisualMimic).
- Box Interaction: Achieved a 66.7% success rate compared to 0% for standard PPO and 50% for AMP, with lower tracking errors for both the robot root and the object.
- Cart Interaction: Achieved a 91.7% success rate, vastly superior to the 41.7% (PPO) and 58.3% (AMP) of baselines, effectively handling non-holonomic constraints.
- Skateboard Interaction: Achieved an 83.3% success rate, while PPO failed completely (0%) and AMP achieved only 12.5%.
Generalization: Real-world hardware tests confirmed robustness to out-of-distribution conditions, including varying velocity commands, payload weights, interaction sides, and box sizes. The system successfully maintained balance and interaction stability without retraining.
Mechanism Analysis: Comparison of the predicted object states against ground truth showed that the depth-conditioned estimator tightly tracks object position and orientation even during self-occlusion. Conditioning on the interaction indicator $c_t$ was found to significantly reduce temporal drift compared to purely proprioceptive alternatives.

Significance and Claims

The paper claims that VAIC presents a practical paradigm for autonomous humanoid deployment by eliminating the restrictive reliance on dense joint-level reference trajectories and perfect state observability. By successfully bridging the gap between simulation and reality through vision-guided distillation, VAIC enables humanoids to execute highly diverse, dynamic, and agile object interactions in unstructured environments. The authors assert that this unified approach advances the field by demonstrating that a single policy can handle radically different dynamic profiles (carrying, pushing, skateboarding) and generalize to unmodeled physical variations, thereby addressing a critical bottleneck in real-world humanoid robotics.

Limitations

The authors acknowledge several limitations:

The depth-based Object Adaptation module is susceptible to sensor noise and may fail with transparent or highly reflective objects.
Extreme out-of-distribution physical variations (e.g., sudden, drastic mass shifts) can still induce representation collapse before the robot can adjust.
The decoupled command interface is efficient for gross loco-manipulation but lacks the granularity required for dexterous, fine-grained in-hand manipulation.