💻 computer science

ViBe: Visual Behavior Adaptation for Perceptive Humanoid Whole-Body Control

ViBe is a parameter-efficient, post-training framework that adapts motion trackers for perceptive humanoid whole-body control by grafting task-relevant visual feedback via low-rank adapters, enabling robust zero-shot sim-to-real transfer across diverse locomotion and manipulation tasks.

Original authors: Lokesh Krishna, Sarvesh Venkatesan, An Zhang, Quan Nguyen

Published 2026-09-10

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Lokesh Krishna, Sarvesh Venkatesan, An Zhang, Quan Nguyen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots that walk and move like humans have long been a goal of engineering, but teaching them to do so in the real world is a puzzle of two distinct parts. First, there is the ability to mimic human movement, a skill researchers have mastered by training machines to copy vast libraries of human motion data. These "trackers" are excellent at following a script, moving their limbs in natural, fluid ways across flat, predictable ground. However, they are intentionally designed to be blind to their surroundings; they do not see a curb, a ball, or a box. Second, there is the ability to perceive the world and react to it. Traditionally, engineers have solved this by building a separate system that acts as a planner: it looks at the environment, decides what the robot should do, and then tells the blind tracker to execute that plan. This separation works, but it creates a gap. The tracker cannot make small, instant adjustments, like shifting a foot to avoid a rock or dodging an incoming object, because it is waiting for instructions from above. It lacks the visual reflexes that allow a human to stumble and recover without thinking.

A team of researchers at the University of Southern California has developed a new way to bridge this gap, allowing a robot to see and react directly while it moves. They call their system ViBe, a method that takes a robot already trained to walk like a human and gives it the ability to adapt its movements based on what it sees, without having to relearn how to walk from scratch. Instead of building a new brain or a new planner, they inserted a small, efficient interface between the robot's eyes and its muscles. This interface learns to pick out the specific visual details that matter for a task—like the edge of a curb or the position of a ball—and feeds that information directly into the robot's movement system. The result is a machine that can perform complex, dynamic tasks in the real world, such as running parkour over uneven ground, dodging a thrown ball, or reorienting a cube in its hand, all while retaining the natural, human-like motion it was originally taught.

The researchers began with a robot that had already learned to track human motion perfectly on flat ground. This robot was "blind" in the sense that it did not use camera images to guide its steps; it simply followed a pre-set sequence of movements. To give it vision, the team attached a pre-trained visual system, one that had already learned to recognize objects and scenes from millions of images. However, simply showing the robot a camera feed was not enough; the robot needed to know which parts of the image were important. A wall of pixels contains too much information, and most of it is irrelevant to the task of walking or dodging. The team designed a small module, an extractor, that acts like a filter. It looks at the robot's current body position and the task it is trying to perform, then uses that context to scan the camera image and select only the most useful visual clues. If the robot is trying to jump over a curb, the extractor focuses on the curb's edge. If it is trying to catch a ball, it focuses on the ball's trajectory.

This selected visual information is then injected into the robot's movement system through a technique that changes only a tiny fraction of the robot's internal settings. The researchers kept the original motion tracker frozen, preserving its natural, human-like gait, and only adjusted the small pathways that carried the new visual data. This approach allowed them to train the robot for specific tasks using a method called reinforcement learning, where the robot learns by trial and error, receiving rewards for successful actions. They tested this system on four very different challenges. In one, the robot had to walk and run over curbs and uneven terrain, adjusting its foot placement in real time to avoid tripping. In another, it had to perform parkour, leaping across gaps and landing on narrow surfaces. A third task involved moving large objects, where the robot had to adjust its balance and grip while carrying a suitcase or a trash can. The final challenge was dodgeball, where the robot had to stand still, watch a ball fly toward it, and move its body out of the way without falling over.

The results showed that this method worked remarkably well. When tested in the real world, the robot performed these tasks with a high degree of success, often matching the performance of systems that were given perfect, privileged information about the environment that real robots cannot have. For instance, in the parkour task, the robot successfully navigated obstacles that caused a blind version of the same robot to crash and fall. In the dodgeball scenario, the robot learned to dodge the ball while maintaining its balance, a feat that required it to deviate from its standard standing pose in a controlled, reactive way. The researchers also found that the system was robust; it worked well even when the lighting changed from bright sunlight to dim indoor conditions, or when the colors of the objects shifted. This suggests that the robot learned to understand the shape and position of things rather than just memorizing specific colors or textures.

To prove that the robot was truly reacting to what it saw, the researchers compared their system against a version that could not see. The blind robot, even with the same underlying motion training, failed to traverse curbs or avoid obstacles, often drifting off course or colliding with objects. The version with vision, however, made precise, local corrections to its movement. It adjusted its foot placement to land on a curb, shifted its weight to carry a heavy object, and moved its body to avoid a ball. These were not large, pre-planned maneuvers but small, immediate adjustments that happened as the robot moved. The team also demonstrated that this visual adaptation could be combined with a simple, high-level planner. In a task where the robot had to reorient a cube so that a specific colored face was on top, a very basic planner simply chose a sequence of moves. The robot's visual system then handled the difficult work of finding the cube, grabbing it, and adjusting its grip to make the move successful, even if the cube was in a slightly different position than expected.

The study highlights a significant shift in how robots can be taught to interact with the world. Rather than training a robot from scratch for every new task, or relying on complex, separate planning systems, it is possible to take a robot that already knows how to move and give it the ability to see and react. The researchers showed that by keeping the core movement skills intact and only training the small connection between vision and action, they could create a robot that is both natural in its motion and capable of handling the unpredictability of the real world. While the system is not perfect and can sometimes be confused by fast-moving objects or unusual visual distractions, it represents a powerful step forward. It demonstrates that a robot does not need to be taught everything from the beginning; it can learn to adapt its existing skills to new situations by simply learning what to look for. This approach offers a practical path toward creating humanoids that can move through our world with the same fluidity and adaptability that we expect from a human.

Technical Summary: ViBe – Visual Behavior Adaptation for Perceptive Humanoid Whole-Body Control

1. Problem Statement

Humanoid whole-body control has achieved scalability through motion tracking, where controllers learn to mimic large collections of human motion data. While these "trackers" serve as effective motor priors preserving natural motion, they are intentionally designed to be blind to their environment. They lack exteroceptive feedback, meaning they cannot react to environmental constraints (e.g., curbs, moving obstacles) or external dynamics.

Conventional solutions address this via a strict hierarchy: perception builds an environment representation, a high-level planner modifies the reference motion, and the tracker executes the plan. This architecture places all environment-dependent corrections at the planning level, preventing lower-level modules from making reactive, local adjustments (e.g., correcting foot placement on uneven terrain or dodging an incoming object).

Existing learning-based perceptive controllers often train task-specific encoders and trackers from scratch or rely on teacher-student distillation, where a privileged-observation expert is distilled into a perceptive student. This approach trades semantic richness for sim-to-real ease and requires significant retraining. There remains a gap in generality-preserving approaches that can adapt pre-trained vision encoders and pre-trained whole-body trackers to diverse perceptive tasks without retraining the core components or relying on distillation.

2. Methodology: The ViBe Framework

ViBe (Visual Behavior Adaptation) is a post-training framework designed to bridge a frozen vision encoder and a frozen whole-body tracker. It adapts a pre-trained tracker to perceptive control tasks by learning only a compact, intermediate pathway.

2.1 Architecture

The framework consists of three primary components (see Fig. 2 in the paper):

Frozen Visual Encoder: A pre-trained Vision Transformer (default: Theia-Tiny ViT) processes the egocentric camera image into a grid of patch tokens and a global CLS token.
Task-Conditioned Extractor: A multi-query cross-attention module that selects task-relevant visual feedback. It generates queries from three signals:
- The global image token (CLS).
- Robot proprioception (state).
- Optional task commands (e.g., goal pose).
  The extractor attends to the patch tokens (keys/values) to produce a compact, fixed-size percept (128-dimensional).
Low-Rank Adapters (LoRA): The extracted percept is injected into the frozen whole-body tracker's decoder via LoRA adapters. This creates a "visual bypass" that modifies the tracker's behavior without altering its pre-trained weights.

2.2 Training Strategy

Objective: The system is trained via Policy Optimization (PPO) to maximize a reward function combining motion tracking ( $r_{track}$ ) and task-specific rewards ( $r_{task}$ ), such as object tracking or avoidance.
Parameter Efficiency: Only the extractor and adapters are updated. The vision encoder and the base tracker remain frozen.
Initialization: Adapters are zero-initialized so the policy initially behaves exactly like the base tracker.
Reference-Phase Annealing: To prevent policies from specializing only to short motion segments, the authors introduce a curriculum where the reference state initialization (RSI) phase is annealed. Training starts with resets spanning the full reference motion and gradually contracts to the start of the reference, forcing the agent to master the entire trajectory.

2.3 Task Formulation

The approach treats downstream tasks as episodic Markov Decision Processes (MDPs). The actor observes proprioceptive history, a future window of the reference motion, the egocentric image, and optional task commands. The policy outputs actions to construct PD targets for actuated degrees of freedom.

3. Key Contributions

The paper presents two primary contributions:

A Visual Behavior Adaptation Framework: A modular architecture that bridges frozen vision and motor priors using a task-conditioned extractor and low-rank adapters. It enables direct post-training via reinforcement learning without teacher-student distillation or auxiliary representation losses.
Zero-Shot Sim-to-Real Transfer: The framework demonstrates successful transfer to hardware across four distinct tasks:
- Perceptive Locomotion: Walking and parkour on uneven terrain (curbs).
- Object Reorientation: Repose Cube (manipulating a cube in-hand to a target face).
- Omni-object Loco-manipulation: Transporting various objects while navigating.
- Dodgeball: Reactively dodging a moving ball while maintaining a standing reference.

4. Experimental Results

Experiments were conducted on a Unitree G1 humanoid robot using the mjlab simulator (MuJoCo Warp).

4.1 Performance Metrics

Comparison to Privileged Observations: ViBe achieves 89.6% to 95.5% of the performance of policies trained with privileged observations (ground truth state) across the four tasks. This validates that post-training directly via visual feedback is effective without needing a privileged teacher.
Blind vs. Visual Tracking: In controlled comparisons, the "blind" tracker (without ViBe) fails to traverse curbs or jump effectively, often colliding or drifting. ViBe-SONIC successfully adjusts foot placement and builds momentum for jumps.
Visual Robustness: The adapted policies demonstrate robustness to out-of-distribution visual conditions, including bright midday sunlight, low-light environments, and moving RGB distractors.

4.2 Ablation Studies

Vision Encoders: The framework is flexible regarding the backbone. Pre-trained encoders (DINOv3, SigLIP2, Theia-Tiny) significantly outperform a CNN trained from scratch (Tabula Rasa).
Query Design: The multi-query design (combining CLS token, proprioception, and task command) is critical. Removing proprioception or the CLS token reduced success rates by ~15% on the Repose Cube task.
Phase Annealing: This curriculum strategy improved success rates marginally (88.5% to 93.3%) and stabilized learning dynamics.

4.3 Planner Integration

To demonstrate the efficacy of the perceptive controller, the authors employed a deliberately simple planner for the Repose Cube task. The planner had no dynamics model and simply selected fixed reference clips (e.g., "right flip"). The ViBe-adapted controller handled the complex local corrections (localization, contact placement, recovery), proving that a perceptive controller can effectively solve tasks even with a straightforward high-level planner.

5. Significance and Limitations

Significance:
The paper argues that perceptive whole-body control can be achieved by learning only the intermediate connection (extractor and adapters) while retaining pre-trained modules. This offers a general, parameter-efficient recipe for adapting humanoids to diverse environments without retraining the entire system or relying on complex distillation pipelines.

Limitations:
The authors explicitly note two limitations:

Dynamic Distractors: While robust to static appearance shifts, the policies can still be triggered by unmodeled dynamic physical distractors (e.g., mistaking a thrower's head for a ball in dodgeball).
Open-Loop Planning: The planner and controller are not trained in a closed loop. The planner may select reference transitions that fall outside the adapter's training distribution, potentially limiting task-optimal performance.

The work concludes that while ViBe provides a robust foundation, future work should target learning invariances through domain randomization and exploring closed-loop reference generation.