Technical Summary: Perceptive Behavior Foundation Model

Problem Statement

Humanoid behavior foundation models aim to acquire reusable whole-body control policies from broad human motion priors, enabling a single controller to produce diverse and expressive behaviors. However, existing motion-centric foundation policies operate under a critical assumption: that the reference motion is already physically compatible with the robot's surroundings.

This assumption fails in scenarios of operator–environment mismatch, where the human demonstrator, operator, and robot inhabit different environments. A human motion command specifies intent and style but does not inherently encode the footholds, swing clearance, body height, or contact timing required by the robot's local terrain (e.g., stairs, sparse supports, or obstacles). Consequently, a robot attempting to track a flat-ground reference on complex terrain may fail due to collisions, balance loss, or infeasible kinematics. The core challenge is to ground human motion priors in robot-centric perception without altering the raw motion command interface.

Methodology: Perceptive Behavior Foundation Model (Perceptive BFM)

The authors propose Perceptive BFM, a single-policy framework that preserves raw kinematic motion references as the deployment command while using local terrain observations to adapt contacts, posture, and timing. The approach is built around a staged Perceptive Motion Tracking (PMT) training algorithm.

1. Terrain-Conformal Reference Synthesis (TCRS)

To provide scalable terrain supervision, the authors develop an offline synthesis pipeline that converts raw human motion clips and sampled height fields into terrain-consistent references ($m_{tcrs}$). This module does not solve full contact-rich dynamics but constructs contact-consistent, smooth, and style-preserving references through:

• Contact-aware foothold construction: Latching stance feet to terrain surfaces while preserving the global gait phase and nominal footfall timing.
• Foot-geometry-aware swing optimization: Optimizing a virtual mid-foot trajectory (balancing toe and heel clearance) using a sampling-based trajectory optimizer to avoid terrain discontinuities.
• Support-aware root reconstruction: Reconstructing the root (pelvis) height based on support contacts and leg reachability.
• Collision repair and multi-point leg IK: Solving a damped, support-aware multi-point Jacobian Inverse Kinematics (IK) problem for the twelve leg joints to resolve collisions and ensure kinematic feasibility.

2. Teacher-Student Training Pipeline

The training process involves four stages:

• Stage 1 (Offline Synthesis): TCRS generates paired datasets of raw references ($m_{raw}$), terrain-conformal references ($m_{tcrs}$), and terrain maps ($\tau$).
• Stage 2 (Blind Teacher Training): A "blind" Transformer-based teacher policy is trained via Proximal Policy Optimization (PPO) to track the synthesized $m_{tcrs}$ references. This teacher has access to privileged information (the adapted reference) but no visual terrain input.
• Stage 3 (Vision Student Distillation): A "vision student" policy, which receives the original raw reference ($m_{raw}$) and a local terrain scan, is distilled from the teacher.
  • Target-Frame Action Alignment: Since the teacher acts on $m_{tcrs}$ and the student on $m_{raw}$, the student learns to mimic the teacher's effective joint target expressed relative to the raw reference: $a^*_t = (q_{tcrs} + \mu_{tea}) - q_{raw}$.
  • Identity-Gated Architecture: The student is an identity-gated Transformer. Terrain features enter through residual pathways initialized to zero. This ensures that at initialization, the student behaves exactly as a raw-reference tracker, learning terrain-conditioned corrections only when they improve the tracking objective.
• Stage 4 (PPO Fine-tuning): The student is fine-tuned with PPO using raw-reference commands and terrain observations, conservatively updating the transferred motion-tracking prior while learning local perception-conditioned corrections.

Key Contributions

1. TCRS Pipeline: A scalable offline synthesis method that converts raw human motion and terrain height fields into terrain-consistent supervision via contact-aware foothold construction, swing optimization, root reconstruction, and multi-point IK.
2. PMT Algorithm: A raw-reference teacher–student framework that deploys terrain-aware behavior without changing the command interface. It transfers terrain-conformal behavior from a blind teacher to a vision student via target-frame action alignment.
3. Identity-Gated Policy Design: A Transformer policy that initializes as a raw-reference tracker and learns terrain-conditioned residual corrections. This enables a single command interface to support locomotion, expressive motions, acrobatics, and operator–environment mismatch across diverse terrains.

Results

The paper evaluates Perceptive BFM across five levels:

• TCRS Quality: On stepping stones with stairs, TCRS significantly reduces collision-sensitive metrics compared to baselines (Z-offset and Cubic Interp+IK). Specifically, it reduces penetration depth from 5.48 cm (Z-offset) to 2.38 cm and clearance violations from 14.3% (Cubic) to 7.4%.
• End-of-Training Diagnostics: In a common 10k-iteration PPO setup, the full PMT policy achieves a logged closed-loop success rate of 55.1%, significantly outperforming ablations:
  • Without TCRS (raw reference only): 27.3%
  • Without vision (blind): 26.5%
  • Without target-frame alignment: 26.7%
  • Without identity gating: 30.4%
• Cross-Terrain Simulation: In fixed-policy rollouts across 30 motion commands and 5 terrain families (stairs, slopes, sparse supports, recessed obstacles, mixed indoor), PMT achieves a 55.1% completion rate compared to 27.3% for the raw-reference baseline. PMT also demonstrates lower lower-leg collision incidence (10.0% vs. 24.7%).
• Real-Robot Deployment: A single policy was deployed on a 29-DoF Unitree G1 robot. It successfully executed diverse behaviors (one-leg backflips, stair dancing, arm-waving runs, sideways stair walking) across indoor, outdoor, and mocap-mismatch settings. The robot preserved the commanded upper-body behavior while perception resolved footholds and clearance online.

Significance and Claims

The paper claims that robot-centric perception can turn human motion priors into terrain-compatible whole-body behavior without changing the raw motion command interface.

The significance lies in bridging the gap between generalist behavior foundation models (which excel at expressive tracking but ignore terrain) and perceptive locomotion policies (which handle terrain but often replace human intent with autonomous skills). Perceptive BFM demonstrates that a single policy can retain the "intent and style" of a human operator while dynamically adapting the physical realization (contacts, posture, timing) to the robot's local environment. The authors note that this approach allows for the reuse of broad human motion priors for complex tasks like acrobatics and teleoperation in mismatched environments, provided the upper-body command remains feasible after lower-body correction.

The paper acknowledges limitations, including the assumption of static, rigid terrain and the lack of upper-body collision awareness (arms/torso may still strike obstacles if the command is collision-unaware). Future work is suggested to address dynamic/deformable terrain and upper-body adaptation.