Generalizing from References using a Multi-Task Reference and Goal-Driven RL Framework

1. Problem Statement

The paper addresses a fundamental trade-off in learning agile humanoid robot behaviors:

• Reference-Tracking RL: Policies trained to strictly follow motion capture (MoCap) data produce high-quality, natural, and coordinated movements. However, they are often brittle, failing to generalize when faced with novel goals, environments, or initial conditions outside the demonstration dataset.
• Goal-Driven (Tabula Rasa) RL: Policies trained purely on task objectives (e.g., "reach this point") are highly adaptable but often produce unnatural, uncoordinated, or aggressive movements because they lack the inductive bias of human motion data.

Existing approaches (e.g., distillation, adversarial imitation) attempt to bridge this gap but often remain tightly coupled to reference data during inference or struggle with stability in dynamic, contact-rich scenarios. The authors aim to create a controller that learns structured, human-like motor skills from references but can flexibly adapt to novel goals without requiring reference trajectories at execution time.

2. Methodology

The authors propose a Unified Multi-Task Reinforcement Learning (RL) Framework that trains a single goal-conditioned policy on two distinct but sharing tasks.

A. Core Architecture

• Single Policy: A goal-conditioned policy $\pi(s_t, g_t)$ is trained jointly on two tasks.
• Observation Space: The policy observes the robot state $s_t$ (joint configuration, velocities, gravity projection) and a goal $g_t$ (target root position). Crucially, reference motions are never provided as inputs to the policy.
• Action Space: The policy outputs residual actions mapped to joint PD controllers. Unlike standard tracking, no feedforward reference joint positions are used.

B. The Two Tasks

1. Reference-Guided Imitation Task:
   • Goal: Derived from reference motions.
   • Reward: Dense imitation rewards ($r_{track}$) encouraging the robot to match reference pose, velocity, and joint configurations, plus regularization and survival rewards.
   • Purpose: Provides dense supervision to shape the policy toward natural, coordinated, and physically plausible behaviors (Representation Learning).
2. Goal-Conditioned Generalization Task:
   • Goal: Randomly sampled independent of any reference.
   • Reward: Sparse task rewards ($r_{goal}$) based on reaching the target position/orientation, plus the same regularization and survival terms.
   • Purpose: Forces the policy to adapt the learned motor skills to achieve objectives under diverse conditions without relying on the reference.

C. Curriculum Learning & Training

• Assistive Wrench: An automatic difficulty scheduler ($\lambda$) controls a virtual assistive wrench applied to the robot's base. This wrench is strong early in training (low $\lambda$) to stabilize learning of dynamic skills (like jumping) and vanishes as training progresses.
• Task Mixing: The sampling probability of the imitation vs. generalization task is linearly interpolated based on $\lambda$. The curriculum starts with heavy imitation (for skill acquisition) and gradually shifts toward generalization (for adaptation).
• State Initialization:
  • Imitation: States initialized near reference trajectories.
  • Generalization: States and goals sampled from wide distributions to force robustness.
• Algorithm: Proximal Policy Optimization (PPO) with an asymmetric actor-critic architecture. The critic receives privileged information (full state, contact forces) available only in simulation to improve training stability.

3. Key Contributions

1. Decoupling Motion Quality from Reference Inputs: The framework treats reference motion as a prior for behavioral shaping rather than a deployment-time constraint. The final policy executes purely based on state and goal, enabling generalization beyond the training distribution.
2. Unified Multi-Task Formulation: By jointly optimizing imitation and generalization tasks within a shared observation space, the policy acquires structured motor skills while learning to adapt them to novel scenarios.
3. No Adversarial or Explicit Tracking at Inference: The method avoids the instability of adversarial imitation learning (AIL) and the rigidity of explicit trajectory tracking during deployment.
4. Long-Horizon Skill Composition: The learned skills can be composed via a rule-based state machine to perform complex, multi-stage parkour sequences (e.g., walk $\to$ jump $\to$ climb $\to$ descend).

4. Experimental Results

The method was evaluated on the Unitree G1 humanoid (29 DoF) in simulation (MuJoCo, Isaac Lab) and on real hardware.

• Tasks: Walk-Jump, Walk-Climb, and Climb-Down over a box-based parkour environment.
• Robustness & Generalization:
  • The proposed method achieved high success rates (up to 100% nominal, ~76% beyond-nominal) even when initial conditions (position, orientation, box height) were significantly perturbed.
  • In contrast, ZEST (tracking-based) failed under beyond-nominal conditions (success rate dropped to ~17-57%) because it tried to force alignment with the reference trajectory, causing instability.
  • Tabula Rasa RL achieved high success rates but produced uncoordinated, aggressive, and unnatural movements (e.g., jumping straight off a box instead of climbing down).
• Motion Quality: The proposed method maintained low root orientation error and reasonable joint position errors, preserving human-like style while adapting to new goals.
• Ablation Studies:
  • Removing the curriculum caused complete failure on dynamic tasks (jumping/climbing).
  • Removing the imitation task resulted in greedy, uncoordinated behaviors.
  • Removing the generalization task preserved style but failed to adapt to new conditions.

5. Significance

This work demonstrates a practical pathway to flexible, goal-driven humanoid autonomy in unstructured environments. By successfully transferring skills learned from human data to novel, complex scenarios (parkour) without requiring reference inputs at runtime, the framework overcomes the "brittleness" of traditional imitation learning. It proves that a single policy can simultaneously master the style of human motion and the adaptability required for real-world deployment, paving the way for more robust and versatile humanoid robots capable of complex athletic behaviors.