Technical Summary: Stubborn – A Unified Reinforcement Learning Framework for Humanoid Motion Tracking and Fall Recovery

1. Problem Statement

Recent advances in reinforcement learning (RL) have improved humanoid motion tracking and fall recovery, yet significant challenges remain in unifying these capabilities. Current approaches typically treat motion tracking and fall recovery as distinct tasks, necessitating multi-stage training, specialized recovery rewards, or separate recovery policies. Furthermore, standard RL training often employs "hard termination," where episodes end immediately upon severe tracking failure (e.g., a fall). This design limits the agent's ability to explore recovery behaviors in unstable or fallen states, as the learning signal is cut off before recovery can be attempted. Additionally, uniformly sampled training data often under-represents difficult motion segments and unstable states, hindering the policy's exploration of robust recovery strategies.

2. Methodology: The Stubborn Framework

The authors propose Stubborn, a streamlined and unified RL framework designed to train a single policy capable of robust motion tracking and autonomous fall recovery. The framework is built upon an asymmetric Actor-Critic architecture and integrates three core technical components:

A. Yaw-Aligned Tracking Representation

To mitigate sensitivity to global drift and heading disturbances while preserving gravity-related balance information, Stubborn adopts a yaw-aligned tracking representation. This approach removes global heading rotation from the reference motion, allowing the policy to focus on relative body dynamics rather than correcting for absolute translation and heading deviations after perturbations.

B. Bernoulli-Based Probabilistic Termination (PT) Mechanism

Instead of deterministically terminating an episode when tracking errors exceed a threshold, Stubborn employs a conditional Bernoulli termination mechanism.

• Mechanism: When the root height error ($e_{root,z}$) or orientation error ($e_{root}$) exceeds predefined thresholds, the episode terminates with a user-specified probability ($p_{term}$) rather than immediately.
• Rationale: This creates a probabilistic "recovery window." If the episode continues, the policy is encouraged to explore resilient actions and learn emergent recovery behaviors without requiring auxiliary, recovery-specific reward designs.
• Parameterization: The termination probability is derived from physical recovery constraints. With a control frequency of 50 Hz and a target physical recovery time of 4 seconds, the expected survival steps are set to 200 ($p_{term} = 0.005$). This ensures that >99% of unstable episodes terminate before the maximum step limit, mitigating temporal difference (TD) distortion while allowing sufficient time for recovery exploration.

C. Probabilistic Termination and Tracking-Error-Driven Adaptive Sampling

Under the PT mechanism, episode termination is no longer a reliable proxy for segment difficulty due to stochastic termination. To address this, the authors propose an adaptive sampling strategy driven by online tracking performance:

• Weight Update: The sampling weight of reference frames is dynamically reshaped based on the average keypoint tracking error ($\bar{e}$) of the current episode.
• Logic:
  • If the average error is high ($\bar{e} \ge \theta_{success}$), the weights for the involved frames are increased to prioritize these difficult segments.
  • If the episode terminates early despite low error, weights are redistributed to focus on the specific segments where termination occurred.
  • If an episode runs to maximum length with low error, weights are slightly decayed to prevent over-training on easy segments.
• Goal: This strategy increases the training frequency for difficult motion segments and unstable states, improving learning efficiency without relying on hard termination statistics.

3. Key Contributions

The paper outlines three primary contributions:

1. Unified Policy Learning: A single policy that achieves robust motion tracking and fall recovery without separate recovery controllers or multi-stage training.
2. Bernoulli-Based Soft Termination: A mechanism that encourages the learning of resilient actions in unstable states, synthesizing emergent recovery behaviors without specialized reward engineering.
3. Adaptive Sampling Strategy: A novel sampling distribution that dynamically adjusts based on online tracking errors and probabilistic termination, effectively targeting difficult motion segments and recovery states.

4. Experimental Results

The framework was evaluated on the 29-DoF Unitree G1 humanoid robot in both simulation (IsaacLab/MuJoCo) and real-world settings, using motions from the LAFAN1 and AMASS datasets.

Motion Tracking Performance

Compared to state-of-the-art baselines (HoloMotion, Any2Track, BFM-Zero, and a From-scratch RL baseline), Stubborn achieved superior tracking accuracy:

• MPBPE (Root Position Error): Reduced from 62.68 mm (From-scratch) to 48.85 mm.
• MPJVE (Joint Velocity Error): Reduced from 887.87 rad/s to 624.03 rad/s.
• The method demonstrated stable performance with lower standard deviations across the full LAFAN1 dataset.

Robustness and Fall Recovery

Ablation studies under strong external perturbations (5 m/s) demonstrated the efficacy of the proposed components:

• Probabilistic Termination (PT): The full Stubborn framework achieved a 100% recovery success rate at both strict (0.15 m) and moderate (0.25 m) thresholds. In contrast, the variant without PT (w/o PT) achieved only 77.5% and 85.0%, respectively. The PT variant also converged faster and required fewer recovery steps.
• Adaptive Sampling (AdpS): The inclusion of the adaptive sampling strategy further reduced joint position, velocity, and acceleration errors, particularly in the middle and late stages of training.

Real-World Deployment

Real-world experiments on the Unitree G1 confirmed the policy's ability to track diverse dynamic motions (including flips and acrobatics), withstand disturbances, and recover from falls in physical environments.

5. Significance and Claims

The paper claims that Stubborn represents a significant step toward unifying motion tracking and fall recovery within a single, streamlined framework. By harnessing natural transitions into fall states rather than avoiding them, the framework achieves high-fidelity tracking and robustness without the architectural complexity of multi-stage training or separate recovery policies. The authors assert that the proposed probabilistic termination mechanism and adaptive sampling strategy are the primary drivers of the observed performance gains in both simulation and real-world deployment.

The work concludes by noting that while the current framework is effective, future efforts will focus on improving the agility and smoothness of recovery motions following severe falls and extending the framework to more challenging terrains.