Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data

1. Problem Statement

Learning athletic, high-dynamic sports skills on humanoid robots is a significant challenge due to the need for rapid reactions, high precision, and complex whole-body coordination. Specifically for tennis:

• Data Scarcity: Collecting perfect, complete human-tennis motion sequences from real-world matches is extremely difficult due to the large movement range (spanning half a court), multi-rally interactions, and the need for high-speed motion capture.
• Imperfect Data Limitations: Existing solutions often rely on broadcast videos (requiring complex pipelines) or tele-operation (impossible for high-speed sports). The available "imperfect" data (motion fragments of primitive skills like forehands or shuffles) suffers from two main issues:
  1. Imprecision: Wrist motions during ball striking are often inaccurate due to capture difficulties and the embodiment gap between humans and robots.
  2. Incompleteness: The data provides priors for how to move (primitive skills) but lacks knowledge on how to use these skills to return a ball to a specific target.

The core problem is how to leverage these imperfect, fragmentary human motion data to train a humanoid robot that can stably sustain multi-shot rallies with natural motion styles.

2. Methodology: LATENT System

The authors propose LATENT (Learns Athletic humanoid TEnnis skills from imperfect human motioN daTa), a three-stage framework combining hierarchical control, latent action spaces, and robust sim-to-real transfer.

A. Data Collection & Pre-processing

• Imperfect Data Collection: Instead of full matches, the team collected 5 hours of motion fragments (primitive skills: forehand, backhand, lateral shuffle, crossover step) from five amateur players using a compact optical motion capture system (3m×5m).
• Retargeting: Human motions are retargeted to the humanoid robot (Unitree G1) using Loco-MuJoCo.
• Handling Imprecision: The right wrist (racket-holding) control signals are removed during training, and random perturbations are applied to the wrist joints. This forces the system to learn to handle wrist inaccuracies.

B. Correctable Latent Action Space Construction

To bridge the gap between noisy data and precise control, the authors construct a latent space via Online Distillation:

1. Motion Tracker Pre-training: A motion tracker (RL policy) is trained to imitate the collected primitive skills.
2. Variational Bottleneck: A student network (Encoder-Decoder) is trained to distill the tracker's skills into a continuous latent space ($z$).
   • Encoder: $E(z|s, \tilde{s})$ maps state and motion target to a latent code.
   • Decoder: $D(a|s, z)$ maps the latent code back to robot actions.
   • Learnable Prior: Unlike standard VAEs with fixed Gaussian priors, they use a learnable conditional prior $P(z|s)$ to capture state-dependent action distributions (e.g., different distributions for shuffling vs. swinging).
   • Robustness: The wrist joints are excluded from the latent action output, allowing the high-level policy to correct them later.

C. High-Level Policy Learning

A high-level planner ($\pi_{planner}$) is trained using PPO to correct and compose the primitive skills stored in the latent space.

• Hybrid Control: The policy outputs two components:
  1. Latent Action ($a_{latent}$): Samples from the latent space to drive the body.
  2. Correction Action ($a_{correct}$): Direct control commands specifically for the right wrist to fix the imprecise retargeted motions.
• Latent Action Barrier (LAB): To prevent the RL agent from "hacking" the latent space (sampling suboptimal actions that break natural motion), a barrier is imposed.
  • The exploration of the residual latent action is constrained within a Mahalanobis distance around the prior's mean ($\mu_p$).
  • The range is adaptively scaled based on the prior's standard deviation ($\sigma_p$), ensuring the policy stays within the "skill manifold" while allowing necessary corrections.

D. Sim-to-Real Transfer

To deploy on the real Unitree G1 robot, the authors employ:

• Dynamics Randomization: Randomizing robot parameters (friction, mass, CoM) and ball physics (mass, restitution, air drag).
• Observation Noise: Adding noise, frame dropouts, and latency to observations. Crucially, they use a 4-frame sliding window to average ball velocity, mitigating errors from finite-differencing noisy position data.

3. Key Contributions

1. LATENT Framework: A novel system enabling athletic tennis skills using only imperfect, fragmentary human motion data, significantly lowering data collection barriers.
2. Novel Latent Space Designs:
   • Correctable Latent Space: Decouples body control from wrist control to handle embodiment gaps.
   • Latent Action Barrier (LAB): Constrains RL exploration to preserve natural motion styles while allowing task-specific corrections.
3. Real-World Deployment: Successful deployment on the Unitree G1 humanoid, achieving stable multi-shot rallies with human players.
4. Comprehensive Validation: Extensive experiments in simulation and the real world demonstrating superiority over baselines (PPO, MotionVAE, AMP, ASE, PULSE).

4. Experimental Results

Simulation Results

• Success Rate (SR): LATENT achieved 96.52% success rate for forehand returns and 82.10% for backhands, significantly outperforming baselines (e.g., PULSE at ~71%, AMP at ~41%).
• Action Quality: LATENT demonstrated lower Distance Error (DE), higher Smoothness (lower joint acceleration), and lower Torque requirements compared to baselines.
• Ablation Studies:
  • Removing wrist correction dropped SR to ~82%.
  • Removing the Latent Action Barrier caused jittery motions and dropped SR to ~93%, proving LAB is essential for motion naturalness.

Real-World Results

• Performance: The robot successfully sustained rallies with human players.
  • Forehand SR: 90.90%
  • Backhand SR: 77.78%
• Robustness: The method proved that extensive dynamics randomization (without precise system identification) is sufficient for sim-to-real transfer.
• Comparison: Baselines without dynamics randomization or observation noise failed significantly (SR < 25% in some cases).

5. Significance

This work represents a major step forward in humanoid sports robotics. It demonstrates that high-level athletic skills do not require perfect, expensive motion capture datasets. By effectively utilizing "imperfect" data fragments and correcting them via hierarchical control and latent barriers, LATENT achieves:

• Natural Motion: The robot moves with human-like fluidity rather than robotic, jerky motions.
• High Dynamics: It handles balls traveling at >15 m/s and reacts within milliseconds.
• Generalizability: The framework is applicable to other sports (soccer, parkour) where complete human motion data is unavailable.

The paper concludes that combining imperfect human priors with robust RL correction strategies is a viable path toward deploying agile, athletic humanoid robots in the real world.