Learning Agile Striker Skills for Humanoid Soccer Robots from Noisy Sensory Input

Technical Summary: Learning Agile Striker Skills for Humanoid Soccer Robots from Noisy Sensory Input

Problem Statement
Learning fast and robust ball-kicking skills remains a critical yet challenging capability for humanoid soccer robots. Unlike quadrupeds, humanoids must maintain dynamic balance on a single support foot while executing rapid leg swings. This task requires precise coordination of whole-body control under significant perceptual uncertainty, including sensor noise, delays, and external perturbations (e.g., opponents pushing the robot or the ball). Existing methods often struggle to generalize across diverse ball-goal configurations or fail to maintain stability when perception is imperfect, creating a significant gap between simulation training and real-world deployment.

Methodology
The authors propose a four-stage reinforcement learning (RL) training pipeline designed to bridge the sim-to-real gap and enable robust continual ball-kicking under imperfect perception. The framework utilizes a teacher-student architecture:

1. Long-Distance Ball Chasing (Teacher): A "privileged" teacher policy is trained with access to ground-truth ball and goal positions, as well as the ball's velocity and physical parameters. The environment employs domain randomization (e.g., external pushes, randomized mass/friction) to encourage recovery from imperfect states. The reward function focuses on tracking linear and angular velocities aligned with the robot-ball vector.
2. Directional Kicking (Teacher): The teacher policy is extended to learn precise kicking motions. The environment simulates a RoboCup Adult Size field. New reward components are introduced: a ball velocity reward (encouraging high velocity toward the goal, capped to prevent exploitation) and a head tracking reward (encouraging visual contact with the ball).
3. Teacher Policy Distillation (Student): The privileged teacher policy is distilled into a "student" policy that operates under realistic, noisy perception. The student observes a history of proprioceptive data and noisy estimates of ball/goal positions.
   • Noise Modeling: The system simulates realistic uncertainty via a velocity-dependent noise model (noise magnitude scales with object velocity), asynchronous updates (ball position updates at ~10 Hz vs. proprioception at 50 Hz), and frame drops due to occlusion.
   • Training: The student is trained using Dataset Aggregation (DAgger), iteratively collecting rollouts under the student policy and querying the teacher for corrective actions to learn recovery from its own errors.
4. Student Adaptation and Refinement: The final stage applies online constrained RL to refine the student policy.
   • Constrained RL: The authors identify that standard fixed regularization coefficients can cause "inhomogeneous credit assignment," leading to jittery motions or unsafe sharp turns immediately before a kick (where future task rewards are high). To mitigate this, they reformulate the problem as a constrained optimization task using N-P3O. The objective maximizes task performance (kicking accuracy/speed) while constraining the aggregated regularization cost (e.g., torque, joint acceleration) to a specific threshold. This ensures smoother, safer motions without sacrificing performance.

Real-World Deployment
The policy is deployed on a Booster T1 humanoid robot (1.18m tall, 23 DoF). The perception pipeline uses a YOLOv8 model for ball detection and depth unprojection, combined with a learned legged-inertial odometry module (inspired by Legolas) for goal localization. The control policy executes at 50 Hz on an NVIDIA AGX Orin.

Key Results

• Simulation: The policy achieved an average kick accuracy of 0.956 and a success rate of 79.5% across diverse ball-goal configurations. The maximum ball velocity reached 4.13 m/s, with an average energy cost of 110.8 J/s. The system demonstrated an emergent energy-saving strategy, applying lighter kicks when the ball was closer to the goal.
• Real-World: On the physical robot, the policy achieved an average success rate of 66.7% across five different ball-goal configurations, with kick accuracies ranging from 0.94 to 0.99.
• Ablation Studies:
  • Constrained RL (N-P3O): Compared to standard PPO with fixed regularization, N-P3O significantly improved stability (lower max angular velocity) and energy efficiency (108.6 J/s vs. 255.8 J/s for PPO-1.0) while maintaining high success rates.
  • Student Adaptation: The online adaptation stage was critical; the pre-adaptation student achieved only 52.3% success and 0.807 accuracy with double the energy cost, whereas the adapted student matched the simulation performance (79.5% success).

Significance and Claims
The paper claims to present a comprehensive framework for learning robust, continual humanoid ball-kicking skills under imperfect perception. By combining a staged teacher-student curriculum with realistic noise modeling and constrained online adaptation, the work establishes a benchmark for agile visuomotor skill learning in humanoid whole-body control. The authors emphasize that this approach successfully closes the sim-to-real gap, enabling zero-shot deployment on physical hardware without task-specific tuning for each configuration.

Limitations
The authors note that the long-distance chasing gaits rely on reward functions engineered from walking gait training, and the transfer to kicking depends on the similarity of these motions. They suggest that generalizing to tasks with fundamentally different motion patterns may require future work on learning from diverse human motions.