💻 computer science

HierKick: Hierarchical Reinforcement Learning for Vision-Guided Soccer Robot Control

This paper introduces HierKick, a vision-guided hierarchical reinforcement learning framework that combines a 5 Hz YOLOv8-based high-level task planner with a 50 Hz low-level controller to achieve robust, multi-stage soccer robot control with success rates of up to 95.2% in simulation and 80% in the real world.

Original authors: Yizhi Chen, Zheng Zhang, Zhanxiang Cao, Yihe Chen, Shengcheng Fu, Liyun Yan, Yang Zhang, Jiali Liu, Haoyang Li, Yue Gao

Published 2026-03-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yizhi Chen, Zheng Zhang, Zhanxiang Cao, Yihe Chen, Shengcheng Fu, Liyun Yan, Yang Zhang, Jiali Liu, Haoyang Li, Yue Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are coaching a soccer team, but your star player is a robot. This robot has to run, dribble, and kick a ball into a goal. Sounds simple? Not when you consider that the robot has to do this while balancing on two legs, reacting to a moving ball, and doing it all in a split second.

This paper introduces HierKick, a new "brain" for soccer robots that solves a major problem: How do you tell a robot to think about the big picture while also controlling its tiny muscles?

Here is the breakdown of how HierKick works, using some everyday analogies.

The Problem: The "Overthinking" Robot

In the past, trying to teach a robot to play soccer was like trying to teach a toddler to play chess while simultaneously teaching them how to tie their shoelaces.

The "Big Picture" (Strategy): Should I run left or right? When should I stop to aim?
The "Tiny Details" (Execution): How much force do I put in my left knee? How do I keep my balance so I don't fall over?

If you try to teach a robot to do both at once (a method called "End-to-End"), it gets confused. It's like asking a driver to navigate a complex city map and manually adjust every single piston in the engine at the same time. The result is usually a crash or a very slow, clumsy robot.

The Solution: The "Coach" and the "Athlete"

The authors of this paper realized that in nature, our brains work in layers.

The Brain (Cerebrum): Decides what to do (e.g., "Dribble past the defender").
The Cerebellum: Handles how to do it (e.g., "Adjust muscle tension to stay upright").

HierKick copies this biological design by splitting the robot's control system into two distinct layers that talk to each other at different speeds.

1. The "Coach" (The High-Level Brain)

Speed: Slow and steady (5 times per second).
Job: This is the strategist. It looks at the field, sees where the ball and the goal are, and decides the plan.
How it works: It doesn't tell the robot "move your left foot 2 inches." Instead, it says, "Hey, accelerate forward a bit more," or "Turn slightly left."
The Secret Sauce: It uses a "Coach Model" trained to understand the game. It knows that when the ball is far away, the goal is to run fast. When the ball is close, the goal is to slow down and aim.

2. The "Athlete" (The Low-Level Body)

Speed: Super fast (50 times per second).
Job: This is the muscle memory. It takes the Coach's vague instructions ("Go faster") and instantly figures out exactly how to move the robot's 12 leg joints to make that happen without falling over.
The Secret Sauce: This part is pre-trained. Think of it like a gymnast who has already spent years learning how to balance and walk. The robot doesn't need to relearn how to stand up every time it wants to kick a ball; it just follows the Coach's orders.

The Game Plan: Four Steps to a Goal

The system breaks the soccer task into four distinct phases, like levels in a video game. The "Coach" switches between these modes automatically:

Approach: The ball is far away. The Coach says, "Run straight toward it!"
Alignment: The ball is getting closer. The Coach says, "Slow down and turn your body so you are facing the goal perfectly."
Dribble: The ball is right at your feet. The Coach says, "Gently nudge it forward, don't kick it away yet."
Shoot: You are in the perfect spot. The Coach says, "KICK IT HARD!"

Why is this better?

The paper tested this system in three places: a perfect computer simulation, a slightly imperfect simulation, and the real world with a real robot.

The Results: In the computer, it succeeded 95% of the time. In the real world, it succeeded 80% of the time.
The Comparison: When they tried the old "End-to-End" method (where the robot tries to learn everything at once), it only succeeded about 25% of the time.

The "Magic" Ingredients

The Eyes: The robot uses a camera (YOLOv8) to see the ball, just like a human player.
The Reward System: Imagine training a dog. If it sits, it gets a treat. HierKick gives the robot "digital treats" (rewards) for doing the right thing at the right time.
- Running fast when far away? Good treat.
- Aiming perfectly when close? Huge treat.
- Falling over? No treat.
The Safety Net: The system includes a "regularization" rule. If the Coach tries to tell the robot to spin wildly or stop instantly, the system smooths it out so the robot doesn't trip.

The Bottom Line

HierKick is a breakthrough because it stops trying to teach a robot to be a genius and a gymnast simultaneously. Instead, it gives the robot a Coach to make the smart decisions and a trained Athlete to execute them.

This allows the robot to handle the chaos of a real soccer game—running, balancing, and kicking—with a success rate that is close to what we see in professional human players, all while reacting in the blink of an eye (about 20 milliseconds). It's a huge step toward robots that can actually play sports with us, not just watch.

1. Problem Statement

Controlling humanoid soccer robots involves multi-time-scale decision-making, requiring a balance between long-term tactical planning (e.g., approaching a ball, positioning for a shot) and short-term motion execution (e.g., maintaining balance, precise joint control).

Challenges: Traditional end-to-end Reinforcement Learning (RL) methods struggle in complex dynamic environments due to reward sparsity, convergence difficulties, high control latency, and instability.
Specific Constraints: The robot must handle unpredictable ball trajectories, visual occlusions, lighting changes, and the need to simultaneously maintain balance while executing precise kicking mechanics. A disconnect between high-level strategy and low-level execution often leads to task failure.

2. Methodology: The HierKick Framework

The authors propose HierKick, a dual-frequency Hierarchical Reinforcement Learning (HRL) framework designed to decouple tactical decision-making from motion execution. The system operates in three interconnected modules:

A. Hierarchical Architecture

High-Level Control (5 Hz):
- Role: Tactical decision-making and speed regulation.
- Perception: Utilizes YOLOv8 running at 10 Hz for real-time detection of the ball and goal.
- Policy: A "Coach Policy" (trained via PPO) takes perceptual data (ball/goal positions, robot state) and historical commands to output velocity increment commands ( $\Delta v$ ).
- Function: It does not control joints directly but guides the robot's speed and direction, integrating these increments to generate velocity commands for the low-level controller.
Low-Level Execution (50 Hz):
- Role: Precise joint control and locomotion.
- Policy: A pre-trained motion controller (fixed during high-level training) that receives velocity commands and outputs 12-dimensional joint position targets.
- Function: Executes specific skills (Approach, Alignment, Dribble, Shoot) with low latency (~20 ms), ensuring stability and responsiveness.

B. Multi-Stage Reward Mechanism

To address conflicting objectives across different phases of the soccer task, the authors designed a distance-adaptive, multi-stage reward function. The reward structure changes dynamically based on the robot's distance to the ball ( $d_{ball}$ ) and the goal:

Approach Phase: Rewards moving directly toward the ball.
Alignment Phase: Rewards aligning the "Robot-Ball-Goal" vector for optimal shooting angles.
Dribbling Phase: Rewards moving the ball toward the goal while maintaining control (preventing the ball from being kicked away).
Shooting Phase: Rewards high-speed movement and precise alignment for a powerful shot.
Regularization: Includes terms for command smoothness ( $\Delta v$ limits), survival (staying upright), and energy efficiency to prevent erratic behavior.

C. Training Strategy

Curriculum Learning: The system is trained using a curriculum that gradually increases task complexity.
Domain Randomization: Extensive randomization of friction, ball mass, sensor noise, and lighting is used in simulation (Isaac Gym) to improve sim-to-real transfer.
Asymmetric Actor-Critic: Uses privileged information (full state) during training but relies on partial observations (visual + proprioception) during deployment.

3. Key Contributions

Dual-Frequency HRL Framework: Introduction of HierKick, which successfully decouples high-level tactical planning (5 Hz) from low-level motor control (50 Hz), effectively solving multi-time-scale decision problems.
Coach Model with Adaptive Speed Control: A novel "Coach Policy" trained via curriculum learning that outputs acceleration/velocity increments rather than raw joint commands, allowing the robot to master complex skills sequentially.
Real-World Implementation: Successful deployment on the Booster T1 physical humanoid robot, achieving an end-to-end control latency of approximately 20 ms at 50 Hz, meeting real-time requirements.
Modular Design & Skill Reuse: The framework utilizes pre-trained low-level skills, significantly reducing training time and improving convergence stability compared to end-to-end approaches.

4. Experimental Results

The framework was evaluated in three environments: Isaac Gym (high-fidelity simulation), MuJoCo (physics engine validation), and the Real World (Booster T1 robot).

Success Rates:
- Isaac Gym: 95.2%
- MuJoCo: 89.8%
- Real World: 80.0%
Ablation Studies:
- Removing critical distance observations ( $d_{ball}, d_{goal}$ ) dropped success rates to 23.2%.
- Replacing the acceleration-based output with direct velocity commands dropped success rates to 8.2%.
- Standard End-to-End RL achieved only 25.6% success, highlighting the superiority of the hierarchical approach.
Performance Metrics:
- HierKick demonstrated the most concentrated kick distance distribution (lowest variance) and highest precision compared to other methods.
- The system maintained robustness against real-world uncertainties like turf friction variations and partial ball occlusions.

5. Significance and Conclusion

HierKick provides an effective paradigm for controlling humanoid robots in complex, dynamic environments. By separating strategic planning from execution, it overcomes the limitations of traditional end-to-end RL, such as instability and high latency.

Impact: The 80% success rate in the real world demonstrates strong sim-to-real transferability, proving that hierarchical control with pre-trained skills is a viable path for intelligent robot soccer.
Future Outlook: The modular design and skill reuse capabilities suggest this framework can be extended to other multi-time-scale tasks beyond soccer, offering a new pathway for intelligent robot control in unstructured environments.