Tree Learning: A Multi-Skill Continual Learning Framework for Humanoid Robots

This paper proposes Tree Learning, a multi-skill continual learning framework for humanoid robots that utilizes a hierarchical parameter inheritance mechanism and multi-modal adaptation to efficiently acquire new skills while preventing catastrophic forgetting and enabling seamless real-time control.

The Gist

Imagine you are teaching a robot to be a human. You start by teaching it the most basic thing: how to walk on flat ground. Once it masters that, you want to teach it to run, climb stairs, do push-ups, and kick a ball.

In the past, trying to teach a robot all these things at once was like trying to teach a student to be a doctor, a pilot, and a chef simultaneously by shoving all three textbooks into their brain at the same time. The robot would get confused, forget how to walk while trying to learn to run, or just crash because the instructions conflicted. This is called "catastrophic forgetting."

This paper introduces a new method called Tree Learning. Here is how it works, explained simply:

1. The Family Tree Analogy 🌳

Instead of one giant brain trying to remember everything, the researchers built a family tree for the robot's skills.

• The Root (The Trunk): This is the basic skill: Walking on flat ground. The robot learns this first and perfectly. It's the foundation.
• The Branches: Once the "trunk" is strong, the robot grows "branches."
  • One branch learns Running.
  • Another branch learns Climbing Stairs.
  • A smaller twig on the "Standing" branch learns Kicking a Ball.

The Magic Trick: When a new branch grows, it doesn't start from scratch. It inherits the muscle memory and balance from the trunk. It's like a child learning to ride a bike; they already know how to balance, so they only need to learn how to pedal. Because the new skill "borrows" the old skill's brainpower, the robot never forgets how to walk while learning to run.

2. The "Specialized Muscle" System 💪

The paper mentions "physically isolated sub-networks." Think of this like a gym with separate rooms.

• In the old way, everyone tried to exercise in one big room, bumping into each other and getting in the way.
• In Tree Learning, the robot has a main control room (the trunk) and private training rooms (the branches).
• When the robot needs to run, it goes into the "Running Room." When it needs to walk, it goes back to the "Walking Room."
• Because the rooms are separate, practicing running doesn't mess up the walking muscles. This is why the robot can switch skills instantly without stumbling.

3. The "GPS and Compass" Guide 🧭

To make learning faster, the researchers gave the robot two special tools:

• The Rhythm (Phase Modulation): For things that repeat, like walking or running, the robot gets a built-in metronome. It knows, "Left leg, right leg, left leg," automatically. It doesn't have to guess the rhythm; it just has to focus on not falling.
• The Map (Interpolation): For things that don't repeat, like squatting down, the robot gets a simple map. It knows, "Start high, go low, stop." It fills in the details on its own.

4. The "Video Game" Tests 🎮

To prove this works, they put the robot in two video game-style worlds:

• Super Mario Mode: The robot had to run from a ghost, climb stairs, crawl through a tunnel, and kick a ball into a goal. It did this by switching skills instantly. When the ghost got close, it switched from "Walk" to "Run" without tripping.
• Chinese Garden Mode: The robot had to navigate a complex garden with bridges and stairs for 4 minutes straight. It automatically decided when to run (because the target was far away) and when to climb stairs (because it saw steps), all while staying perfectly balanced.

Why This Matters

Before this, teaching robots new tricks was slow, expensive, and risky because they kept forgetting old ones. Tree Learning is like giving the robot a modular brain.

• Efficiency: It learns new skills much faster because it builds on what it already knows.
• Safety: It never forgets the basics, so it won't suddenly fall over when trying a new move.
• Versatility: It can handle everything from a slow walk to a high-speed kick, just like a human athlete.

In short, Tree Learning turns the robot from a confused student trying to memorize everything at once into a master craftsman who builds new skills on top of a solid foundation, one branch at a time.

Technical Summary

1. Problem Statement

The paper addresses critical challenges in embodied intelligence and reinforcement learning (RL) for humanoid robots as they transition from single-task to multi-skill paradigms. The core problems identified are:

• Catastrophic Forgetting: Existing methods often degrade previously learned fundamental capabilities when learning new motions.
• Scalability and Efficiency: Current approaches like Mixture-of-Experts (MoE) require complex topology adjustments or training massive models, making lightweight deployment on edge devices difficult.
• Training Costs: Motion imitation methods often require full policy retraining for new skills, leading to high computational costs and inefficient exploration.
• Generalization: Existing robust control strategies are often limited to basic gait adaptation and struggle with cross-modal motion generalization at low training costs.

2. Methodology: Tree Learning Framework

The authors propose Tree Learning, a hierarchical continual learning framework inspired by biological evolution (prior inheritance and specialized evolution). The system is built on the Unitree G1 humanoid robot and utilizes Proximal Policy Optimization (PPO) within the Unity ML-Agents simulation environment.

A. Hierarchical Architecture (Root-Branch Structure)

• Root Skill: A pre-trained flat-ground walking gait serves as the foundational "root."
• Branch Skills: New skills (e.g., running, stair climbing, squatting, ball kicking) are learned as "branches" that inherit parameters from their parent nodes.
• Physical Isolation: To prevent gradient interference and catastrophic forgetting, the framework uses physically isolated sub-network clusters. Each skill has its own network weights, but they share the same state space and action representation dimensions. This allows for seamless switching without action discontinuities.

B. Multi-Modal Feedforward Adaptation

To accelerate learning and provide motion priors, the framework employs two distinct feedforward mechanisms:

1. Phase Modulation (Periodic Motions): For cyclic tasks like walking, running, and jumping, a cosine phase modulation method generates gait activation factors ($u_1, u_2$) to schedule alternating leg movements. This provides a baseline trajectory for the RL policy to refine.
2. Interpolation (Aperiodic Motions): For non-cyclic tasks like squatting or crawling, linear interpolation is used to generate reference joint angles with adjustable depth, providing strong physical boundaries for the policy.

C. Reward Shaping and Curriculum Learning

• Reward Design: A composite reward function includes velocity tracking, angular velocity, center-of-mass (CoM) height, orientation constraints, survival rewards, and action energy penalties.
• Curriculum Learning: For complex tasks (e.g., stair climbing), the environment difficulty is progressively increased (e.g., step height from 0.01m to 0.15m) to prevent the agent from getting trapped in local optima.

D. Inference and Control

• Seamless Switching: The system maintains a consistent global state space. When switching skills (e.g., from walking to running), the system loads the corresponding ONNX model. Because the input/output dimensions and state representations are identical, transitions are smooth and continuous.
• User Interaction: Supports both manual switching (via keyboard) and automatic switching based on environmental cues (e.g., detecting stairs or a "ghost" pursuer).

3. Key Contributions

1. Tree-Structured Parameter Inheritance: A novel architecture that fundamentally eliminates catastrophic forgetting by isolating sub-networks while leveraging shared state spaces. This enables lossless incremental expansion of skills.
2. Low-Cost Iteration Pipeline: A design that supports incremental fine-tuning of specific branches without retraining the entire system, significantly reducing computational and time costs.
3. Multi-Modal Adaptation: A unified mechanism combining phase modulation and interpolation to handle both periodic gaits and aperiodic large-range motions within a single framework.
4. Validation on Unitree G1: Successful demonstration of 11 highly dynamic skills (including running, jumping, crawling, and ball kicking) with real-time interactive control.

4. Experimental Results

The framework was validated through extensive simulations, including comparisons with a simultaneous multi-task baseline and complex scenario tests.

• Performance vs. Multi-Task Baseline:

  • Tree Learning achieved significantly higher final rewards across all tested skills compared to a single-network multi-task baseline.
  • Example: For the "Run" skill, Tree Learning achieved a reward of 6138, whereas the multi-task baseline only reached 609. For "Jump," Tree Learning reached 990 vs. 99 for the baseline.
  • Retention: Tree Learning maintained a 100% skill retention rate, whereas multi-task training suffered from negative transfer and gradient conflicts.

• Super Mario Scenario (Interactive Control):

  • The robot successfully navigated a complex, dynamic environment involving chasing ghosts, climbing stairs, crawling through tunnels, and kicking balls.
  • Automatic Switching: The system demonstrated smooth transitions between walking, running, and climbing without posture instability or step-like jumps in velocity/CoM height.

• Autonomous Navigation (Chinese Garden):

  • The robot performed a 240-second continuous navigation task covering ~30m x 30m, including path planning, obstacle avoidance, and gait switching.
  • Reliability: Successfully reached 22 target points with only one stuck recovery.
  • Stability: CoM height remained stable (~0.78m) on flat ground and transitioned smoothly to ~1.78m when climbing stairs. Roll and pitch angles were kept within safe limits (±2.5° on flat, ±5–7° on stairs).

5. Significance and Impact

• Solving Catastrophic Forgetting: By decoupling skills into isolated sub-networks while sharing state representations, Tree Learning offers a mechanistic solution to the negative transfer problem inherent in multi-task RL.
• Edge Deployment Feasibility: The framework avoids the need for massive, monolithic models, making it suitable for lightweight deployment on resource-constrained edge devices.
• Real-World Applicability: The ability to seamlessly switch between heterogeneous locomotion modes (walking, running, climbing) in real-time is a crucial step toward general-purpose humanoid robots operating in unstructured, real-world environments.
• Future Work: The authors acknowledge the current "Sim-to-Real" gap and plan to incorporate domain randomization and implicit state estimation to deploy these skills on physical robots, as well as extending the framework to long-period complex motions like dance.

In conclusion, Tree Learning provides an efficient, scalable, and robust system-level solution for the continual learning of diverse motor skills in humanoid robots, bridging the gap between specialized single-task controllers and general-purpose embodied intelligence.