🤖 machine learning

Learning Transferability: A Two-Stage Reinforcement Learning Approach for Enhancing Quadruped Robots' Performance in U-Shaped Stair Climbing

This paper presents a two-stage deep reinforcement learning approach that successfully trains a Unitree Go2 quadruped robot to autonomously climb U-shaped stairs and demonstrates the policy's robust transferability to various other staircase geometries, including straight, L-shaped, and spiral configurations.

Original authors: Baixiao Huang, Baiyu Huang, Yu Hou

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Baixiao Huang, Baiyu Huang, Yu Hou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart, four-legged robot dog how to navigate a tricky building site. Specifically, you want it to climb a U-shaped staircase—the kind where you go up, turn 180 degrees on a landing, and go up again. This is hard for robots because they can easily get confused, slip, or just give up and sit down.

This paper is about a new "training camp" the researchers built to teach the robot dog how to do this using a method called Reinforcement Learning. Think of this not as teaching the dog with treats and commands, but as a video game where the robot learns by trial and error, getting "points" for good moves and "time-outs" for bad ones.

Here is the story of how they did it, broken down into simple steps:

1. The Two-Stage Training Camp

Instead of throwing the robot straight into the hardest challenge, the researchers used a "level-up" system, similar to a video game.

Stage 1: The Training Wheels (Pyramid Stairs): First, they put the robot in a simple, pyramid-shaped stair environment. This is like teaching a child to walk on a flat sidewalk before trying to climb a mountain. The robot learned the basics: "How do I lift my leg? How do I keep my balance?"
Stage 2: The Real Challenge (U-Shaped Stairs): Once the robot was good at the basics, they moved it to the U-shaped stairs. This is the real test. The robot had to learn to go up, stop, turn around, and go up again without falling off the edge.

2. The "Don't Give Up" Rule (The Stall Penalty)

Here is the clever part. The researchers noticed a problem: when the stairs got too hard, the robot would realize, "Hey, if I just stay still, I won't fall, and I won't get a 'falling' penalty." So, it would just sit at the bottom of the stairs and do nothing.

To fix this, they added a special rule called the "Stall Penalty."

The Analogy: Imagine a coach yelling, "If you stand still for too long, you lose points!"
The Result: The robot learned that staying still was a bad idea. It was forced to keep moving and exploring different ways to climb, even if it meant taking a risk. This pushed the robot to actually solve the puzzle instead of quitting.

3. The "Brain" of the Robot

The robot uses a "brain" (a neural network) that looks at two things:

How its body feels: Is it leaning too far left? Are its legs moving too fast?
What it sees: It has a "height map" (like a 3D topographic map) of the stairs right in front of it.

The robot's brain is like a GPS combined with a sense of balance. It doesn't just know where the top of the stairs is; it understands the shape of the stairs. The researchers found that the robot learned to "feel" the safe path (the middle of the stairs) and avoid the dangerous edges (the walls), just like a human would.

4. The Magic of "Transfer Learning" (The Swiss Army Knife Effect)

The most exciting part of the paper is what happened when they tested the robot on other types of stairs. They asked: "If we train the robot on a U-shaped staircase, can it also climb straight stairs, L-shaped stairs, or spiral stairs?"

The Result: It was a huge success!
- Straight Stairs: The robot was a pro (97% success rate). It was like a master chef who learned to cook a complex dish and could easily make a simple sandwich.
- L-Shaped Stairs: It did very well (74% success).
- Spiral Stairs: It struggled a bit (35% success). Why? Because spiral stairs are like a twisting slide that never stops turning, which is very different from the sharp 180-degree turn of a U-stair.

The Big Reveal: The researchers found a funny asymmetry.

If you train a robot on straight stairs, it fails miserably on U-shaped stairs (it gets confused by the turn).
But if you train a robot on U-shaped stairs, it becomes a "Swiss Army Knife" that can handle straight stairs and L-shaped stairs easily.

The Analogy: Think of the U-shaped stair as the "Master Class." If you learn the Master Class, you automatically know how to do the easier classes. But if you only learn the easy class, you are lost when the Master Class comes around.

Why Does This Matter?

Construction sites are messy. They have debris, uneven floors, and lots of stairs. Wheeled robots (like Roombas) can't go up stairs. This research shows that we can teach robot dogs to be autonomous construction workers that can climb up and down different types of stairs without a human holding their hand.

By using this two-stage training and the "don't give up" penalty, we are one step closer to having robot dogs that can safely inspect buildings, carry tools up to the second floor, and help build our future cities.

1. Problem Statement

While quadruped robots (robot dogs) offer significant advantages for construction tasks due to their stability and payload capacity, their deployment in multi-story buildings is hindered by the inability to autonomously and robustly navigate complex staircases. Current solutions often require human intervention or exhibit instability, particularly on non-linear stair geometries like U-shaped stairs. Existing research largely focuses on flat-ground locomotion or simple straight stairs, leaving a gap in handling the combined challenges of elevation changes, directional turns, and landings found in real-world indoor construction environments.

2. Methodology

The authors propose a two-stage end-to-end Deep Reinforcement Learning (RL) framework using the NVIDIA Isaac Lab simulator and the Unitree Go2 robot dog. The approach utilizes the Proximal Policy Optimization (PPO) algorithm.

A. Two-Stage Training Framework

Stage 1 (Foundation): The robot is trained on Isaac Lab's default pyramid-stair terrain. This stage establishes basic climbing skills and locomotion stability.
Stage 2 (Specialization): The policy is fine-tuned on custom U-shaped indoor staircases. This stage develops terrain-specific navigation skills required for complex turns and landings.

B. Neural Network Architecture

Inputs/Observations: The policy takes proprioceptive states (linear/angular velocity, gravity vector, joint positions/velocities) and environmental states (a 21x21 local height map) as input.
Structure: A shallow Convolutional Neural Network (CNN) encodes the height map into a 128-dimensional latent feature vector. This is concatenated with proprioceptive data and fed into a 3-layer Multilayer Perceptron (MLP).
Shared Encoder: Both the Actor (policy) and Critic (value) networks share the CNN encoder to improve sample efficiency and ensure a common perception module, though they maintain independent MLP weights.

C. Reward Design & Curriculum

Task Rewards:
- Stage 1: Navigation rewards based on Euclidean distance to the goal.
- Stage 2: Replaced with centering rewards (staying on the stair path) and path rewards.
Regularization Penalties: Penalties for excessive torque, motor power, rapid action changes, and joint limits to ensure safe movement.
Special Penalty (Stall Penalty): A critical innovation where the robot receives a constant negative reward if its velocity drops below 0.3 m/s outside goal regions. This prevents the robot from "freezing" or hesitating when facing high-difficulty terrain, forcing it to explore alternative strategies rather than staying stationary.
Curriculum Learning: The terrain difficulty (riser height and tread width) is divided into 10 progressive levels. Robots level up or down based on success rates, preventing catastrophic forgetting and accelerating learning.

3. Key Contributions

Stall Penalty Strategy: The introduction of a velocity-based stall penalty effectively mitigates the "hesitation" problem in high-difficulty RL tasks, encouraging continuous exploration and climbing behavior.
Two-Stage Transfer Learning: Demonstrating a successful workflow where a general climbing policy is specialized for complex U-shaped geometries.
Asymmetric Transferability Analysis: The study provides empirical evidence regarding the directionality of knowledge transfer between different stair geometries (U-shaped vs. Straight/L-shaped/Spiral).
Critic Network Visualization: The authors visualize the critic's value function, proving that the learned policy encodes terrain-conditioned long-horizon feasibility rather than simple Euclidean distance to the goal.

4. Experimental Results

The experiments evaluated success rates on U-shaped stairs and the transferability of the trained model to other stair types (Straight, L-shaped, Spiral) and vice versa.

A. U-Shaped Stair Performance (EXP1)

The robot achieved high success rates (91.3% at Level 3, 87.7% at Level 4) on U-shaped stairs.
Performance dropped sharply at Level 6, indicating limits in current RL strategies for extreme difficulty.
Critic Analysis: Heat maps showed high value regions aligned with the stair path and goal, while low-value regions corresponded to walls and edges, confirming the policy understands geometric constraints.

B. Transferability: U-Shaped Model $\to$ Other Terrains (EXP2)

Straight Stairs: 97.2% success rate (97.6% transferability). The model generalized exceptionally well to straight paths.
L-Shaped Stairs: 73.9% success rate (83.6% transferability). The model handled discrete turns effectively.
Spiral Stairs: 35.3% success rate (50.6% transferability). Performance was poor, suggesting the U-shaped training environment lacks the continuous curvature and asymmetric geometry of spiral stairs.

C. Transferability: Other Models $\to$ U-Shaped Terrain

Models trained on Straight, L-shaped, or Spiral stairs achieved extremely low success rates (1.0% – 1.5%) when transferred to the U-shaped terrain.
Conclusion: The U-shaped environment induces more diverse and demanding locomotion behaviors. It acts as a "superset" of challenges; training on simpler terrains does not prepare the robot for the complexity of U-shaped stairs, whereas training on U-shaped stairs prepares the robot for simpler configurations.

5. Significance and Future Work

Significance: This work addresses a critical bottleneck in construction robotics: autonomous multi-floor navigation. It demonstrates that a single policy trained on a complex, representative geometry (U-shaped) can generalize to simpler, common geometries (Straight/L-shaped), making it a viable strategy for real-world deployment.
Future Work: The authors plan to conduct Sim-to-Real experiments to validate the simulation-based policies on physical Unitree Go2 robots in real-world construction sites.

In summary, the paper establishes that a two-stage RL approach with a specific stall penalty enables quadruped robots to master complex U-shaped stair climbing. Crucially, it reveals an asymmetry in transferability: training on complex, multi-segment stairs yields a robust policy capable of handling simpler stairs, while training on simple stairs fails to generalize to complex ones.