Technical Summary: Training and Simulation of Quadrupedal Robot in Adaptive Stair Climbing and Descending for Indoor Firefighting

Problem Statement

Indoor firefighting primary searches require rapid, thorough environmental surveys to locate victims and monitor flammable materials under hazardous conditions. While quadrupedal robots (robot dogs) offer superior adaptability to uneven terrain and cluttered spaces compared to wheeled or tracked robots, they face two critical technical hurdles in this domain:

1. Situational Awareness: Navigating complex, dynamic indoor environments with visual obstructions (e.g., smoke) and interpreting sensor data to map 3D space.
2. Robust Locomotion: Developing high-speed, reliable capabilities to ascend and descend diverse staircase types (straight, L-shaped, spiral, and damaged steps) found in residential and commercial buildings.

Existing approaches often rely on hierarchical planning (separating perception, planning, and control) or simplified stair environments, which can increase system complexity and limit generalization to varied stair topologies.

Methodology

The authors propose a two-stage end-to-end deep reinforcement learning (RL) framework using the NVIDIA Isaac Lab platform and the Unitree Go2 quadruped robot. The approach unifies navigation and locomotion into a single neural network policy, avoiding hierarchical planning.

1. Training Framework

The training utilizes the on-policy algorithm Proximal Policy Optimization (PPO) and is divided into two distinct stages:

• Stage 1 (Skill Acquisition): The robot is trained on a preprogrammed "pyramid-stair" terrain that tapers to a flat platform. The goal is to acquire fundamental stair-climbing and descending locomotion skills. The robot navigates to specific goal positions and orientations.
• Stage 2 (Generalization & Navigation): The policy from Stage 1 is transferred to a curriculum of realistic, customized indoor staircases (straight, L-shaped, and spiral). The task shifts to tracking a stair centerline, requiring the robot to adapt to specific stair topologies while maintaining locomotion.

2. Network Architecture and Observations

• Input (Observations): The policy receives proprioceptive states (linear/angular velocity, joint positions/velocities, gravity vector), user-commanded goals (position and yaw), and environmental states. The environmental state includes a 21x21 local height map (0.2m resolution) representing terrain around the robot's feet and center.
• Architecture: The policy network consists of a shallow Convolutional Neural Network (CNN) encoder (processing the height map) and a 3-layer Multilayer Perceptron (MLP). The CNN outputs a 128-dimensional feature vector representing terrain information, which is combined with other observations and fed into the MLP to output 12 joint position commands.
• Critic Network: Shares the CNN encoder with the policy network to estimate long-term returns, reducing the training burden for terrain feature extraction.

3. Reward Function and Curriculum

The reward function is a weighted sum of task/navigation rewards and regularization penalties:

• Task Rewards:
  • Stage 1: Coarse and fine navigation rewards based on Euclidean distance to the goal.
  • Stage 2: Centering reward (penalizing deviation from the stair centerline) and path reward (encouraging forward progress along the centerline).
• Regularization/Penalties: Includes penalties for excessive motor power, torque, joint velocity/acceleration, action changes, and collisions (body/head/hip vs. ground/obstacles). A "flying gait" penalty and step reward encourage stable, grounded movement.
• Curriculum Learning: Training progresses through 10 difficulty levels per terrain type. Difficulty increases via step height (0–12 cm), stair width (2.0–1.4 m), and stair length (including turn lengths for L-shaped and spiral lengths). Robots level up only upon successfully reaching the goal with the specified yaw.

Key Contributions

The paper identifies four primary contributions:

1. Two-Stage End-to-End RL Framework: A method that successfully transfers climbing/descending skills from abstract pyramid terrain to realistic indoor stair topologies (straight, L-shaped, spiral).
2. Centerline-Based Navigation Formulation: A unified learning approach for navigation and locomotion that eliminates the need for hierarchical planning, relying solely on local height-map perception.
3. Policy Generalization: Demonstration that the learned policy can generalize across diverse stair shapes using only local environmental information, without global maps.
4. Empirical Analysis: A comprehensive evaluation of success rates, efficiency, and failure modes under increasing stair difficulty.

Results and Evaluation

The policy was evaluated on 300 episodes across straight, L-shaped, and spiral staircases with varying step heights (4cm to 14cm).

• Success Rates: Success rates generally decreased as difficulty levels increased. Spiral stairs proved the most challenging, with a significant drop in success rates at Level 6.
• Locomotion Metrics:
  • Velocity: Mean linear velocity generally decreased as difficulty increased.
  • Climb/Descend Rate: Showed an overall increasing trend with difficulty levels, suggesting the robot maintained vertical progress despite reduced forward speed.
  • Errors: Position and heading errors increased with difficulty. Straight stairs yielded the lowest errors, while L-shaped and spiral stairs showed higher errors, particularly surging at Level 6.
  • Power Consumption: Power usage increased with staircase complexity. However, at Level 6, power consumption decreased, attributed to the policy adopting conservative, slower motions to avoid failure penalties.
• Stage Comparison: Stage 2 training significantly improved performance over Stage 1 models. Stage 1 models failed on L-shaped stairs, often "short-cutting" the turn and colliding, whereas Stage 2 models successfully recognized stair patterns and navigated corners.
• Robustness: The robot demonstrated the ability to navigate stairs using only local height maps, indicating robustness to partial observability, a critical feature for firefighting where global building maps may be unavailable.

Significance and Claims

The paper claims that the proposed end-to-end RL approach effectively balances navigation and locomotion, enabling quadrupedal robots to adapt to various stair shapes while maintaining high agility. The authors emphasize that the centerline-based formulation allows for unified learning without hierarchical complexity.

The study highlights that while the system performs well on standard and moderately complex stairs, challenges remain at the highest difficulty levels (Level 6), where the policy tends to adopt conservative strategies to avoid penalties. The authors suggest that future work could incorporate exploration rewards to mitigate this risk-aversion. Additionally, the current work assumes intact stairs; future iterations aim to test the system in harsher environments with damaged or obstructed stairs, eventually deploying the learned policies on real hardware via Sim-to-Real workflows.