TOLEBI: Learning Fault-Tolerant Bipedal Locomotion via Online Status Estimation and Fallibility Rewards

This paper introduces TOLEBI, a novel learning-based framework that enables fault-tolerant bipedal locomotion by training policies on simulated hardware faults and integrating an online joint status estimation module to classify and adapt to real-time joint conditions during sim-to-real transfer.

The Gist

Imagine teaching a robot to walk like a human. It's already hard enough to get a robot to take a few steps without falling over. But what happens if, while it's walking, one of its legs suddenly locks up like a rusty hinge, or its battery dies and the leg goes limp? In the real world, robots face these "hardware heart attacks" all the time. If they can't handle it, they crash, break, and become useless.

This paper introduces TOLEBI, a new way of teaching robots to keep walking even when their body parts start failing. Think of TOLEBI not just as a walking robot, but as a robotic gymnast who has trained specifically for the moment their leg gives out.

Here is how it works, broken down into simple concepts:

1. The "Chaos Training Camp" (Curriculum Learning)

Usually, when you train a robot, you let it practice walking on a perfect, flat floor until it gets good. Then you might add some wind or bumps.

TOLEBI does something different. It uses a "curriculum," which is like a video game that gets harder only when you're ready.

• Level 1: The robot learns to walk normally on a flat floor.
• Level 2: Once it's good at Level 1, the trainer (the computer) starts randomly "breaking" the robot's joints in the simulation. Sometimes a hip locks up; sometimes a knee loses power.
• Level 3: The robot learns to walk while broken. It learns that if its left leg is stuck, it has to shuffle differently to stay upright.

If they tried to break the robot on Day 1, it would just fall over and give up. By starting easy and getting harder, the robot builds "muscle memory" for disaster.

2. The "Internal Doctor" (Online Status Estimator)

In the real world, a robot doesn't have a dashboard that says "ERROR: LEFT HIP LOCKED." It just feels weird.

TOLEBI gives the robot a built-in internal doctor. This is a small AI program that constantly checks the robot's own body sensors (proprioception).

• If the robot tries to move a joint and it doesn't move, or if the motor is silent when it should be humming, the "doctor" instantly diagnoses the problem.
• It tells the main brain: "Hey, your right ankle is dead. Adjust your plan immediately!"
• This happens in real-time, without needing to stop and reboot.

3. The "Soft Landing" Strategy (Fallibility Rewards)

This is the cleverest part. When a robot's leg fails, it often tries to force itself to walk normally, which causes it to slam its foot down hard, leading to a fall.

The researchers designed a special "reward system" (like giving points for good behavior) that teaches the robot to be gentle.

• The Analogy: Imagine you are walking and suddenly your shoe sole falls off. A normal person might stomp and trip. A smart person would lift their foot higher and place it down softly to avoid hurting their foot or losing balance.
• TOLEBI rewards the robot for not slamming its feet. If the robot detects a broken joint, it learns to change its gait (how it walks) to land softly, reducing the impact force. This prevents the "crash" that usually happens when a robot tries to walk with a broken leg.

4. The "Phase Shifter" (Adaptive Timing)

When a leg is broken, the robot can't just keep the same rhythm.

• The Analogy: Imagine you are dancing with a partner, but suddenly your partner freezes. You can't keep dancing the same steps; you have to pause, wait, or change the rhythm to avoid tripping.
• TOLEBI has a "phase shifter" that allows the robot to speed up or slow down its walking cycle instantly. If one leg is stuck, the robot shortens the time that leg spends on the ground, effectively "hopping" or shuffling to keep moving forward without falling.

The Result: From Simulation to Reality

The team tested this on a real humanoid robot named TOCABI.

• They trained it in a super-accurate video game (simulation) where they broke its legs thousands of times.
• Then, they put the "brain" they learned in the game onto the real robot.
• The Test: They made the real robot walk on flat ground and even walk down stairs while one of its joints was locked or dead.
• The Outcome: The robot didn't fall. It adjusted its balance, changed its walking style, and kept going.

Why This Matters

Before this, if a robot's leg broke, it was usually game over. This paper shows that we can teach robots to be resilient. Just like a human who can hop on one foot if they twist an ankle, TOLEBI teaches robots to adapt to their own failures, making them safer and more useful for real-world jobs where things can go wrong.

Technical Summary

Here is a detailed technical summary of the paper "TOLEBI: Learning Fault-Tolerant Bipedal Locomotion via Online Status Estimation and Fallibility Rewards."

1. Problem Statement

While Reinforcement Learning (RL) has achieved significant success in bipedal locomotion, most existing methods assume ideal hardware conditions. In real-world deployments, robots face unexpected hardware faults (e.g., joint locking, power loss) and environmental disturbances.

• The Challenge: Bipedal robots are inherently less stable than quadrupeds; a failure in a single leg often leads to catastrophic falls.
• Limitations of Current Approaches:
  • Model-based methods: Rely on explicit fault modeling, which is difficult to scale to unseen failure modes or unstructured environments.
  • Existing Learning-based methods: Primarily focus on quadrupeds or assume faults are known. They often lack mechanisms to handle the "black-box" nature of RL when faults occur during runtime, leading to a lack of robustness in real-world scenarios.

2. Methodology: The TOLEBI Framework

The authors propose TOLEBI (a faulT-tOlerant Learning framEwork for Bipedal locomotIon), a reinforcement learning framework designed to handle motor failures through simulation-to-real (Sim2Real) transfer.

A. Core Components

1. Online Joint Status Estimator:

   • A single-layer Gated Recurrent Unit (GRU) trained concurrently with the policy.
   • Input: Proprioceptive observations (joint positions, velocities, base orientation).
   • Output: A binary status vector indicating the health of each of the 12 leg joints (Healthy vs. Faulty).
   • Function: This estimator runs online during inference, allowing the policy to adapt its control strategy based on real-time fault detection without requiring pre-labeled fault data at runtime.

2. Curriculum Learning Strategy:

   • Training is structured in phases to prevent instability:
     • Phase 1: Nominal locomotion (no faults) to learn basic walking skills.
     • Phase 2: Introduction of motor failure simulations (Joint Locking and Power Loss) once the agent achieves stability (>20s).
     • Phase 3: Introduction of external push perturbations once the agent handles faults (>24s).
   • This progressive exposure ensures the policy learns robust recovery strategies rather than failing immediately upon encountering faults.

3. Action Space & Phase Modulation:

   • The action space includes 12 joint torque commands and a phase modulation action ($a_{\delta\phi}$).
   • The phase modulation allows the robot to dynamically adjust its gait timing (e.g., shortening the stance phase of an impaired leg) to maintain balance under partial actuation.

4. Reward Function Design:

   • Task Rewards: Velocity tracking, foot contact synchronization.
   • Regulation Terms: Penalizing excessive joint torque, velocity, and body orientation deviations.
   • Fallibility Rewards (Novelty):
     • Trajectory Mimicking: Encourages the robot to follow nominal joint trajectories even when faulty, preventing the adoption of overly conservative (crouching) gaits.
     • Contact Force Tracking: Specifically penalizes high impact forces during early foot contact caused by faults, mitigating the risk of falling due to impulsive forces.

B. Simulation & Training

• Environment: Isaac Gym with 4,096 parallel environments.
• Algorithm: Proximal Policy Optimization (PPO).
• Fault Simulation:
  • Joint Locking: Torque is computed via PD control to hold the joint at the failure position.
  • Power Loss: Torque command is set to zero.
• Sim2Real Transfer: Achieved via Domain Randomization (randomizing velocities, pushes, noise) and Dynamics Randomization (mass, inertia, friction, motor constants).

3. Key Contributions

1. First Learning-Based Fault-Tolerant Bipedal Framework: TOLEBI is the first work to present a learning-based approach specifically for fault-tolerant bipedal locomotion in real-world environments, addressing a gap left by quadruped-focused research.
2. Online Joint Status Estimation: Integrates a GRU-based estimator trained online to infer joint health from proprioception, enabling the policy to react to unknown faults in real-time.
3. Fallibility Rewards: Introduces a novel reward structure that balances maintaining nominal gait patterns with mitigating impact forces, preventing the policy from learning "safe but useless" gaits (like crouching).
4. Curriculum Learning for Faults: Demonstrates that gradual exposure to faults and perturbations is essential for stabilizing training in complex bipedal tasks.

4. Experimental Results

The framework was validated on the humanoid robot TOCABI in both simulation and real-world scenarios (flat walking and stair descent).

• Simulation Performance (Success Rates):

  • Baseline (Standard RL): Failed in most fault scenarios (e.g., 0% success for hip roll locking).
  • TOLEBI (Full Approach): Achieved an average success rate of 81.27% for joint locking and 52.67% for power loss.
  • Notably, it achieved 99.51% success for ankle roll locking and 97.53% for hip yaw power loss.

• Ablation Study:

  • Removing the Joint Status Observation caused a massive performance drop, proving online estimation is critical.
  • Removing Fallibility Rewards increased tracking errors and reduced robustness to impact forces.
  • Removing Phase Modulation resulted in the highest Mean Bias Error (MBE), as the robot could not adapt gait timing.
  • Removing Curriculum Learning prevented the robot from learning basic walking, leading to failure even in healthy conditions.

• Real-World Validation:

  • Flat Walking: The robot successfully maintained stable linear and angular velocity tracking ($v_x = 0.3$ m/s) under joint locking and power loss.
  • Stair Descent: The robot successfully descended 9cm stairs with faults, despite no specific stair-training curriculum, demonstrating strong generalization.
  • Force Mitigation: The fallibility reward successfully reduced impulsive contact forces (which could reach 2000N without the reward) to safe levels.

5. Significance

This work represents a significant step toward deploying humanoid robots in unstructured, real-world environments where hardware failures are inevitable. By moving beyond "perfect simulation" assumptions, TOLEBI demonstrates that:

• RL policies can be made robust to hardware faults without explicit fault modeling.
• Online status estimation allows for adaptive control strategies that are superior to static fault-tolerance methods.
• The proposed framework bridges the gap between theoretical RL and practical, resilient robotic deployment, paving the way for more reliable humanoid assistants and service robots.