Mixture-of-Experts RL for Fault-Tolerant Legged Locomotion

Technical Summary: Mixture-of-Experts RL for Fault-Tolerant Legged Locomotion

Problem Statement
Legged robots are increasingly vital for planetary exploration and remote operations due to their ability to traverse unstructured terrain. However, long-duration deployment in these environments exposes hardware to degradation and actuator failures, which can severely compromise mobility. While Reinforcement Learning (RL) has demonstrated robustness in legged locomotion, conventional monolithic policies struggle to efficiently represent the diverse control strategies required for different fault conditions. A single network attempting to encode terrain perception, standard locomotion, and multiple distinct fault-recovery behaviors often suffers from representational interference and reduced learning efficiency, particularly as task complexity increases.

Methodology
The authors propose a fault-aware modular control architecture that explicitly leverages fault-diagnosis information to activate specialized control experts. Unlike standard Mixture-of-Experts (MoE) approaches that rely on a learned gating network to infer the operating mode, this framework uses explicit fault information to select the appropriate expert.

The control pipeline consists of three main components:

1. Fault-Aware Mixture-of-Experts Policy: The policy network utilizes a shared encoder to process proprioceptive observations (base linear/angular velocities, gravity projection, joint positions/velocities) and visual terrain heightmaps. Instead of a learned routing mechanism, the system employs a one-hot selection mechanism based on the diagnosed fault condition $k$. This activates a specific expert $\pi_k$ tailored to that failure mode while keeping other experts inactive. This design reduces active inference cost ($C_{active} = C_{enc} + C_k$) compared to dense routing ($C_{dense} = C_{enc} + \sum C_k$).
2. State Estimation: To handle the breakdown of model-based assumptions during severe failures (e.g., body dragging), the framework employs a learning-based state estimator. This neural network predicts base linear velocity directly from sensor observations (including IMU acceleration) without relying on explicit contact modeling, accommodating dynamic changes induced by actuator failures.
3. Fault Detection: A dedicated neural network ($f_{fault}$) is trained as a supervised learning problem to identify the current actuator failure condition from a temporal window of proprioceptive history. This module provides the categorical distribution of fault states used to trigger the corresponding expert.

The system was trained using Proximal Policy Optimization (PPO) within the IsaacLab framework on a Unitree Go2 robot. The reward function includes task-oriented objectives (velocity tracking, base height/orientation) and regularization terms (joint energy, smoothness). Crucially, specific reward terms are dynamically activated or disabled based on the number of failed legs ($n_f$) to adapt the learning objective to the severity of the fault.

Key Contributions

• Fault-Aware MoE Policy: The proposal of a modular RL framework where specialized experts are explicitly conditioned on diagnosed actuator failures, eliminating the need for a learned routing function and enabling conditional computation.
• Efficiency under Constraints: An evaluation demonstrating that modular policies maintain competitive locomotion performance even with significantly reduced network capacity (Small variants), outperforming monolithic baselines of comparable size.
• Reproducibility: The release of the associated code to provide a reproducible baseline for the community.

Results
Experiments were conducted in simulation and on a real Unitree Go2 robot across four scenarios: No Fail, Left-Front (LF) Fail, Left-Hind (LH) Fail, and simultaneous Left-Hind + Right-Hind (LH+RH) Fail.

• Performance Comparison: The proposed MoE architecture consistently outperformed monolithic PPO baselines across all fault scenarios. While performance was comparable in nominal conditions, the advantage of the MoE approach became progressively more evident as fault severity increased. In the most severe scenario (LH+RH Fail), where locomotion requires dragging behaviors distinct from standard trotting, the MoE policy achieved substantially higher cumulative rewards.
• Network Capacity Sensitivity: When network capacity was reduced (from Large to Small architectures), the monolithic baseline suffered significant performance degradation. In contrast, the MoE architecture retained robust performance, with its "Small" variant achieving results comparable to the "Large" monolithic baseline. This suggests that structured specialization allows for effective learning even with fewer active parameters.
• Real-World Validation: The framework was successfully deployed on a physical robot, demonstrating successful traversal of rocky terrain in nominal conditions, three-legged locomotion (LH Fail), and two-legged dragging locomotion (LH+RH Fail).

Significance and Limitations
The paper claims that explicit fault-conditioned specialization offers a superior alternative to simply increasing model size for fault-tolerant locomotion. The architecture is particularly significant for compute-constrained robotic platforms, such as those used in space exploration, where onboard resources are limited. By decoupling fault-specific behaviors into dedicated experts, the system achieves high robustness without the computational overhead of a massive monolithic network.

The authors modestly acknowledge a limitation regarding sample efficiency. Because expert networks do not share all parameters, policy-gradient estimates are noisier, as each expert is updated only on trajectories from its specific fault condition. While the current experiments (using ~4,096 agents per fault condition) were sufficient, scaling to a larger number of fault conditions may require additional simulation environments or more sophisticated parameter-sharing strategies.