Quadruped Parkour Learning: Sparsely Gated Mixture of Experts with Visual Input

1. Problem Statement

Legged robots face significant challenges when navigating highly dynamic and discontinuous terrain, such as large steps, gaps, and uneven surfaces (parkour). While recent reinforcement learning (RL) approaches have achieved impressive results, they predominantly rely on sequential Multi-Layer Perceptron (MLP) architectures with densely activated layers.

• Limitations of Current Approaches: Standard MLPs require a large number of parameters to achieve high performance, leading to high computational costs during inference. Scaling these models to handle complex visual inputs (depth images) and diverse terrains often results in diminishing returns regarding efficiency.
• The Gap: There is a lack of research applying Sparsely Gated Mixture-of-Experts (MoE) architectures—proven effective in Large Language Models (LLMs) for scaling performance with conditional computation—to the domain of vision-based robotic locomotion.

2. Methodology

The authors propose a novel control framework that integrates a Sparsely Gated MoE actor into a vision-based RL pipeline for a Unitree Go2 quadruped robot.

A. Two-Phase Training Strategy

To bridge the sim-to-real gap and handle complex observations, the system employs a two-phase training approach:

1. Phase 1 (Simulation with Privileged Information):
   • Observations: The agent receives privileged information (ground truth center of mass, friction coefficients, motor strengths, linear velocity) alongside proprioception and a "scandot" heightmap.
   • Architecture: The actor uses a standard MLP to process inputs.
   • Goal: Learn a robust policy using full environmental knowledge.
2. Phase 2 (Distillation to Real-World Deployment):
   • Observations: Privileged information is replaced with estimates derived from proprioceptive history (using MLPs) and depth images processed by a Gated Recurrent Unit (GRU).
   • Teacher-Student Distillation: The Phase 1 policy acts as the "teacher." The Phase 2 "student" policy is trained to mimic the teacher's actions using Mean Squared Error (MSE) loss, while also minimizing yaw error.
   • Depth Pipeline: Depth images undergo rigorous preprocessing (clipping, contour detection, artifact injection, blurring, and downsampling) to simulate real-world sensor noise and bridge the sim-to-real gap.

B. Sparsely Gated Mixture-of-Experts (MoE) Architecture

The core innovation is the replacement of the standard MLP actor with an MoE layer:

• Structure: The actor consists of three layers: Linear $\rightarrow$ MoE Layer $\rightarrow$ Linear.
• Sparsity: The MoE layer contains $N=16$ expert networks. During inference, a gating mechanism selects only the top-$K=4$ experts to activate for a given input.
• Conditional Computation: This allows the model to have a large total parameter count ($1.6 \times 10^6$) while only activating a fraction ($0.8 \times 10^6$) at inference time, significantly improving efficiency.
• Loss Function: A load balancing loss is added to the PPO objective to prevent "expert collapse" (where only a few experts are used), encouraging diversity in expert utilization.

C. Domain Randomization

To ensure robustness, the training environment randomizes:

• Physical properties (mass, friction, motor strength).
• Sensor noise (Gaussian noise on joint states, boolean contact flipping).
• Depth camera artifacts (noise, blur, contour errors).

3. Key Contributions

1. Novel Framework: The first application of sparsely gated MoE architectures specifically for vision-based quadruped parkour, leveraging conditional computation to scale performance.
2. Vision Integration: Successful incorporation of raw depth camera data into the MoE policy via a GRU encoder and a robust preprocessing pipeline.
3. Real-World Deployment: A robust deployment on the Unitree Go2 robot, demonstrating successful traversal of obstacles up to 80% of the robot's height in both laboratory and outdoor (skate park) environments.
4. Analysis of Expert Specialization: Evidence that different experts specialize in different phases of the locomotion cycle (cyclical specialization) or specific sensory inputs (e.g., depth vs. proprioception).

4. Experimental Results

Experiments were conducted on a 32cm box obstacle (80% of robot height) using a Unitree Go2 robot.

• Performance vs. Standard MLP:
  • The MoE policy achieved a 100% success rate (10/10 trials).
  • A standard MLP with the same number of active parameters (Medium/Large) achieved only 20-50% success.
  • To match the MoE's performance, a standard MLP required scaling up to the total parameter count of the MoE (Extra-Large), achieving 90% success but with more instability ("scrambling").
• Computational Efficiency:
  • The MoE policy was 14.3% faster in inference time compared to the Extra-Large MLP (which has the same total parameters).
  • This demonstrates that MoE provides a superior trade-off between performance and computational cost.
• Expert Analysis:
  • Cyclical Specialization: Experts 5, 6, and 8 showed strong cyclical weighting patterns corresponding to the gait cycle.
  • Input Specialization: Expert 7 was uniquely sensitive to the depth latent but utilized less frequently, suggesting specific experts handle visual depth perception while others handle proprioception or dynamics.
• Ablation Studies:
  • Removing observation noise or Regularized Online Adaptation (ROA) significantly degraded performance, confirming both are critical for the sim-to-real transfer.

5. Significance and Conclusion

This paper establishes that Sparsely Gated MoE architectures are highly effective for legged robot control, particularly in vision-based parkour scenarios.

• Efficiency: MoE allows for "scaling up" the model's capacity (total parameters) without incurring the linear computational cost of dense MLPs during inference.
• Robustness: The approach enables robots to traverse challenging, discontinuous terrain with higher success rates and stability than traditional dense networks of comparable inference cost.
• Future Directions: The work suggests that future research could explicitly design expert architectures to specialize in specific obstacles or gait phases, further optimizing the "organic" specialization observed in this study.

In summary, the authors successfully demonstrate that conditional computation via MoE is a viable and superior paradigm for scaling control policies in complex, vision-driven robotic locomotion tasks.