Simulator Adaptation for Sim-to-Real Learning of Legged Locomotion via Proprioceptive Distribution Matching

1. Problem Statement

The core challenge addressed is the Sim-to-Real gap in legged locomotion. Policies trained in simulation often fail on real hardware due to dynamics discrepancies (e.g., friction, motor latency, unmodeled actuator effects).

Traditional solutions face significant limitations:

• Dynamics Randomization (DR): While robust, it does not close the gap but merely makes policies robust to it. It often leads to conservative behaviors and struggles with complex, non-parametric dynamics.
• System Identification (State Matching): Existing methods typically require time-aligned trajectory matching between simulation and hardware using privileged state information (e.g., base pose/velocity from motion capture). This is impractical for legged robots because:
  • Legged systems are unstable; small initial errors cause trajectories to diverge rapidly.
  • Motion capture infrastructure is expensive and not always available.
  • Precise synchronization and reset mechanisms are difficult to implement on physical hardware.

Goal: Develop a practical method to adapt the simulator to match real-world hardware behavior using only proprioceptive data (joint positions, velocities, and actions) without requiring time alignment, motion capture, or privileged state information.

2. Methodology

The authors propose a Distributional Sim-to-Real framework that treats simulation and hardware rollouts as statistical distributions rather than time-series sequences.

A. The Metric: Proprioceptive Distribution Matching

Instead of minimizing point-wise errors (MSE) between aligned trajectories, the method minimizes the discrepancy between the stationary distributions of proprioceptive observations.

• Input: Distributions of joint positions, joint velocities, and policy actions collected from hardware and simulation.
• Cost Function: The Wasserstein Distance (Earth Mover's Distance).
  • To avoid the computational cost of high-dimensional optimal transport, the authors use a 1D Marginal Approximation. They compute the 1D Wasserstein distance for each joint dimension independently and average them.
  • Justification: Simulator modifications primarily affect marginal distributions of joint states more than inter-joint correlations. This approach is computationally efficient, sample-efficient, and robust to trajectory divergence.
• Advantage: No time alignment is needed. The metric compares the "shape" of the behavior (gait patterns, contact statistics, actuator lag) regardless of when specific events occur.

B. Simulator Adaptation Strategies

The framework optimizes a parameterized function to modify the simulator dynamics to minimize the distributional cost. Three modification types are evaluated:

1. Static Parameter Identification (FricArm): Adjusts fixed physics parameters (e.g., joint friction, armature inertia).
2. Action-Delta Model: A neural network that outputs a residual action added to the policy's output, correcting state-dependent actuation errors.
3. Residual Actuator Model: A neural network that outputs a residual torque added directly to the PD controller output. This operates at the simulation rate (1kHz), offering higher expressivity to capture unmodeled motor dynamics and torque saturation.

C. Optimization Loop

1. Data Collection: Run a policy $\pi_0$ on hardware to collect proprioceptive trajectories ($D_{hw}$). Run $\pi_0$ in the simulator to get $D_{sim}$.
2. Optimization: Use CMA-ES (Covariance Matrix Adaptation Evolution Strategy) to find simulator parameters $\theta$ that minimize the Wasserstein cost between $D_{sim}(\theta)$ and $D_{hw}$.
3. Fine-tuning: Retrain the policy in the adapted simulator to obtain the final controller $\pi_1$.

3. Key Contributions

1. Distributional Metric: Introduction of a sim-to-real cost function based on 1D Wasserstein distances of proprioceptive data, eliminating the need for motion capture or time-aligned trajectories.
2. Practical Adaptation Pipeline: A complete framework that successfully adapts simulators using only local sensor data, making it deployable on standard legged robots without external infrastructure.
3. Comprehensive Evaluation: Systematic comparison across three axes:
   • Scenarios: Idealized parameter shifts, complex non-parametric changes (springs), and challenging real-world tasks (bipedal walking).
   • Metrics: Comparison against privileged state-matching baselines.
   • Models: Evaluation of static parameters vs. learned neural residual models.
4. Real-World Validation: Demonstration of substantial drift reduction on a Unitree Go2 quadruped using less than 5 minutes of hardware data.

4. Experimental Results

Sim-to-Sim (Validation)

• Parameter Recovery: In idealized scenarios (recovering known friction/armature), the Wasserstein metric achieved accuracy comparable to privileged state-matching baselines (MatchS).
• Robustness to Noise: Under noisy conditions (simulating latency and sensor noise), the Wasserstein metric significantly outperformed non-privileged state matching (MatchO). At high noise levels, MatchO failed (86% error), while Wasserstein maintained low error (~18%).
• Complex Dynamics: For the "Spring Joint" scenario (non-parametric dynamics), static parameter identification failed completely. However, the Residual Actuator Model combined with the Wasserstein metric successfully learned the complex dynamics, matching the performance of privileged baselines.

Sim-to-Real (Hardware Experiments)

• Spring Joint Experiment: A physical spring was attached to the Go2's leg.
  • Result: The original policy failed with >2m drift. The adapted policy (using the Residual Actuator model) reduced drift to ~0.4m (20% error), successfully generalizing to unseen speeds.
• Bipedal Walking: The Go2 was trained to walk on two legs.
  • Result: The original policy exhibited significant drift on hardware. The adapted policy reduced drift by up to 50% in the most challenging directions.
• Efficiency: The entire adaptation process required only ~5 minutes of hardware data collection.

Limitations (Humanoid Case Study)

The authors attempted to apply this to the Agility Robotics Digit humanoid. The method failed when the initial policy caused the robot to fall (backward walking).

• Reason: The optimization assumes the initial transfer is "informative." If the robot falls immediately, the hardware data distribution does not cover the failure modes, leading the optimizer to learn degenerate solutions that match the failure distribution but do not fix the underlying dynamics.

5. Significance and Impact

• Democratization of Sim-to-Real: By removing the dependency on motion capture and privileged state information, this method makes advanced sim-to-real adaptation accessible to a wider range of robotics labs and applications.
• Handling Instability: The distributional approach is uniquely suited for unstable systems (like legged robots) where time-aligned trajectory matching is mathematically fragile due to sensitivity to initial conditions.
• Practicality: The ability to achieve significant performance gains with minimal hardware data (<5 mins) and no external sensing infrastructure represents a major step toward deploying learned locomotion policies on real-world robots.
• Future Direction: The paper suggests future work on learning stochastic dynamics (distributions of parameters) rather than fixed modifications to better capture real-world noise and variability.