Technical Summary: Shield-Loco

Problem Statement

Reinforcement learning (RL) policies have demonstrated success in enabling dynamic legged locomotion by mapping desired contact locations to low-level joint commands. However, these policies lack active mechanisms to prevent safety violations when encountering environmental constraints (e.g., obstacles) that were not present during training. While large-scale offline safe learning is impractical for covering all edge cases, existing safety frameworks face significant limitations:

• Reduced-order models: Methods like Control Barrier Functions (CBF) often rely on simplified models that cannot reason about whole-body behaviors, leading to failures under unmodeled contact dynamics.
• Conservative recovery: Model Predictive Shielding (MPS) and Hamilton-Jacobi (HJ) reachability approaches often resort to conservative recovery controllers that sacrifice task performance or require hand-crafted certificates that fail in complex scenarios.
• Scalability: HJ reachability scales exponentially with state dimension, making whole-body legged models intractable.

The core challenge is to intervene on the policy to ensure safety without degrading task performance or requiring a complete retraining of the RL policy.

Methodology

The authors propose Shield-Loco, a predictive safety filter that operates post-hoc on the contact locations fed to a contact-conditioned RL policy. The key insight is that foot contact locations serve as a kinematic abstraction sufficient for enforcing whole-body collision avoidance while remaining tractable for online optimization.

Framework Overview

At each control step, the filter receives a nominal sequence of contact locations from a high-level planner. It performs a forward rollout using a full-physics simulator (MuJoCo) and the frozen RL policy to predict future states.

1. Prediction: If no safety violation is predicted within the receding horizon, the nominal contact is passed to the policy.
2. Intervention: If a violation is predicted, a sampling-based optimizer searches for an alternative contact sequence that maximizes a finite-horizon return while satisfying safety constraints.
3. Execution: The first optimized contact location is fed to the RL policy, which generates joint torques. The process repeats asynchronously, allowing the low-level policy to run at high frequency (50 Hz) even if the optimizer runs at a lower frequency (3 Hz).

Optimization Formulation

The filter solves the following optimization problem when intervention is required:
$$\max_{p_{0}, \dots, p_{H-1}} \mathbb{E} \left[ \sum_{t=0}^{H-1} \gamma^t r(s_t, a_t) + \gamma^H V(s_H) \right]$$
Subject to safety constraints $g(s_t, p_t) \leq 0$, where $a_t = \pi(s_t, p_t)$ and $s_{t+1} = f(s_t, a_t)$.

• Reward ($r$): Encodes task performance (e.g., tracking desired contacts, energy efficiency).
• Value Function ($V$): A learned value function bootstraps long-horizon returns, trained offline via temporal-difference learning on trajectories collected by the optimizer.
• Safety Constraints ($g$):
  • Foot Contact Penalty: Soft penalties based on the signed distance function (SDF) between feet and obstacles.
  • Whole-Body Collision: A large penalty is applied if any robot link contacts an obstacle, detected via MuJoCo's contact data.

Algorithmic Components for Tractability

To make optimization feasible in a discontinuous contact landscape, the authors introduce three key components:

1. Geometric Projection: Sampled contact locations are projected onto the collision-free set via a Quadratic Program (QP) before rollout evaluation. This ensures candidates satisfy hard constraints locally, though the full rollout is still evaluated for safety.
2. Momentum-Augmented Updates: Derived from Accelerated Proximal Natural Gradient Descent, this update rule accelerates convergence by extrapolating in the direction of improvement, reducing the number of iterations required.
3. Replica Exchange (Parallel Tempering): To escape shallow local minima, multiple optimizer replicas with different inverse temperatures ($\lambda$) are maintained. Adjacent replicas swap contact sequences based on the Metropolis–Hastings criterion, allowing high-temperature replicas to explore broadly while low-temperature ones exploit promising solutions.

Key Contributions

1. Predictive Safety Filter on Contact Locations: A method that operates directly on the contact locations input to RL policies, enabling whole-body collision avoidance without modifying the underlying policy.
2. Tractable Sampling-Based Optimization: A novel optimization scheme incorporating geometric projection, momentum-augmented updates, and replica exchange to handle the non-smooth, discontinuous cost landscape of legged locomotion.
3. Validation in Simulation and Real World: Comprehensive evaluation on a Unitree Go2 quadruped in dense, cluttered environments, demonstrating substantial reductions in safety violations with minimal deviation from nominal gait.

Results

The method was evaluated on a Unitree Go2 quadruped in both MuJoCo simulation and real-world hardware experiments.

• Safety vs. Performance Trade-off: Compared to baselines (CBF and HJ reachability using unicycle approximations), Shield-Loco achieved significantly lower tracking costs and fewer actual safety violations. The reduced-order baselines struggled in dense, cluttered environments, whereas Shield-Loco maintained a strong safety-tracking trade-off.
• Optimizer Variants: Among the optimizers tested (MPPI, CEM, and Replica Exchange MPPI), the Replica Exchange variant achieved the lowest tracking cost (0.0086) while maintaining low violation rates, though it incurred higher optimization time.
• Ablation Studies:
  • Momentum: Non-zero momentum coefficients ($\beta$) consistently improved tracking costs, with $\beta=0.1$ yielding optimal performance.
  • Horizon and Iterations: A moderate computational budget ($N=3$ iterations, $H \in [4, 6]$) provided the best balance between safety and reactivity.
  • Projection: Applying geometric projection both during rollout sampling and on the final optimized plan was critical for minimizing violations.
• Real-World Deployment: The system successfully navigated cluttered environments with obstacles (phones, packages, cables) and large static obstacles. The asynchronous design allowed the robot to remain stable even when the optimizer took longer than the planning cycle (e.g., 400ms for $H=8$), as the low-level policy continued tracking the last known safe contact targets.

Significance and Limitations

The paper positions Shield-Loco as a minimally invasive safety intervention that bridges the gap between the expressiveness of full-physics models and the tractability required for online safety filtering. By operating on contact locations rather than joint torques, it avoids the "curse of dimensionality" while retaining the ability to reason about whole-body collisions.

Limitations and Future Work:

• Theoretical Guarantees: The method lacks formal theoretical guarantees of optimality or safety. The filtered plan may deviate from the true safe set if the optimizer converges to a local optimum.
• Heuristic Projection: The geometric projection is a heuristic and may not strictly preserve the optimizer's convergence properties.
• Policy Limitations: Actual violations in experiments were sometimes caused by the RL policy generating unsafe intermediate swing-phase configurations, even when the contact targets were safe.
• Scope: Validation is currently limited to quadrupeds on flat terrains. Extending to multi-terrain locomotion, humanoids, or scenarios requiring active torso twisting remains future work.
• Perception: The current framework assumes known obstacle geometry; integrating online perception pipelines for fully autonomous deployment in unseen environments is identified as a necessary future direction.