Technical Summary: Safe Execution of RL Policies via Acceleration-based CBF-QP

Problem Statement

Reinforcement Learning (RL) has demonstrated significant potential for solving complex robotic control tasks, such as agile locomotion and dexterous manipulation. However, the deployment of RL policies on physical hardware is severely limited by a lack of safety guarantees. While Safe RL methods exist that embed constraints during training (e.g., via Constrained Markov Decision Processes or Control Barrier Functions), they often rely on statistical guarantees that degrade under distributional shifts. Furthermore, strict safety constraints during training can be overly conservative, restricting exploration and slowing convergence. Crucially, existing methods often fail to ensure consistent constraint satisfaction for Out-of-Distribution (OoD) states encountered during real-world deployment, leading to hardware damage or system shutdowns.

The core challenge addressed is how to enforce explicit safety constraints (joint position, velocity, torque, and collision limits) on pre-trained RL policies at runtime without requiring retraining or modifying the learned policy itself.

Methodology: Acc-CBF-QP

The authors propose Acc-CBF-QP, an acceleration-based Quadratic Program (QP) safety filter that acts as a post-processing layer around any pre-trained RL policy. The framework operates by filtering the RL command in real-time to ensure it adheres to a safe set defined by Control Barrier Functions (CBFs).

Core Components

1. Acceleration-Based QP Formulation:
   The control problem is formulated as a QP where the decision variables are generalized accelerations ($\ddot{q}$). This allows for the simultaneous satisfaction of dynamic and kinematic constraints, which are often second-order in nature. The QP minimizes a weighted sum of task errors subject to linearized constraints:
   $$\ddot{q}^* = \arg\min_{\ddot{q}} \sum_k \|W_k(A_k \ddot{q} - b_k)\|^2 \quad \text{s.t.} \quad A_c \ddot{q} \leq b_c$$

2. Constraint Enforcement (CBFs):
   Safety is defined as the forward invariance of a closed set $\mathcal{C}$. The authors formulate constraints for:

   • Torque Limits: Enforced via inverse dynamics.
   • Joint Velocity and Position Limits: Enforced using standard CBFs for velocity (relative degree 1) and Exponential CBFs (ECBF) for position (relative degree 2).
   • Self-Collision Avoidance: Enforced using ECBFs on the distance between monitored body pairs.

3. RL Integration and Task Formulations:
   Since RL policies typically output actions interpreted as desired joint displacements (which translate to torques via a low-level PD controller), the QP must minimize deviation from the effective RL torque command. The paper introduces two specific task formulations to achieve this:

   • TorqueTask: Minimizes the deviation between the QP-computed torque and the desired RL torque ($\tau_{RL}$). This formulation is robust to modeling errors as it relies on inverse dynamics.
   • Forward Dynamics Task (FDTask): Minimizes the deviation between the QP-computed acceleration and the acceleration implied by the desired torque via forward dynamics. This formulation is more sensitive to model inaccuracies but theoretically minimizes acceleration deviation when the model is perfect.

4. Robustness and Disturbance Estimation:
   To handle modeling errors and external disturbances (critical for floating-base robots), the system employs a Generalized Momentum Observer (GMO) to estimate external torques ($\hat{\tau}_{ext}$). This estimate is used in the inverse dynamics calculation to compensate for unmodeled dynamics. The authors provide a robustness analysis showing that under bounded disturbances, the system maintains input-to-state safety, keeping trajectories within a neighborhood of the safe set.

5. Handling Infeasibility:
   In cases where the QP becomes infeasible (e.g., due to conflicting constraints or severe perturbations), the system employs fallback mechanisms: switching to a damping mode or rolling back to a torque-saturated RL command.

Key Contributions

1. Open-Source Framework: The authors release an open-source pipeline (Acc-CBF-QP) that enforces safety on robotics RL policies via acceleration-based QP with CBF constraints.
2. Dual Task Formulations: The introduction of TorqueTask and FDTask provides principled control over the safety-performance trade-off, allowing users to choose between minimizing torque deviation or acceleration deviation depending on the application and model fidelity.
3. Integration of State Estimation: The framework integrates state estimation and disturbance observers to recover external forces, which is essential for consistent CBF enforcement on torque-controlled and floating-base systems.
4. Runtime Safety Without Retraining: The method acts as a modular interface that wraps arbitrary RL policies, enforcing constraints at runtime without the need for additional policy retraining or modification of the training process.

Experimental Results

The framework was validated on two distinct platforms: a 7-DoF Kinova Gen3 manipulator and a 19-DoF Unitree H1 humanoid robot, in both simulation and on real hardware.

• Kinova Gen3 (Manipulator):

  • Using an off-the-shelf RL policy trained without safety constraints, Acc-CBF-QP fully eliminated constraint violations (joint position, velocity, torque, and collisions) in both simulation and hardware.
  • Nominal task performance (Cartesian trajectory tracking) was preserved, with deviations statistically indistinguishable from the intrinsic repeatability of the RL policy.

• Unitree H1 (Humanoid):

  • Compared against a Safe-RL baseline trained with the "Constraints as Terminations" (CaT) method.
  • Violation Reduction: On real hardware, the Safe-RL policy alone yielded 10.04 violations/second. Augmenting this with Acc-CBF-QP reduced violations by 92% to 0.80 violations/second (TorqueTask).
  • Survival Time: Under aggressive velocity commands, Acc-CBF-QP prevented constraint-induced shutdowns, resulting in longer survival times compared to the baseline.
  • Robustness: On uneven terrain, the QP layer shifted failure modes from "constraint violations" (e.g., joint limits) to "physically plausible falls," without statistically significant degradation in survival duration compared to the baseline.
  • Task Formulation Comparison: The TorqueTask demonstrated greater robustness to modeling errors on hardware, while the FDTask showed slightly better acceleration tracking in simulation.

Significance and Claims

The paper claims that Acc-CBF-QP constitutes a practical step toward the reliable real-world deployment of learned robotic behaviors. Its primary significance lies in bridging the gap between the flexibility of RL and the strict safety requirements of physical systems.

• Modularity: It provides a modular interface between learned control and acceleration-level safety filtering, allowing for dynamic modification of safety constraints at deployment without retraining.
• Safety vs. Performance: The experiments demonstrate that safety enforcement can be achieved with no measurable degradation in nominal task performance when constraints are not active.
• Real-World Applicability: By successfully deploying on a complex, floating-base humanoid (Unitree H1) with imperfect dynamic models, the work validates that optimization-based safety layers can effectively mitigate the risks associated with OoD states and model inaccuracies in real hardware.

The authors remain modest regarding limitations, noting that the framework currently handles QP infeasibility pragmatically (via damping or rollback) rather than through formal stability guarantees for the closed-loop projected system, and that broader comparisons with other Safe-RL training methods are left for future work.