Technical Summary: PPO-EAL for Safe Robotic Control

Problem Statement

Reinforcement Learning (RL) has demonstrated significant potential in solving complex robotic control tasks, ranging from manipulation to locomotion. However, standard RL approaches often neglect feasibility constraints during policy updates, failing to satisfy critical safety requirements. While Safe RL aims to maximize task performance under explicit physical constraints using the Constrained Markov Decision Process (CMDP) framework, existing algorithms face two primary challenges:

1. Efficiency vs. Precision: First-order Lagrangian-based methods (e.g., PPO-Lagrangian) often suffer from slow multiplier updates relative to policy parameters, leading to oscillations or overshooting around cost boundaries.
2. Numerical Instability: Penalty-based methods require impractically large or infinite penalty factors to guarantee equivalence with the original constrained problem, often resulting in ill-conditioned optimization and numerical instability. Barrier-based methods introduce bias that can lead to suboptimal solutions.

The paper addresses the need for a first-order constrained policy optimization framework that achieves theoretically grounded constraint enforcement without requiring infinite penalty factors or suffering from unstable dual-variable dynamics.

Methodology: PPO-EAL

The authors propose PPO-EAL (Proximal Policy Optimization with Exact Augmented Lagrangian), a novel framework that integrates exact augmented Lagrangian optimization into the PPO algorithm.

Core Formulation

The method formulates the constrained problem using an Exact Augmented Lagrangian function ($L_{EAL}$):
$$L_{EAL}(\pi, \lambda) = J_R(\pi) - \sum_{i=1}^n \lambda_i \phi_i(\pi) - \frac{\beta}{2} \sum_{i=1}^n [\phi_i(\pi)]_+^2$$
Where:

• $J_R(\pi)$ is the reward objective.
• $\phi_i(\pi) = J_{C_i}(\pi) - b_i$ represents the constraint violation.
• $\lambda_i$ are the dual variables (Lagrange multipliers).
• $\beta$ is the quadratic penalty coefficient.
• $[\cdot]_+$ denotes the ReLU function.

Unlike previous approaches that may require infinite $\beta$ for exactness, PPO-EAL relies on finite-penalty exactness theory (Rockafellar augmented Lagrangian), ensuring mathematical equivalence to the original constrained problem without impractically large penalty factors.

Algorithmic Components

1. Clipped Surrogate Objective: Following standard PPO, the method employs clipped surrogate objectives for both reward and cost to avoid expensive trust-region optimization and prevent aggressive policy updates.
2. Dual-Variable Stability (Momentum Regulation): To address the oscillation issues common in primal-dual schemes, the authors introduce a momentum-regulated update for the Lagrange multipliers. This involves adding a derivative-type correction term based on the rate of change of the cost:
   $$\lambda'_i = \max \{0, \lambda_i + \alpha_\lambda \phi_i(\pi_\theta) + k_d (J_{C_i}(\pi_\theta) - J_{C_i}(\pi_\theta^{old}))\}$$
   This acts as a damping component, reducing transient oscillations in constraint satisfaction while preserving convergence properties.
3. Multi-Time-Scale Updates: The policy parameters and Lagrange multipliers are updated at different time scales to guarantee convergence under standard stochastic approximation assumptions.

Theoretical Guarantees

The paper provides rigorous analysis under standard regularity assumptions:

• Exactness: Theorem 1 establishes that for a sufficiently large finite penalty $\beta$, any stationary point of the augmented Lagrangian corresponds to a KKT point of the original constrained problem.
• Convergence: Theorem 2 proves that the sequence of iterates generated by PPO-EAL converges almost surely to an internally chain-transitive invariant set of the limiting projected primal-dual dynamics. If the approximation bias vanishes, the iterates converge to KKT points.

Key Contributions

1. Novel Algorithm: The proposal of PPO-EAL, which integrates Lagrangian duality with an exact quadratic penalty, achieving provable exactness and convergence without infinite penalty factors.
2. Implementation & Stability: The implementation of a safe RL pipeline under a deep actor-critic architecture using standard first-order optimizers, enhanced by a momentum-based modification for the dual update to reduce oscillations.
3. Extensive Validation: Comprehensive evaluation across diverse, increasingly complex robotic benchmarks (cart-pole, double pendulum, 7-DoF Franka manipulator, and quadrupedal locomotion) and a zero-shot sim-to-real deployment in a contact-rich gear assembly task.

Experimental Results

The authors evaluated PPO-EAL against state-of-the-art first-order safe RL baselines (PPO-Lagrangian, P3O, and APPO) and standard PPO.

Simulation Benchmarks

• Safety Precision: PPO-EAL (and its momentum-regulated variant, PPO-EAL-m) consistently satisfied safety constraints across all tasks. In contrast, standard PPO and some baselines frequently violated safety thresholds.
• Performance Trade-off: PPO-EAL-m achieved the highest task rewards while maintaining precise safety compliance. For instance, in the inverted cart-pole task, it achieved a reward of ~597.7 with a violation rate of ~0.99%, outperforming PPO-L which had lower rewards.
• Robustness: Sensitivity analyses showed that PPO-EAL-m is more robust to random seeds and varying safety thresholds compared to the non-momentum variant, particularly in maintaining torque constraints in quadrupedal locomotion.
• Penalty Factor Sensitivity: The method remained effective with moderate penalty factors ($N \le 20$), whereas excessively large factors ($N \ge 50$) led to constraint violations due to amplified estimation noise.

Real-World Deployment (Sim-to-Real)

The method was deployed on a physical robot for a contact-rich gear assembly task, a scenario requiring high precision and handling of contact forces.

• Success Rate: PPO-EAL achieved an 80% success rate in hardware tests, compared to only 30% for standard PPO.
• Safety Metrics: PPO-EAL reduced the peak contact force by approximately 46% compared to PPO (28.7 N vs. 53.4 N). The momentum-regulated variant (PPO-EAL-m) further reduced peak force to 21.5 N and peak torque.
• Generalization: PPO-EAL demonstrated robust generalization to new target shaft positions and gear masses (maintaining ~80% success), whereas standard PPO failed completely (0% success) in unseen configurations.

Significance and Claims

The paper positions PPO-EAL as a general and practically deployable safe RL framework for safety-critical robotic systems. The authors claim that by combining clipped policy updates with exact quadratic penalties and stabilized dual-variable updates, the method overcomes the trade-off between safety precision and task performance that plagues existing first-order methods.

The work highlights that PPO-EAL:

• Provides theoretically grounded constraint enforcement without the need for infinite penalty factors.
• Achieves superior safety precision and reward performance compared to state-of-the-art baselines in both simulation and real-world hardware.
• Enables zero-shot sim-to-real transfer in complex, contact-rich tasks, demonstrating practical applicability for systems where safety is paramount.

The authors conclude that while the current implementation relies on feedforward architectures and soft constraints, the framework establishes a strong foundation for safe robotic control, with future work potentially incorporating safety shielding and recurrent networks for richer temporal reasoning.