Safe and Optimal Variable Impedance Control via Certified Reinforcement Learning

1. Problem Statement

Robots operating in dynamic, unstructured environments require Variable Impedance Control (VIC) to safely interact with humans and objects by dynamically adjusting stiffness and damping. While Reinforcement Learning (RL) offers a powerful model-free approach to learn complex VIC policies (often combined with Dynamic Movement Primitives, or DMPs), it faces critical safety challenges:

• Instability: Standard RL exploration (e.g., Gaussian noise) can sample gain schedules that violate Lyapunov stability conditions, leading to oscillatory or divergent behaviors.
• Actuator Limits: Unconstrained sampling may result in torque commands that exceed physical actuator limits.
• Post-hoc Fixes: Existing safety methods (e.g., reward penalties or safety filters applied after learning) do not guarantee stability during the learning process and often degrade performance or fail to handle time-varying gains effectively.

The core problem is how to perform model-free policy optimization for VIC while mathematically guaranteeing that every explored policy is stable and physically realizable, without relying on expensive model-based constraints or post-hoc corrections.

2. Methodology: Certified Gaussian-Manifold Sampling (C-GMS)

The authors propose C-GMS, a trajectory-centric RL framework that embeds stability constraints directly into the policy sampling process. Instead of sampling gains from a generic Gaussian distribution and checking for safety later, C-GMS restricts sampling to a certified manifold where stability is guaranteed by construction.

Key Components:

• Policy Representation:
  • Trajectory: Parameterized using Dynamic Movement Primitives (DMPs) with Radial Basis Functions (RBFs).
  • Impedance Gains: Time-varying stiffness $K(t)$ and damping $D(t)$ are parameterized via slack variables ($S_D, S_K$) rather than direct sampling.
• Stability Certification (Lyapunov Conditions):
  • The framework enforces the stability conditions derived by Kronander & Billard [11] for time-varying impedance:
    1. $\alpha H - D(t) \preceq 0$
    2. $\dot{K}(t) + \alpha \dot{D}(t) - 2\alpha K(t) \preceq 0$
  • These inequalities are satisfied by defining the gains through matrix-valued slack variables:
    • $D(t) = \alpha H + S_D(t)S_D(t)^\top$
    • $\dot{K}(t) + \alpha \dot{D}(t) - 2\alpha K(t) = -S_K(t)S_K(t)^\top$
  • The slack variables are sampled from a Gaussian distribution, but the resulting $K(t)$ and $D(t)$ are analytically constructed to satisfy the Lyapunov inequalities, ensuring Global Uniform Stability.
• Actuator Limit Awareness (Torque Governor):
  • To handle torque saturation, the method introduces a scaling factor $\beta \in [0, 1]$ that uniformly scales the slack variables.
  • An analytical calculation determines the maximum admissible $\beta^*$ such that the resulting torque command $\tau(\beta)$ remains within actuator limits ($\tau_{min} \leq \tau \leq \tau_{max}$).
  • Crucially, scaling by $\beta$ preserves the Lyapunov structure, ensuring stability is maintained even when torque limits are active.
• Learning Algorithm:
  • The framework utilizes Path Integral (PI2) reinforcement learning.
  • Policy updates are computed via Monte Carlo rollouts, but every rollout is sampled from the certified manifold.
  • This eliminates the need for reward penalties for instability or safety filters, as unsafe trajectories are mathematically impossible to generate.

Theoretical Guarantee:

The paper provides a theorem proving that under the C-GMS framework, the tracking error remains Uniformly Ultimately Bounded (UUB) even in the presence of bounded model errors, sensor noise, and feedforward uncertainties. The error bound decreases as the stability margins increase.

3. Key Contributions

1. C-GMS Framework: A novel RL method that unifies model-free policy search with rigorous stability analysis by constraining exploration to a Lyapunov-certified manifold.
2. By-Construction Safety: Unlike prior works that penalize instability, C-GMS ensures every policy rollout is stable and physically realizable, removing the need for post-hoc validation.
3. Actuator-Feasible Exploration: Introduction of a "torque governor" that scales gains to respect actuator limits without violating stability certificates.
4. Robustness Proof: Formal proof of bounded tracking error under model mismatch and external disturbances.
5. Real-World Validation: Successful deployment on a Franka Research 3 (FR3) robot for a human-robot handover task, demonstrating safe adaptation to obstacles.

4. Experimental Results

The method was evaluated in simulation (MuJoCo) and on a real 7-DoF Franka robot performing a collaborative handover task with a human.

• Scenario: The robot must reach a goal, avoid an obstacle via a via-point, and hand over an object.
• Comparison (C-GMS vs. Unconstrained):
  • With C-GMS: The robot learned a smooth, collision-free trajectory. Impedance gains adapted locally (stiffness increased near the via-point) and remained stable throughout. The eigenvalues of the stability condition remained strictly negative.
  • Without C-GMS: While the cost function (tracking error) converged, the learned policy violated Lyapunov conditions. This led to unstable oscillations, divergent gains, and collisions with the environment/human.
• Hardware Metrics:
  • The C-GMS policy achieved low RMSE (Root Mean Square Error) across five diverse scenarios.
  • The Torque Governor successfully prevented torque saturation in scenarios where the unconstrained approach would have exceeded limits (even with virtual torque limits reduced by 50%).
• Learning Curves: C-GMS showed steady cost reduction with guaranteed stability at every iteration, whereas the unconstrained method showed cost reduction but eventual instability.

5. Significance and Limitations

Significance:

• This work bridges the gap between data-driven learning and formal safety guarantees. It demonstrates that complex, compliant behaviors can be learned safely without relying on accurate dynamic models (which are hard to obtain) or conservative safety filters that hinder learning.
• It provides a practical solution for safe human-robot interaction, ensuring that learning robots do not harm humans or themselves during the exploration phase.

Limitations:

• Free-Space Assumption: The current stability criterion is formulated for free-space dynamics. It does not explicitly account for contact-rich tasks (e.g., pushing, sliding) where external forces $f_e$ are significant and time-varying in a way that challenges the current certificate.
• Cost Function Scope: The current implementation focuses on via-point tracking and does not include orientation control terms, limiting its application to full 6-DoF manipulation tasks.
• Future Work: The authors plan to extend the framework to contact-rich tasks and orientation-sensitive interactions, and to adapt the governor for varying payload conditions.

In conclusion, C-GMS represents a significant step toward reliable autonomous interaction, proving that safety can be an intrinsic property of the learning algorithm rather than an external add-on.