Technical Summary: From Learning to Safety: A Direct Data-Driven Framework for Constrained Control

1. Problem Statement

The paper addresses the critical challenge of ensuring safety (constraint satisfaction) in learning-based control systems, particularly in model-free settings where explicit system dynamics are unavailable. While traditional safety filters (e.g., Control Barrier Functions or CBFs) effectively enforce safety in model-based contexts by modifying control inputs, they typically rely on predictive models derived from physics or data. This reliance creates a gap: existing data-driven safety filters are mostly "indirect," requiring system identification first, which compounds errors and degrades safety performance. Furthermore, "direct" data-driven methods are often limited to linear constraints or lack formal safety guarantees.

The core problem is to design a control framework that:

1. Operates directly from state-transition data without explicit system identification.
2. Integrates seamlessly with arbitrary model-free learning algorithms (e.g., Reinforcement Learning, Supervised Learning) to optimize performance.
3. Provides formal guarantees on safety and recursive feasibility, even in the presence of learning inaccuracies.

2. Methodology

The proposed solution is an optimization-based control framework that decouples performance optimization from safety enforcement. The framework relies on a novel safety certificate called the State-Action Control Barrier Function (SACBF).

A. Core Concept: State-Action Control Barrier Function (SACBF)

Unlike classical CBFs, which evaluate safety solely based on the state $x$, the SACBF, denoted as $Q_B(x, u)$, is a function of both the state and the control input.

• Definition: A function $Q_B$ is an SACBF if its zero sub-level set defines a safe set $S_Q$ such that for any state in $S_Q$, there exists an input satisfying $Q_B(x, u) \leq 0$ that keeps the next state within $S_Q$.
• Advantage: This formulation allows the safety filter to assess the safety of a candidate control action directly at the current state-action pair without propagating dynamics forward or requiring a model $f(x, u)$.

B. The Control Framework

The control policy $\pi(x)$ is defined as the solution to a constrained optimization problem:
$$\pi(x) := \arg \min_{u \in U} Q(x, u) \quad \text{s.t.} \quad Q_B(x, u) \leq 0$$
Here, $Q(x, u)$ is an objective function (learned via RL or SL) representing task performance, and $Q_B(x, u)$ is the learned SACBF acting as the safety constraint.

C. Learning SACBFs

The paper proposes three distinct data-driven synthesis methods for $Q_B$:

1. Reinforcement Learning (RL): Treats the SACBF synthesis as a reachability problem. It adapts value-based RL methods (like Temporal Difference learning) to approximate the optimal value function of a reachability problem. Note: This method lacks formal convergence guarantees for the safety certificate itself due to the non-contractive nature of the undiscounted Bellman equation in reachability.
2. Expert-Guided Learning: Assumes the availability of an expert safe controller (e.g., Artificial Potential Fields). It formulates the synthesis as a robust optimization problem to find an SACBF that is consistent with the expert's safe trajectories.
3. Supervised Learning (SL): Assumes a valid classical CBF $B(x)$ is already known. It uses supervised regression to learn the mapping from state-action pairs to the SACBF value, derived from the known CBF.

D. Robustness Analysis: Error-to-State Safety (ESSf)

To address learning errors (where the learned $Q_B^\omega$ deviates from the true $Q_B$), the authors introduce an Error-to-State Safety (ESSf) framework.

• Mechanism: Instead of relying on model-based prediction, the framework quantifies the impact of learning errors ($\epsilon$) directly on the safety constraints.
• Strategy: It employs a constraint tightening and relaxation approach:
  • Tighten the original state constraints ($h(x) \leq 0$) to $h(x) + \delta(\epsilon) \leq 0$.
  • Relax the SACBF constraint in the filter to $Q_B^\omega(x, u) \leq \kappa(\epsilon)$.
• Result: Theorems 1 and 2 provide closed-form relationships showing that if the learning error is bounded, the system remains recursively feasible and safe within the tightened state constraints.

E. Policy Refinement

Once a valid SACBF is learned, the framework extends unconstrained value-based RL algorithms (like Fitted Q Iteration) to the constrained setting. The algorithm (Constrained FQI) iteratively updates the performance objective $Q_\theta$ while enforcing the learned SACBF constraint, ensuring the resulting policy is near-optimal for the task while satisfying safety.

3. Key Contributions

1. Direct Data-Driven Safe Control Framework: The paper proposes a framework that learns safe controllers directly from data without system identification, bridging the gap between model-free learning and constrained control.
2. State-Action Control Barrier Functions (SACBF): It introduces and utilizes SACBFs, which evaluate safety on state-action pairs, enabling safety filters that do not require explicit dynamics models.
3. Three Synthesis Schemes: It presents three methods to learn SACBFs: via RL (general but less guaranteed), via expert demonstrations (robust), and via supervised learning from known CBFs (sample-efficient).
4. Error-to-State Safety (ESSf) Analysis: It provides a novel robustness analysis framework that quantifies how learning errors affect safety, offering formal guarantees through state constraint tightening and SACBF constraint relaxation.
5. Decoupled Optimization: The framework separates performance learning (optimizing $Q$) from safety enforcement (optimizing $Q_B$), allowing the integration of various learning algorithms while maintaining safety.

4. Results

The framework was validated through simulations on an autonomous vehicle moving in a 2D space with obstacles.

• Case Study Setup: A kinematic vehicle model with state and input constraints was used. The SACBF was learned using both the expert-guided approach (using Artificial Potential Fields as the expert) and the RL approach.
• Safety Performance:
  • The proposed 3DSF (Direct Data-Driven Safety Filter) achieved a 100% safety rate across various initial conditions and reference policies (both unsafe APF and unsafe DDPG).
  • In contrast, Indirect Data-Driven Safety Filters (which rely on a learned model) achieved lower safety rates (e.g., ~79% for unsafe APF), attributed to the compounding of model identification errors and CBF learning errors.
  • MPC (Model Predictive Control) achieved 100% safety but suffered from poor task performance (only ~78% of trajectories reached the target) due to non-convex optimization issues and high computational cost.
• Task Performance:
  • The Constrained FQI algorithm (learning $Q$ with SACBF constraints) achieved a total cost of 173.85, outperforming both safe and unsafe DDPG policies filtered by 3DSF. This demonstrates that incorporating safety constraints directly into the RL training process improves task performance compared to post-hoc filtering or reward shaping.
• Computational Efficiency: The 3DSF demonstrated significantly lower online CPU times (approx. 2.2 ms) compared to MPC (approx. 33 ms), making it suitable for real-time applications.

5. Significance and Claims

The paper claims to offer a modest but rigorous advancement in safe learning-based control:

• Bridging the Gap: It successfully bridges the gap between model-free learning and constrained control by removing the dependency on explicit system models for safety enforcement.
• Formal Guarantees without Models: It provides formal safety guarantees (recursive feasibility and constraint satisfaction) even when the system dynamics are unknown, provided learning errors are bounded.
• Superiority over Indirect Methods: The results suggest that direct data-driven approaches avoid the error compounding inherent in indirect methods (model identification + controller learning), leading to better safety rates.
• Flexibility: The framework is compatible with a wide range of existing learning algorithms (RL, SL) and can be updated online.

Limitations Acknowledged:

• The method requires sampling in the state-action space, which increases sample complexity compared to state-only methods.
• Safety constraints may be violated during the data collection phase for training, which may be unsuitable for real-world systems where safety is paramount at all times (though simulators can mitigate this).
• The RL-based synthesis of SACBFs lacks formal convergence guarantees for the safety certificate itself due to the nature of the reachability problem.
• The current work focuses on discrete-time systems; extension to continuous-time is future work.