Technical Summary: Constrained Policy Optimization via Sampling-Based Weight-Space Projection (SCPO)

1. Problem Statement

The paper addresses the challenge of safety-critical learning, specifically the need to improve the performance of learning-based controllers without violating safety constraints or leaving a known safe operating regime. The authors focus on a setting where:

• Safety Constraints are Rollout-Based: Safety is defined by trajectory-level metrics (e.g., state constraints, stability margins) that can be evaluated via simulation (rollouts) but are not analytically differentiable with respect to policy parameters.
• Black-Box Constraints: The constraint functions $g(\theta)$ lack closed-form Jacobians, rendering standard constrained optimization methods (like those requiring explicit gradients of constraints) inapplicable.
• Goal: To perform constrained policy optimization that ensures recursive feasibility (staying safe at every training step) and monotonic improvement of the task objective, starting from a certified safe initialization.

2. Methodology: SCPO

The authors propose SCPO (Sampling-Based Constrained Policy Optimization), a method that enforces safety directly in the weight space of the policy network.

2.1 Architecture and Initialization

Inspired by Low-Rank Adaptation (LoRA), the policy $\pi_\theta$ is structured as a combination of a certified baseline controller ($\pi_{safe}$) and a residual neural network ($\phi_\theta$):
$$\pi_\theta(x) = \begin{cases} \pi_{safe}(x) + \phi_\theta(x) & x \notin X_f \\ \pi_{safe}(x) & x \in X_f \end{cases}$$

• $\pi_{safe}$: A non-parametric, stabilizing controller (e.g., LQR with clipping) that guarantees safety and stability within a region $\Omega$.
• $\phi_\theta$: A neural network initialized to output zero ($\phi_{\theta_0} \equiv 0$). This ensures the initial policy is identical to the safe baseline.
• Learning Objective: The training process is viewed as "safe adaptation" in the neighborhood of $\pi_{safe}$, where $\phi_\theta$ learns to improve performance while the projection mechanism ensures the combined policy remains stabilizing.

2.2 Local Safe Region Construction

Since $g(\theta)$ is non-differentiable, SCPO constructs a local safe set in weight space using:

1. Pointwise Evaluations: Safety metrics are evaluated on a finite set of candidate weight updates (rollouts).
2. Smoothness Bounds: The method assumes local Lipschitz continuity of the safety constraints. It uses a quadratic upper bound to approximate the constraint surface:
   $$g_i(\theta_t + \Delta\theta) \leq g_i(\theta_t) + \nabla g_i(\theta_t)^\top \Delta\theta + \frac{L_i}{2}\|\Delta\theta\|^2$$
   where $L_i$ are curvature constants estimated from sampled updates.

2.3 Sampling-Based Projection

Instead of solving a high-dimensional constrained optimization problem directly, SCPO restricts the update to the span of sampled candidate updates ($D$).

• Candidate Bank: A buffer $D$ stores recent updates and their corresponding safety evaluations $G$.
• Low-Dimensional Projection: The update is parameterized as $\Delta\theta = Dc$, where $c \in \mathbb{R}^m$ is a coefficient vector.
• Convex Optimization: The problem reduces to a Second-Order Cone Program (SOCP) (or Quadratically Constrained Quadratic Program) in the low-dimensional space $c$:
  $$\min_c \|c - e_m\|_S^2 \quad \text{s.t.} \quad (1 - \mathbf{1}^\top c)g(\theta_t) + Gc + \frac{1}{2}(c^\top Sc + |c|^\top \text{diag}(S))L \leq 0$$
  This projects the raw gradient update onto a conservative local safe set, guaranteeing that the new weights satisfy the safety constraints.

2.4 Theoretical Guarantees

• Safe-by-Induction: Starting from a safe initialization ($g(\theta_0) \leq 0$), the projection ensures $g(\theta_{t+1}) \leq 0$ for all subsequent steps, provided the projection is feasible.
• Stability: For linear dynamical systems, the method preserves the Lyapunov margin, ensuring the closed-loop system remains locally asymptotically stable and the safe region $\Omega$ is forward invariant.
• Monotonic Improvement: Using an Armijo-style backtracking line search, the algorithm guarantees a decrease in the task loss $L(\theta)$ without violating constraints.

3. Key Contributions

1. Weight-Space Projection: A novel sampling-based method that performs safety projection directly in the parameter space without requiring gradients of the constraint functions.
2. Safe-by-Induction Mechanism: A learning framework that guarantees recursive feasibility and monotonic objective improvement, starting from any safe initialization.
3. Control-Theoretic Analysis: A formal proof that SCPO maintains closed-loop stability for linear systems by preserving Lyapunov margins.
4. Empirical Validation: Demonstration of robustness against harmful supervision and safe exploration in constrained regression and imitation learning tasks.

4. Experimental Results

The authors evaluate SCPO on two tasks:

4.1 Constrained Regression

• Setup: A 1D regression problem where the target function violates a hard constraint ($|y| \leq 1.4$) in certain regions.
• Result: SCPO successfully fits the target function in safe regions while smoothly saturating at the constraint boundary. Unlike a soft-constraint baseline (which requires differentiable constraints and hyperparameter tuning), SCPO strictly enforces the hard constraint without violation.

4.2 Double-Integrator Imitation Learning

• Setup: A control task where a neural network attempts to imitate a "malicious expert" (an aggressive policy that destabilizes the system) while starting from a safe LQR baseline.
• Result:
  • Rejection of Harmful Updates: SCPO rejects updates that would lead to instability, preventing the policy from mimicking the expert's destabilizing behavior.
  • Stability Preservation: The learned policy maintains the stabilizing region of the original safe controller, whereas the malicious expert fails to stabilize the system over a large portion of the state space.
  • Performance: The policy achieves meaningful improvement in the imitation objective while remaining within the safe operating regime.

5. Significance and Limitations

Significance:
The paper claims that SCPO provides a practical, data-driven approach to safe learning that does not rely on explicit constraint costs or differentiable safety filters. By treating a certified backup controller as a frozen base and projecting updates in weight space, it enables safe adaptation beyond conservative baselines. This is particularly relevant for domains like autonomous driving and robotics where formal guarantees are difficult to derive but safety is non-negotiable.

Limitations:
The authors acknowledge that the method is conservative. The projection step does not generate new directions for improvement; it only selects the best safe direction from the existing sampled candidates. The authors note that future work aims to integrate other techniques to modify the objective for improved safe exploration, addressing this conservativeness.

Conclusion:
SCPO offers a rigorous framework for constrained policy optimization that bridges the gap between high-performance learning and strict safety requirements, ensuring that learning-based controllers remain safe and stable throughout the training process.