Technical Summary: Learning When to Act via Run-Time Assurance

Problem Statement

Safe reinforcement learning (RL) has largely focused on determining what an agent should do, often neglecting the critical question of when it needs to act. In safety-critical cyber-physical systems, fixed-rate control executes at every timestep regardless of state changes, incurring unnecessary costs in sensing, computation, and bandwidth. While classical analytical methods like Self-Triggered Control (STC) and Event-Triggered Control (ETC) offer strong stability guarantees, they often rely on conservative Lyapunov or model-based triggering rules that fail to exploit communication savings in nonlinear, underactuated, or high-dimensional systems. Conversely, existing learning-based approaches often lack formal safety guarantees on the triggering decision, rely on learned dynamics models, or address continuous-time settings without hard timing-safety mechanisms.

The core challenge addressed is lifting the fixed-rate assumption without sacrificing safety. The goal is to learn a policy that jointly optimizes control inputs and inter-sample intervals (the time until the next update) to maximize the Mean Inter-Sample Interval (MSI) while guaranteeing closed-loop stability.

Methodology

The authors propose a unified RL framework supervised by a Run-Time Assurance (RTA) layer acting as a safety shield. The system operates in discrete-time STC, where at each decision instant $t_k$, the agent selects a control input $u_k$ and a duration $\tau_k$ from a discrete set $\mathcal{T} = \{\tau_{min}, 2\tau_{min}, \dots, N\tau_{min}\}$.

1. Run-Time Assurance (RTA) Layer

The RTA layer provides a hard safety override, distinct from constrained MDP methods that enforce safety only in expectation.

• Prediction: At each step, the system computes a one-step-ahead linearized prediction of a safety-critical scalar $q_k = c^\top x_k$:
  $$\hat{q}_{k+1} = q_k + \tau_k \dot{q}_k + \frac{1}{2}\tau_k^2 \ddot{q}_{lin}(x_k, u_k)$$
• Override Mechanism: If the predicted state violates a safety threshold ($|\hat{q}_{k+1}| > \theta_{RTA}$) or exceeds position bounds, the RL action is immediately overridden. The system substitutes a precomputed Linear-Quadratic Regulator (LQR) backup control $u_k = -Kx_k$ and resets the interval to the minimum safe duration $\tau_{min}$.
• Safety Guarantee: The backup controller is designed such that if it is unsaturated, the Lyapunov function $V(x)$ is guaranteed to decrease pointwise. This provides a strictly stronger guarantee than expectation-based constraints.

2. Learning Framework

• Lyapunov Reward: The framework utilizes a Lyapunov function $V(x) = x^\top P x$ derived from the Continuous-time Algebraic Riccati Equation (CARE) of the linearized plant. The reward function includes a stability term based on the decrease of $V(x)$ and a communication efficiency term proportional to the square of the normalized inter-sample interval.
• Action Space: The agent learns a joint policy for $(u_k, \tau_k)$. For Single-Input Single-Output (SISO) systems, a Deep Q-Network (DQN) is used; for Multi-Input Multi-Output (MIMO) systems with large action spaces (e.g., the 3D Quadrotor), Soft Actor-Critic (SAC) is employed on a continuous action space.
• Preference-Conditioned Extension: To avoid training separate models for different stability-communication tradeoffs, a preference-conditioned agent is trained where a weight $w_c$ (controlling the tradeoff) is sampled at the start of each episode. This allows a single model to recover the full tradeoff frontier.

Key Contributions

1. Joint RL Formulation with Hard Safety: A novel formulation that simultaneously learns control inputs and inter-sample intervals under a hard pointwise safety certificate. Unlike prior shielded RL that acts on fixed time steps, this shield acts directly on the timing decision $\tau_k$.
2. CARE-Derived Lyapunov Reward: A transferable reward signal derived from CARE solutions that works across SISO and MIMO environments without redesign, enabling a single weight $w_c$ to trace the full stability-communication tradeoff frontier.
3. Systematic Empirical Validation: Extensive evaluation on four environments (Inverted Pendulum, CartPole, Planar Quadrotor, and 3D Quadrotor) demonstrating:
   • Performance Gains: The learned policy achieves 1.45× to 3.51× higher MSI compared to classical Lyapunov-STC baselines.
   • Necessity of Adaptive Timing: A fixed LQR controller operating at the same average rate as the learned policy is unstable on all tested plants, proving that adaptive timing (sampling fast near instability and slowly near equilibrium) is essential for safety, not just a reduced average rate.
   • RTA Necessity: Ablation studies show that removing the RTA shield reduces MSI by 1.27–1.84× and significantly degrades state norms, confirming the shield's role in shaping the feasible policy space.
   • Comparison with Lagrangian Methods: Unlike Lagrangian-DQN baselines which enforce safety in expectation, the proposed method achieves zero hard safety violations while Lagrangian methods accumulate up to 35% hard violations and achieve 21–52% lower MSI.
4. Scalability to High Dimensions: A case study on a 12-state 3D Quadrotor where analytical STC design is intractable. The framework achieves an MSI of 0.302 s (94% of the maximum possible interval) with 0% RTA activation, whereas classical STC remains pinned at the minimum interval $\tau_{min}$.

Results

• Communication Efficiency: On the Inverted Pendulum, CartPole, and Planar Quadrotor, the RL-STC policy significantly outperforms classical analytical baselines. For instance, the Planar Quadrotor achieves a 3.51× improvement in MSI over classical STC.
• Robustness: The framework demonstrates graceful degradation under $\pm 30\%$ plant mass variation and additive disturbances. While RTA activation increases with disturbance amplitude, the system remains stable, and the RTA layer absorbs perturbations that the learned policy cannot handle alone.
• Algorithm Agnosticism: Results are consistent across DQN (discrete actions) and SAC (continuous actions), confirming the framework's validity across different optimization paradigms.
• Compute Efficiency: The preference-conditioned extension recovers the full tradeoff frontier from a single model trained with only 2/11 of the compute required for a full weight sweep.

Significance and Claims

The paper claims that the question of when to act is as consequential as what to do. By integrating a hard RTA shield with joint RL optimization, the framework enables agents to discover non-conservative inter-sample intervals that are analytically difficult to derive, particularly in high-dimensional or nonlinear systems.

The authors emphasize that their approach provides formal pointwise safety guarantees (via the Lyapunov-decreasing backup) that expectation-based constraint methods cannot provide by construction. The framework is scoped to stabilization around known equilibria where CARE solutions and linearized predictions are well-defined, but the empirical results on the 12-state Quadrotor suggest the approach can extend to regimes where formal certificates do not strictly apply at the chosen minimum interval, relying instead on the training-time shaping signal to respect safety predicates.

The work establishes that adaptive timing is the key enabler of safe sparsity; static reduction of control rates is insufficient and often leads to instability. The proposed framework offers a tractable path to communication-efficient control in safety-critical systems without sacrificing the rigorous safety guarantees required for deployment.