⚡ electrical engineering

Deep Reinforcement Learning for Reach-Avoid-Stay Problems

This paper proposes a two-step deep reinforcement learning framework that jointly learns the maximal robust Reach-Avoid-Stay set and a corresponding switching control policy, demonstrating superior accuracy and performance in ensuring systems safely reach and remain within target sets under bounded disturbances compared to existing methods.

Original authors: Gabriel Chenevert, Jingqi Li, Achyuta kannan, Sangjae Bae, Donggun Lee

Published 2026-09-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gabriel Chenevert, Jingqi Li, Achyuta kannan, Sangjae Bae, Donggun Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of robotics and autonomous machines, safety is not just about avoiding a crash; it is about knowing exactly where a machine can go and, more importantly, where it must stop. Imagine a self-driving car or a delivery drone navigating a busy city. Engineers use mathematical tools to map out "safe zones," ensuring that if the machine starts from a certain spot, it can reach its destination without hitting anything. However, a common flaw in these safety maps is that they often tell a machine how to arrive at a target but fail to tell it how to stay there. A vehicle might reach a parking spot but, due to wind or a sudden speed, be unable to slow down enough to remain parked, causing it to drift out of the safe zone and into danger. This gap between arriving and staying has long been a difficult problem for computer scientists trying to make machines truly reliable in unpredictable environments.

To solve this, a team of researchers at North Carolina State University and the University of Texas at Austin developed a new way to teach machines how to reach a goal and hold their ground against any disturbance. They focused on a specific challenge called the "reach-avoid-stay" problem. In simple terms, this asks: from which starting points can a machine safely reach a target, avoid all obstacles, and then remain inside that target forever, even if the wind blows or the road gets slippery? Previous methods struggled with this because they either relied on complex mathematical designs that were hard to create for complicated machines, or they made unrealistic assumptions, such as assuming a vehicle could stop instantly. The researchers proposed a two-step learning process using a type of artificial intelligence known as deep reinforcement learning, where a computer learns by trial and error in a simulated world.

The team's approach works like a two-stage journey. First, the computer learns to identify a special inner zone within the target area. This inner zone is a place where the machine can stay safely forever, no matter what disturbances it faces. The researchers call this the "robust viability kernel." To find this, the AI learns a policy, or a set of rules, that keeps the machine moving in a way that never lets it drift out of this safe inner circle. Once the computer has mastered this "stay" behavior, it moves to the second step. Here, it learns how to get the machine from any starting point to that safe inner zone as quickly as possible, while still avoiding all obstacles along the way. The researchers proved mathematically that if a machine can reach this inner zone and then stay there, it has successfully completed the entire task. By combining these two learned behaviors into a single switching system, the machine knows exactly when to drive toward the goal and when to switch to a mode that keeps it locked in place.

The researchers tested this method on several different scenarios, ranging from a simple two-dimensional cart on a track to complex, high-dimensional simulations of a vertical takeoff and landing taxi flying through a city and a tractor unloading crops from a moving combine harvester. In the case of the simple cart, they compared their new method against an older technique that uses complex mathematical functions. Their new approach identified a safe starting area that was nearly fifteen times larger than the area found by the older method, meaning it allowed the machine to start from many more positions without risking a failure. In the more complex simulations involving the flying taxi and the tractor, the new method identified the safe starting areas with over 95 percent accuracy, even when the learning process included small errors typical of computer training.

The results showed a stark difference between machines trained with the old methods and those trained with this new two-step framework. When tested on the flying taxi, the older "reach-and-avoid" method, which only cares about getting to the target, failed completely at the task of staying there, achieving a zero percent success rate. In contrast, the new method succeeded 93 percent of the time. Similarly, for the tractor scenario, the new method succeeded 99.3 percent of the time, while the older method succeeded only 73.5 percent of the time. The researchers found that the older methods often drove the machines into the target area at full speed, leaving them with no way to slow down and stay put. The new method, however, learned to slow the machine down early, ensuring it entered the safe inner zone at a speed where it could remain indefinitely.

While the simulations were highly successful, the researchers noted that their system relies on having a precise understanding of the environment and the machine's physics before training begins. If the real world behaves differently than the computer model—for instance, if the wind is stronger than predicted or the ground is more slippery—the trained machine might not be able to guarantee safety. The team suggests that future work will need to integrate real-world sensor data to handle these unknowns. For now, this two-step learning framework offers a significant step forward, providing a way to teach machines not just how to arrive, but how to stay, turning a theoretical safety guarantee into a practical tool for complex, real-world applications.

Technical Summary: Deep Reinforcement Learning for Reach-Avoid-Stay Problems

Problem Definition

The paper addresses the Reach-Avoid-Stay (RAS) problem for nonlinear dynamical systems subject to bounded adversarial disturbances. Unlike standard Reach-Avoid (RA) problems, which require a system to reach a target set $T$ while avoiding unsafe regions, the RAS problem demands that the system not only reach $T$ but also remain within it indefinitely despite all bounded disturbances.

The core challenge identified is that existing RA methods often fail to distinguish between the target set $T$ and the robust viability kernel (the maximal robust control-invariant subset of $T$ ). If a system reaches $T$ but not the viability kernel (e.g., entering with excessive velocity), it may be unable to stay within $T$ under disturbance, leading to safety violations. Current approaches to RAS face significant limitations:

Control-Lyapunov-Barrier Function (CLBF) methods are difficult to design for high-dimensional or nonlinear systems.
Funnel-based methods rely on the restrictive assumption that systems can stop instantly.
Existing RL approaches often fail to identify the maximal robust RAS set, resulting in conservative policies or inaccurate safety classifications.

The goal is to compute the maximal robust RAS set (the set of all initial states from which the RAS task is achievable) and a corresponding control policy that guarantees task completion under all bounded disturbances.

Methodology

The authors propose a two-step Deep Reinforcement Learning (RL) framework that jointly learns the maximal robust RAS set and the associated control policy. The framework utilizes Deep Deterministic Policy Gradient (DDPG) for high-dimensional systems.

Step 1: Computing the Robust Viability Kernel ( $A_S$ )

The first step identifies the robust viability kernel $A_S$ within the target set $T$ . This is the set of states from which the system can remain in $T \cap C$ (target and constraint sets) indefinitely.

A discounted value function $H(x)$ is defined to characterize $A_S$ .
$H(x)$ is the unique solution to a Bellman equation involving a min-max operator over control and disturbance policies.
The set $A_S$ corresponds to the zero-level set of $H$ (where $H(x)=0$ ).
A policy $\pi_H$ is derived to keep the system within $A_S$ once inside.

Step 2: Computing the Maximal Robust RAS Set

The second step formulates the RAS problem as an RA problem where the target is the robust viability kernel $A_S$ rather than the original set $T$ .

Value Function Transformation: Since $H(x)$ $H (x)$ is zero on $A_S$ $A_{S}$ , it cannot serve as a reward for reaching $A_S$ $A_{S}$ (as it provides no incentive to reach the set quickly). The authors introduce a transformed value function $H_g(x)$ $H_{g} (x)$ :
- $H_g(x) = H(x)$ if $H(x) < 0$ (outside $A_S$ ).
- $H_g(x) = g(x)$ (positive) if $x \in A_S$ .
- This ensures $H_g$ is strictly positive within $A_S$ and non-positive elsewhere.
RA Formulation: A new value function $V(x)$ is defined using $H_g$ as the target reward. $V(x)$ characterizes the maximal robust RAS set.
Switching Policy: A unified policy $\pi_{RAS}$ $π_{R A S}$ is constructed by switching between the "reach" policy ( $\pi_V$ $π_{V}$ ) and the "stay" policy ( $\pi_H$ $π_{H}$ ):
- Use $\pi_V$ when $H_g(x) \leq 0$ (outside $A_S$ ) to drive the system toward $A_S$ .
- Switch to $\pi_H$ when $H_g(x) > 0$ (inside $A_S$ ) to ensure indefinite safety.

Key Contributions

Theoretical Proof of Maximality: The authors prove that, in the absence of training errors, their framework identifies the exact maximal robust RAS set. They demonstrate that the resulting switching policy guarantees the RAS task for all states in this set under all bounded disturbances. This is achieved via the proposed value function transformation ( $H_g$ ), which enables the integration of viability kernel analysis with RA methods.
High-Accuracy Learning with Deep RL: The framework is adapted for high-dimensional systems using DDPG. The authors show that even with training errors inherent to neural networks, the framework characterizes the maximal robust RAS set with high accuracy (>95%).
Practical Applicability: The method handles complex environments where CLBF design is intractable and does not rely on the "instant stop" assumption required by funnel-based methods.
Empirical Validation: Simulations on low-dimensional (Double Integrator) and high-dimensional (VTOL Taxi, Nonlinear Harvester) systems validate the approach.

Results

Maximal Set Identification: In a 2D Double Integrator example, the proposed framework identified an RAS set with a volume of 28.67, significantly larger than the 1.98 volume identified by a CLBF baseline. This indicates the proposed method is less conservative and identifies more feasible states.
Accuracy: In high-dimensional experiments (VTOL and Harvester), the framework achieved 95.8% and 99.4% accuracy, respectively, in identifying the RAS set compared to ground truth generated by the policy itself.
Task Success Rates: The proposed RAS policy significantly outperformed a baseline RA method in completing the RAS task:
- VTOL: 93.0% success rate (RAS) vs. 0% (RA baseline).
- Harvester: 99.3% success rate (RAS) vs. 73.5% (RA baseline).
Failure Analysis: The paper demonstrates that RA policies fail the RAS task because they reach the target set $T$ at high speeds without entering the viability kernel $A_S$ . Consequently, they cannot maintain the state within $T$ under disturbance. The RAS policy correctly slows the system before entering $T$ to ensure it lands within $A_S$ .

Significance and Claims

The paper claims that its primary significance lies in providing a theoretically grounded, data-driven framework that solves the RAS problem for general nonlinear systems without restrictive assumptions.

It bridges the gap between reachability analysis and deep RL, proving that the two-step approach yields the maximal robust set.
It addresses the "false negative" problem of conservative methods (like CLBF) and the "false positive" safety risks of standard RA methods.
The authors acknowledge that the framework relies on prior knowledge of system dynamics and environment constraints. They note that while current safety-critical RL often lacks rigorous theoretical guarantees, this work provides such assurances for the RAS problem, provided the model is accurate. Future work is suggested to integrate real-world sensor data and handle unknown environment information.