🤖 machine learning

Interacting safely with cyclists using Hamilton-Jacobi reachability and reinforcement learning

This paper proposes a hybrid framework that integrates Hamilton-Jacobi reachability analysis with deep Q-learning to enable autonomous vehicles to safely and efficiently interact with cyclists by using safety metrics as structured rewards and modeling human behavioral adaptation.

Original authors: Aarati Andrea Noronha, Jean Oh

Published 2026-02-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Aarati Andrea Noronha, Jean Oh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a self-driving car, and you see a cyclist ahead of you. This is a tricky situation. If you drive too fast or get too close, you might scare the cyclist or cause an accident. But if you drive too slowly or stay too far away, you might block traffic and waste everyone's time.

This paper is about teaching self-driving cars how to find that "Goldilocks" zone: safe enough to not scare the cyclist, but fast enough to get to the destination efficiently.

Here is how the authors solved this problem, broken down into simple concepts and analogies.

1. The Two Old Ways (and why they failed)

The authors looked at two existing ways to solve this:

The "Paranoid Robot" (Hamilton-Jacobi Reachability):
Imagine a robot that is terrified of hitting anything. It calculates every possible way a cyclist could move and says, "If I go anywhere near that path, I might crash!" So, it stops completely or moves incredibly slowly.
- The Problem: It's too safe. It guarantees no crashes, but it makes the car useless because it never actually gets anywhere.
The "Gambler" (Reinforcement Learning):
Imagine a robot that learns by trial and error, like a video game character. It tries different moves to get a high score (getting to the goal fast).
- The Problem: It's fast, but it might take a risky shortcut that leads to a crash because it doesn't have a strict "safety rulebook."

2. The New Solution: The "Safety Net" + "Smart Coach"

The authors combined these two methods into a super-team.

The Safety Net (Hamilton-Jacobi): This acts like a strict referee. It constantly calculates a "Safety Score" for every possible move. If a move puts the car in a zone where a crash is inevitable, the Safety Net says, "Nope, forbidden!"
The Smart Coach (Deep Q-Learning): This is the part that learns how to drive. It uses the Safety Net's score as a reward. If the car stays safe, it gets points. If it gets to the goal quickly, it gets more points. The coach teaches the car to be as fast as possible without breaking the Safety Net's rules.

3. The Secret Sauce: Understanding the Cyclist's Feelings

This is the most creative part of the paper.

In the past, computers treated cyclists like random wind gusts—unpredictable and mechanical. But humans aren't machines; they have feelings. If a car gets too close, a cyclist gets scared and might swerve, even if the car isn't technically going to hit them.

The authors built a "Empathy Engine" (using a tool called an Auto-Encoder):

They fed the computer thousands of real-world driving videos.
They taught it to recognize what a cyclist considers "safe" vs. "scary."
They realized that sometimes, a car is technically safe (it won't hit the bike), but the cyclist feels unsafe because the car is too close.

The computer now models the cyclist's latent response (their hidden feelings). It thinks: "If I pass here, the math says I'm safe, but the cyclist will feel threatened and swerve. So, I will give them a little extra space."

4. The Results: How did it do?

They tested their new "Empathic Robot" against:

Real Human Drivers: Who often get too aggressive or too slow.
The Old "Paranoid Robot" (Fisac 2019): Who was too cautious.

The Winner:

Safety: Their robot was safer than human drivers and the old robot. It avoided "unsafe states" (scary moments) more often.
Speed: It was faster than the old paranoid robot. It didn't just sit still; it found a smooth, efficient path.
Human-Like: It understood that "safe" isn't just about math; it's about making the cyclist feel comfortable.

The Big Picture Analogy

Think of driving past a cyclist like walking past a nervous dog in a park.

The Old Robot would freeze in place and refuse to walk past the dog because it's afraid the dog might bite.
The Old Learning Robot might run past the dog quickly to get to the bench, accidentally startling the dog.
This New Approach is like a calm, experienced dog walker. It knows exactly how close it can get without making the dog nervous. It moves at a steady pace, giving the dog enough space to feel comfortable, ensuring everyone gets where they are going without a scare.

In short: This paper taught self-driving cars to not just calculate physics, but to understand human (and cyclist) psychology, resulting in a driving style that is both safe and efficient.

1. Problem Statement

The core challenge addressed is enabling autonomous vehicles (AVs) to interact with cyclists in a way that balances safety and optimality (time efficiency).

The Conflict: Existing AV algorithms are often overly cautious when passing cyclists, leading to sub-optimal navigation and traffic delays. Conversely, human drivers often exhibit overly aggressive behavior.
The Gap: While Hamilton-Jacobi (HJ) reachability analysis guarantees safety by defining "failure zones," it often results in conservative behavior that prevents the vehicle from reaching its goal in a reasonable time. Conversely, Reinforcement Learning (RL) offers optimality but lacks formal safety guarantees.
Specific Complexity: Cyclist motion is random, slow, and unpredictable. Furthermore, cyclists have a "comfort level" regarding proximity to vehicles, which is a latent variable not easily captured by standard kinematic models.

2. Methodology

The authors propose a hybrid framework integrating Hamilton-Jacobi (HJ) Reachability Analysis with Deep Q-Learning (DQN). The system is modeled as a two-agent zero-sum differential game, where the AV is the control agent ( $u$ ) and the cyclist is the disturbance agent ( $d$ ).

A. Safety Quantification via HJ Reachability

State Space: The system state $s$ is defined by the relative longitudinal range ( $\Delta x$ ), longitudinal range rate ( $\Delta v$ ), and lateral range ( $\Delta y$ ) of the cyclist relative to the AV.
Collision Set ( $C_0$ ): Defined as states where the distance is zero (collision).
Backward Reachable Set ( $B$ ): Computed by solving a time-dependent Hamilton-Jacobi-Bellman (HJB) partial differential equation. This set represents states from which a collision is inevitable regardless of the AV's control input.
Value Function ( $v(s, t)$ ): Serves as a quantitative safety metric.
- $v(s) > 0$ : Safe state (collision can be avoided).
- $v(s) \leq 0$ : Dangerous state (collision is inevitable).

B. Modeling Cyclist Comfort (Disturbance Input)

A key innovation is modeling the cyclist's "disturbance input" not just as acceleration, but as a function of their latent comfort level relative to the AV's state.

Auto-Encoder Approach: To handle the scarcity of "dangerous" interaction data (rare events), the authors use an auto-encoder trained on safe states (derived from the HJ value function).
Anomaly Detection: The auto-encoder learns the pattern of safe interactions. When fed a mixture of safe and dangerous states, high reconstruction errors indicate "dangerous" states (where the cyclist feels threatened).
Mapping: This creates a mapping $w$ that adjusts the disturbance set $D$ based on the cyclist's perceived comfort, making the safety analysis more realistic than standard worst-case assumptions.

C. Reinforcement Learning for Navigation

To overcome the excessive caution of pure HJ reachability, the computed value function is integrated into a Deep Q-Network (DQN):

Reward Signal: The HJ value function is used as a structured reward signal. It penalizes the agent for entering unsafe states while encouraging goal-reaching.
Optimization: The DQN learns a policy to maximize the cumulative discounted reward, effectively finding a trajectory that balances safety (high value function) and time efficiency (reaching the goal).
Training: Uses experience replay and the Adam optimizer to ensure convergence and stability.

3. Key Contributions

Dual-Agent Extension: Extends the single-agent HJ-RL framework (proposed by Fisac et al., 2019) to a two-agent system (AV + Cyclist), treating the cyclist as a dynamic disturbance.
Quantifying Cyclist Comfort: Introduces a novel method to model the cyclist's disturbance input as a function of their comfort level using auto-encoders for anomaly detection on rare event data.
Balanced Navigation: Demonstrates a framework that achieves a trade-off between safety guarantees (via HJ) and time-optimal navigation (via DQN), avoiding the "freezing robot" problem.
Safety Metric: Provides a quantitative definition of safety that includes the ability to pass a cyclist at a reasonable speed and distance, rather than just avoiding collisions.

4. Experimental Results

The framework was evaluated using the Safety Pilot Michigan Database (SPMD), containing 34 million miles of naturalistic driving data.

Dataset: 60 cyclist events were used for testing (not seen during training).
Comparisons: The proposed method was compared against:
1. Human drivers (Ground Truth).
2. The single-agent HJ-RL method (Fisac et al., 2019).

Key Metrics & Findings:

Safety: The proposed framework achieved the highest safety score (lowest negative safety factor: -2.01) compared to Human drivers (-2.31) and Fisac '19 (-2.18). It encountered fewer "unsafe states" (31.6% vs 39.0% for humans).
Optimality (Time): While slightly slower than Fisac '19 (199s vs 233s), the proposed method was significantly faster than human drivers (253s). The authors attribute the slight delay to the model correctly accounting for cyclist comfort, which the single-agent model ignored.
Collision Rate: 0 collisions for all three methods in the test set.
Goal Achievement: 100% success rate for all methods.
Visualization: The value function plots showed that the proposed method correctly identified specific "threat" zones (e.g., close proximity with low relative speed) that the single-agent model missed, reflecting human perception more accurately.

5. Significance

This paper represents a significant step forward in autonomous navigation in mixed-traffic environments.

Human-Centric Safety: By modeling the cyclist's perception of safety (comfort) rather than just physical collision geometry, the AV behaves more like a considerate human driver.
Scalability: It demonstrates that HJ reachability, traditionally limited by computational complexity in high dimensions, can be effectively combined with RL to handle multi-agent interactions.
Practical Application: The method successfully navigates the trade-off between "playing it safe" (which causes traffic jams) and "being efficient" (which risks accidents), providing a robust solution for real-world AV deployment in urban cycling scenarios.