⚡ electrical engineering

Falsification-driven reinforcement learning for maritime motion planning

This paper proposes a falsification-driven reinforcement learning approach that generates adversarial training scenarios based on signal temporal logic specifications to improve maritime traffic rule compliance in autonomous vessel motion planning.

Original authors: Marlon Müller, Florian Finkeldei, Hanna Krasowski, Murat Arcak, Matthias Althoff

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Marlon Müller, Florian Finkeldei, Hanna Krasowski, Murat Arcak, Matthias Althoff

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Robot Captain to Follow the Rules

Imagine you are trying to teach a robot captain how to steer a ship across the open ocean. The ocean is a busy place with other ships, and there are strict rules (like "give way to the right" or "don't cross paths") to prevent crashes. These rules are called COLREGs.

The problem is that if you just let the robot captain practice by sailing randomly, it might never encounter a tricky situation where it almost crashes. It learns to sail straight and happy, but it doesn't learn how to handle the complex, dangerous moments where it needs to make a quick, rule-abiding decision. It's like teaching someone to drive only on empty highways; they'll never learn how to merge into heavy traffic safely.

This paper proposes a new training method called Falsification-Driven Reinforcement Learning.

The Core Idea: The "Devil's Advocate" Coach

Instead of just letting the robot sail randomly, the researchers act as a "Devil's Advocate." They use a special computer program to actively try to break the robot's rules.

The Test: The computer program acts as a tricky opponent ship. It tries to create a situation where the robot captain fails to follow the rules (e.g., the opponent ship forces the robot into a position where it might crash or violate a right-of-way rule).
The "Falsification": In technical terms, finding a way to break the rules is called "falsification." The computer program searches for the perfect combination of speed and direction for the opponent ship that makes the robot captain mess up.
The Lesson: Once the computer finds a scenario where the robot fails, it saves that specific "near-miss" situation.
The Training: The robot captain is then forced to practice only on these difficult, rule-breaking scenarios. It learns, "Oh, when the other ship does this, I must do that to stay safe and follow the rules."

The Analogy: The Chess Grandmaster vs. The Novice

Think of the robot captain as a novice chess player.

Standard Training: The novice plays against random opponents who make random moves. The novice wins a lot, but they never learn how to handle a brilliant, tricky attack.
Falsification Training: A Grandmaster (the computer) plays against the novice. The Grandmaster's only goal is to find a move that traps the novice. Every time the Grandmaster finds a trap, they write it down. Then, the novice practices only those specific traps over and over again until they know exactly how to escape them.

By the end, the novice isn't just good at random games; they are incredibly robust against the most dangerous attacks.

How They Did It (The Technical Magic)

The researchers didn't just guess where the traps were. They used a mathematical language called Signal Temporal Logic (STL).

The Rulebook: They translated the messy, human-written maritime rules into a strict, mathematical code.
The Robustness Score: They gave every situation a "safety score." A high score means the robot is very safe. A low (or negative) score means the robot is breaking a rule or about to crash.
The Optimization: The computer program uses advanced math (like evolutionary algorithms) to tweak the opponent ship's movements to lower that safety score as much as possible. It's like a sculptor chipping away at a block of stone to find the hidden flaw.

What They Found

They tested this on a simulation with two ships: the robot (Ego) and the tricky opponent (Adversary).

The Result: The robot trained with the "Devil's Advocate" method became much better at following the rules than the robot trained with random scenarios.
The Catch: Interestingly, the "Devil's Advocate" robots were slightly worse at just reaching their destination quickly compared to the random-trained robots. Why? Because the random robots learned to ignore the rules to get to the goal fast. The "Devil's Advocate" robots learned that safety and rules come first, even if it means taking a slightly longer path.
The Proof: When they threw the hardest, most confusing traffic situations at the trained robots, the "Devil's Advocate" robots followed the rules correctly about 72% of the time, while the random-trained robots only got it right about 36% of the time.

Summary

The paper shows that to teach an autonomous ship to follow complex traffic rules, you shouldn't just let it practice sailing. You need to actively try to trick it into breaking the rules, show it where it failed, and make it practice fixing those specific failures. This "adversarial" training creates a much safer and more reliable robot captain.

Technical Summary: Falsification-driven Reinforcement Learning for Maritime Motion Planning

Problem Statement
Ensuring autonomous vessels comply with maritime traffic rules, specifically the International Regulations for Preventing Collisions at Sea (COLREGs), is critical for safety but challenging for Reinforcement Learning (RL) agents. The core difficulty lies in the unstructured nature of open-sea environments and the situational, temporal dependencies of traffic rules, which are difficult to encode directly into control policies. Standard RL training often relies on random or non-adversarial scenario generation, which fails to sufficiently expose the agent to the "long-tail" failure modes where rule violations occur. Consequently, agents may learn to prioritize goal-reaching over rule compliance or fail to engage in necessary maneuvers when encountering complex traffic situations.

Methodology
The authors propose a falsification-driven RL framework that integrates optimization-based counterexample generation directly into the RL training loop. The approach operates as follows:

Formal Specifications: Maritime traffic rules (crossing, head-on, and overtaking encounters) are formalized using Signal Temporal Logic (STL). The authors extend existing formalizations with Interface-Aware STL (IA-STL) to separately evaluate the robustness of the antecedent (the encounter condition) and the consequent (the required maneuver). This distinction allows the system to differentiate between "vacuous" satisfaction (where no encounter occurs) and "non-vacuous" compliance (where an encounter occurs and is handled correctly).
Falsification Process: Instead of random sampling, the system employs numerical optimization algorithms (specifically CMA-ES, PSO, and Simulated Annealing) to generate adversarial training scenarios. The objective is to find input sequences for an adversarial vessel that minimize the robustness of the STL specification with respect to the current RL policy.
- The optimization targets the input robustness (to force non-vacuous encounters) and the output robustness (to force rule violations).
- A "Big-M" method is used to prioritize finding non-vacuous scenarios where the agent actually faces a traffic situation.
Training Loop: The generated counterexamples (scenarios where the current policy fails or barely complies) are added to a First-In-First-Out (FIFO) scenario pool. The RL agent, trained using Proximal Policy Optimization (PPO), is then updated using data sampled from this pool, which is enriched with these difficult, rule-violating scenarios.
Robustness Measures: The paper introduces specific robustness measures for atomic predicates (position, orientation, velocity) scaled to the time domain to prevent masking effects in the optimization process. These measures quantify the margin of safety or violation.

Key Contributions

Algorithmic Framework: The authors propose an efficient, practical falsification-driven RL algorithm that generates counterexamples for complex temporal logic traffic rules and utilizes them as training scenarios.
Formalization Extension: The work extends maritime traffic rule formalizations with efficiently evaluable robustness measures, specifically adapting IA-STL to handle the implication structures inherent in COLREGs.
Empirical Validation: The approach is evaluated using multiple falsification methods (CMA-ES, PSO, SA) against a baseline RL agent trained on random scenarios.

Results
Experiments conducted on open-sea navigation scenarios involving two vessels (an ego vessel and an adversarial vessel) demonstrate:

Improved Engagement: Falsification-driven agents exhibit significantly higher "engagement" rates (33–37% vs. 21% for the baseline), meaning they encounter and interact with traffic situations defined by COLREGs more frequently during testing.
Rule Compliance: On a baseline evaluation distribution, falsification-driven agents achieved non-vacuous rule compliance rates of approximately 89–90%, compared to 78.3% for the baseline.
Robustness: The falsification-driven policies maintained higher output robustness values, indicating a larger safety margin. In contrast, the baseline policy showed negative output robustness (indicating violations) when tested on the adversarial (falsification) distribution.
Statistical Significance: Welch's t-tests confirmed that the improvements in input robustness (during training) and output robustness (during testing) are statistically significant ( $p < 0.001$ ).
Trade-off: The results highlight a trade-off where falsification-driven agents sometimes achieve lower goal-reaching rates (e.g., 45% vs. 75% on baseline distribution) in exchange for significantly higher adherence to safety rules, prioritizing correct maneuvering over purely goal-directed behavior in critical situations.

Significance and Claims
The paper claims to be the first application of falsification-driven RL to maritime motion planning. Its primary significance lies in demonstrating that integrating formal methods (falsification) with learning-based control can systematically address the scarcity of critical failure scenarios in training data. By generating adversarial scenarios that specifically target the current policy's weaknesses, the approach produces agents with more consistent rule compliance.

The authors modestly note that while the approach improves compliance, it does not guarantee it in all deployment scenarios. They suggest that for absolute safety guarantees, the learned policy should be combined with a safeguard controller. Furthermore, the current work focuses on satisfiable two-vessel interactions; the authors acknowledge that scaling to multi-agent scenarios with static obstacles requires further development of satisfiable formal rule sets, as naive extensions of two-vessel rules can lead to unsatisfiable constraints. The work positions itself as a step toward bridging the gap between simulation and real-world deployment by ensuring agents are trained on the most relevant, high-risk interactions.