💻 computer science

OGR-MARL: Option-Guided Residual Multi-Agent Reinforcement Learning for Heterogeneous USV Cooperative Pursuit in Constrained Port Waterways

This paper proposes OGR-MARL, a decoupled option-guided residual multi-agent reinforcement learning framework that enhances heterogeneous USV cooperative pursuit in constrained port waterways by enabling rule-guided agents to learn corrective actions, achieving high capture rates and zero-shot generalization across various MARL backbones.

Original authors: Mao Jiayang, Wang Lanfeng, Peng Zhao-Han

Published 2026-08-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Mao Jiayang, Wang Lanfeng, Peng Zhao-Han

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots learn to think and act not just by following a rigid instruction manual, but by playing a game, trying things out, and learning from their mistakes. This is the heart of Multi-Agent Reinforcement Learning (MARL). Think of it like a group of friends learning to play a complex team sport together. Instead of one coach telling everyone exactly what to do every second, each player watches the game, figures out what their teammates are doing, and learns the best moves through trial and error. The "Multi-Agent" part just means there's a whole team of these learning robots working together, not just one lone wolf.

Now, imagine taking that team of learning robots and putting them in a very tricky, crowded environment: a busy harbor. This is where things get messy. In an open ocean, robots have plenty of room to move. But in a port, they have to dodge other ships, avoid hitting the shore, follow strict traffic lanes, and work together to catch a fast-moving "evader" (a boat trying to escape). This is a high-stakes game of tag where the rules are strict, the space is tight, and the other team is smart. Scientists care about this because if we can teach robots to navigate these crowded, dangerous waters safely and effectively, we could use them for real-world jobs like security patrols, environmental monitoring, or search-and-rescue missions without risking human lives.

The Paper's Big Idea: The "Coach" and the "Player"

This paper introduces a new way to teach these robot teams how to play the game of "Catch the Evader" in a busy harbor. The authors call their new system OGR-MARL. To understand what makes it special, let's use an analogy.

Imagine you are teaching a group of new soccer players how to play in a tiny, crowded backyard. If you just tell them, "Go kick the ball!" and let them figure it out on their own (which is what standard AI learning often does), they might spend hours running into the fence, tripping over each other, or kicking the ball into the neighbor's yard. They might never learn the game because the rules are too confusing and the mistakes are too costly.

On the other hand, if you give them a strict coach who tells them exactly where to run and when to kick, they will follow the rules perfectly. But here's the catch: if the other team (the evader) suddenly changes their strategy or runs in a weird direction, the strict coach's plan might fail because it can't adapt quickly enough.

OGR-MARL is the perfect mix of both. It's like having a smart coach who gives the players a general game plan (the "Option Guidance") while letting the players use their own brains to make small, quick adjustments (the "Residual Learning").

How It Works: The Team of Robots

In this study, the robot team is made up of different types of boats, called USVs (Unmanned Surface Vehicles). They aren't all the same; they are a "heterogeneous" team, meaning they have different superpowers:

The Scouts: These are slower boats but they have giant eyes (a huge sensing radius). Their job is to spot the evader from far away and keep track of where it is.
The Interceptors: These are fast boats with small eyes. They can't see far, but they are the only ones fast enough to actually catch the evader.

The goal is to catch the evader before it escapes the harbor, all while obeying strict rules: don't hit the shore, don't crash into other cargo ships, and stay in the correct traffic lanes.

The Secret Sauce: Option-Guided Residual Learning

The paper's main innovation is how it combines the "Coach" and the "Player."

The Coach (Option Guidance): The system uses a simple, rule-based algorithm (like a basic map and a set of traffic laws) to suggest a high-level goal. For example, the coach might say, "Scout, go look for the target," or "Interceptor, move to block the exit." This gives the robots a safe starting point so they don't waste time crashing into walls.
The Player (Residual Learning): This is where the AI magic happens. The robots don't just blindly follow the coach. They have a "residual" brain that learns to make tiny, corrective adjustments. If the coach says "Go left," but the evader suddenly darts right, the robot's brain says, "Okay, I'll go left mostly, but I'll tweak my angle slightly to the right to catch it."

This "Residual" part is crucial. It allows the robots to learn complex, cooperative behaviors—like the Scouts keeping the target in sight while the Interceptors sneak up for the catch—without having to learn everything from scratch in a chaotic environment.

What They Found: The Results

The researchers tested their OGR-MARL system in a simulated harbor (based on a real place called Xiazhimen in China). They pitted their new system against older methods and found some exciting results:

The Old Ways Struggled: When they tried using just the strict rules, the robots caught the evader only 35% of the time. When they tried using just the AI learning without the coach's help, the success rate was terrible (sometimes as low as 1% or 2%). The AI got lost, crashed, or gave up.
The New Way Succeeded: When they combined the coach and the AI, the success rate skyrocketed. The best version of their system, called OGR-MASAC, managed to catch the evader 75% of the time.
It Followed the Rules: Not only did they catch the evader more often, but they also followed the harbor rules much better. They didn't crash into the shore or other ships as often as the other methods.
It Worked on a Real Map (Without Retraining): The most impressive part? The researchers took the system trained on a simple, abstract map and dropped it straight onto a detailed, real-world map of the harbor (using real ship traffic data). They didn't retrain the robots at all. Even in this "zero-shot" transfer, the system still caught the evader about 66.7% of the time. This suggests the system is smart enough to handle real-world messiness without needing to be taught again.

Why This Matters

The paper suggests that by giving AI robots a "safety net" of rules and high-level goals, we can teach them to solve incredibly difficult teamwork problems much faster and more reliably. It's not just about making robots that can catch a boat; it's about creating a framework where different types of robots can learn to work together in complex, real-world environments like busy ports.

While the results are based on simulations (and the real-world test was still a simulation using real data), the authors show that this approach is a promising step forward. It bridges the gap between rigid, rule-based safety and flexible, smart AI, proving that sometimes the best way to teach a robot is to give it a little guidance and let the rest of the learning happen on the fly.

Technical Summary: OGR-MARL for Heterogeneous USV Cooperative Pursuit

1. Problem Formulation

The paper addresses the challenge of heterogeneous Multi-USV (Unmanned Surface Vehicle) cooperative pursuit within constrained port waterways. Unlike open-sea scenarios, port environments impose strict navigation constraints, including narrow lanes, anchorage zones, buffer regions, and bidirectional traffic flows.

The specific problem involves a team of four heterogeneous pursuers tasked with intercepting an evader attempting to escape to an eastern boundary. The team consists of:

Interceptors ( $P_1, P_2$ ): High-speed USVs with capture capability but limited sensing radius.
Scouts ( $P_3, P_4$ ): Slower USVs with large sensing radii, tasked with maintaining target visibility and updating a shared belief state.

The evader is modeled as a mass point utilizing an A* global planner and an Artificial Potential Field (APF) local collision-avoidance module to navigate around shores, moving cargo vessels, and pursuers. The environment is partially observable; only scouts can detect the evader at long range, requiring the team to maintain a shared belief state that decays over time if not updated.

The core difficulty lies in balancing rule compliance (avoiding shores, traffic violations, and anchorage intrusions) with mission effectiveness (capturing a tactical, replanning evader) under partial observability and heterogeneous dynamics.

2. Methodology: OGR-MARL

The authors propose OGR-MARL (Option-Guided Residual Multi-Agent Reinforcement Learning), an algorithm-agnostic framework designed to decouple rule-guided behaviors from the learning process. The framework operates under a Centralized Training and Decentralized Execution (CTDE) paradigm.

Key Architectural Components:

Option Guidance Layer:
- A lightweight, rule-based module that assigns high-level "options" (modes) to agents based on their role, belief confidence, and rule-compliance state.
- Options: {search, track, intercept, block, recover}.
- Function: Scouts select search or track to maintain coverage; Interceptors select intercept or block to form pincer maneuvers; recover guides agents away from violations.
- This layer generates a geometric target and a high-level action ( $a^{opt}$ ) independent of the specific MARL algorithm used.
Residual Policy Learning:
- A generic MARL backbone (e.g., MADDPG, MATD3, MAPPO, MASAC) learns a residual action ( $a^{rl}$ ) conditioned on local observations.
- The final executed action ( $a_i$ ) is a bounded blend of the rule-guided option action and the learned residual action:
  $a_i = \text{clip}((1 - \beta_i)a^{rl}_i + \beta_i a^{opt}_i, -1, 1)$
- This allows the MARL agent to learn corrective actions (timing, local motion adjustments, coordination) on top of safe, rule-compliant priors, rather than exploring the constrained environment from scratch.
Structured Reward and Curriculum Learning:
- Reward Shaping: The reward function ( $r_{ind}$ $r_{in d}$ ) is decomposed into:
  - Task-progress: Encourages harbor departure, tracking, and pincer formation.
  - Rule-compliance: Penalizes shore/anchorage intrusion and traffic violations.
  - Heterogeneous-role: Promotes complementary behaviors (scouts tracking, interceptors closing distance).
  - Terminal: Rewards capture or penalizes escape/timeout.
- Curriculum Learning: Training proceeds through five stages ( $S_0$ to $S_4$ ), gradually increasing evader speed, tightening capture radius, and strengthening rule penalties to guide agents from basic behaviors to precise, compliant coordination.

3. Key Contributions

The paper outlines three primary contributions:

Benchmark Scenario: The establishment of a realistic, constrained port waterway scenario (abstracted Xiazhimen port) featuring heterogeneous USVs, moving traffic, anchorage zones, and partial observability.
OGR-MARL Framework: The proposal of an algorithm-agnostic framework that integrates shared evader belief, role-conditioned option targets, adaptive rule penalties, and residual policy learning. It allows different MARL backbones to learn corrective actions atop rule-guided behaviors.
Comprehensive Evaluation: A rigorous evaluation involving backbone comparisons, reward ablation studies, and zero-shot transfer to a real-world map informed by QGIS and AIS (Automatic Identification System) data.

4. Experimental Results

Experiments were conducted in an abstract 1000x400 domain and transferred to a real-world map without retraining.

Performance Comparison:
- Pure rule-based controllers achieved a 35.0% capture rate.
- Pure MARL baselines (without option guidance) performed poorly (1.0% – 16.0% capture rates), struggling with sparse rewards and rule violations.
- OGR-MARL variants significantly outperformed their pure counterparts.
- OGR-MASAC (OGR instantiated with Soft Actor-Critic) achieved the best performance with a 75.0% capture rate, the lowest mean task time cost (121.94s), and the highest Heterogeneous Coordination Score (HCS: 0.6802).
- OGR-MATD3 also showed significant improvement, rising from 1.0% to 59.0%.
Ablation Studies:
- Removing rule or role rewards significantly degraded performance (e.g., OGR-MASAC without rules dropped to 31.0% SR), confirming the necessity of structured rewards for constrained environments.
Zero-Shot Transfer:
- When transferred to a QGIS/AIS-informed real map of Xiazhimen without retraining, OGR-MASAC achieved a 66.67% capture rate.
- While lower than the abstract simulation, this demonstrates generalization potential despite challenges like curved coastlines and irregular traffic patterns.
Failure Analysis:
- Failures were primarily attributed to inefficient terminal close-in interception rather than a total loss of target visibility. Successful episodes showed interceptors getting significantly closer to the evader (mean distance 4.17) compared to failed episodes (8.19).

5. Significance and Claims

The paper claims that OGR-MARL effectively bridges the gap between the interpretability of rule-based methods and the adaptability of reinforcement learning. By decoupling the MARL algorithm from the specific control logic, the framework allows agents to:

Learn corrective actions on top of safe, rule-guided behaviors, avoiding the inefficiency of exploring constrained environments from scratch.
Achieve heterogeneous coordination where scouts and interceptors perform complementary roles effectively.
Generalize to complex, real-world port scenarios via zero-shot transfer, suggesting the framework is robust to environmental variations without retraining.

The authors conclude that while the method shows promise, future work should focus on improving sample efficiency using world models and validating performance in even more realistic or physical maritime environments.