Sim2Sea: Sim-to-Real Policy Transfer for Maritime Vessel Navigation in Congested Waters

The paper proposes Sim2Sea, a comprehensive framework featuring a GPU-accelerated simulator, a dual-stream spatiotemporal policy with safety-guided action masking, and targeted domain randomization, which successfully enables zero-shot sim-to-real transfer of autonomous navigation for a 17-ton unmanned vessel in congested maritime waters.

The Gist

Here is an explanation of the Sim2Sea paper, translated into simple, everyday language with creative analogies.

🚢 The Big Picture: Teaching a Boat to Drive Itself

Imagine you want to teach a toddler how to drive a massive, heavy cruise ship through a crowded harbor filled with other boats, fishing nets, and narrow channels.

If you tried to teach them by throwing them into the real ocean, they would likely crash immediately. It's too dangerous, too expensive, and the real ocean is unpredictable.

Sim2Sea is a new "training school" that solves this problem. It allows a computer brain (an AI) to learn how to navigate a ship entirely inside a video game, and then—magically—step out of the game and drive a real 17-ton boat in the real world without crashing.

The authors call this "Sim-to-Real" transfer. Think of it like a flight simulator for pilots, but for ships, and it actually works perfectly on the first try.

---

🛠️ The Three Secret Ingredients

The researchers built a three-part system to make this magic happen. Here is how they did it:

1. The "Super-Fast" Video Game (The Simulator)

Most video games for ships are slow or fake. They don't feel like real boats. Real boats are heavy, they slide when they turn, and they are affected by wind and currents.

• The Analogy: Imagine trying to learn to ride a bike on a tricycle that doesn't wobble. It's easy, but you'll crash when you get on a real bike.
• The Fix: The team built a GPU-accelerated parallel simulator.
  • "Parallel" means they can run 65,000 different training sessions at the exact same time (like having 65,000 virtual boats learning simultaneously).
  • "High-Fidelity" means the physics are real. The virtual boat slides, turns, and reacts to water exactly like a real one.
  • Result: The AI learns faster than ever before because it gets millions of hours of practice in a single day.

2. The "Super-Eyes" and "Safety Guard" (The Brain)

The AI needs to see the world and make safe choices.

• The Eyes (Dual-Stream Policy):
  • Stream A (Time): The AI looks at a video of the last few seconds to understand momentum. Analogy: A real driver knows that if they are going fast, they can't stop instantly. The AI learns this "inertia."
  • Stream B (Space): The AI looks at a "Bird's Eye View" map (like Google Maps) showing where other boats and land are.
  • The Mix: It combines the "feeling" of movement with the "picture" of the map to make a decision.
• The Safety Guard (VO-Guided Action Masking):
  • The Problem: Sometimes, the AI gets excited and tries to do something crazy, like driving straight into a wall, just to see what happens.
  • The Fix: They installed a "Safety Guard" based on Velocity Obstacles (VO).
  • Analogy: Imagine playing a video game where the game automatically grays out (disables) all the buttons that would make you fall off a cliff. The AI can still choose from the safe buttons, but it cannot even think about the dangerous ones. This stops the AI from learning bad habits.

3. The "Chaos Training" (Domain Randomization)

This is the most important part for the "Sim-to-Real" jump.

• The Problem: Even the best video game isn't perfect. The real ocean has weird currents, sensor noise, and delays. If the AI trains in a "perfect" game, it will fail in the messy real world.
• The Fix: The researchers intentionally broke their simulator during training.
  • Analogy: Imagine a chef practicing a recipe. If they only cook with perfect ingredients in a quiet kitchen, they might fail when the stove is broken or the ingredients are slightly off. So, Sim2Sea trains the AI by randomly changing the water currents, adding fake sensor glitches, and making the boat heavier or lighter every single time.
  • Result: The AI becomes a "champion of chaos." When it finally meets the real ocean, it thinks, "Oh, the current is weird? I've seen weirder in training!" It adapts instantly.

---

🏆 The Results: From Game to Ocean

The team tested this on a 17-ton unmanned boat in real, crowded waters.

• The Test: They took the AI that was only trained in the computer game and put it on the real boat. They did zero extra training on the real boat. This is called "Zero-Shot Transfer."
• The Outcome:
  • The boat navigated smoothly.
  • It avoided collisions with other boats and land.
  • It reached its destination.
  • Crucially: When they tested versions without the "Chaos Training" or without the "Time-Sense," the boats crashed or drove erratically. This proved that both the safety guard and the chaos training were essential.

💡 The Takeaway

Sim2Sea is like a revolutionary driving school. It doesn't just teach a robot how to steer; it teaches it how to feel the water, how to see the future, and how to handle the unexpected. By training in a chaotic, high-speed virtual world, the robot is ready to handle the messy, real world immediately.

This is a huge step forward for autonomous ships, promising safer ports, fewer accidents, and more efficient shipping in the future.

Technical Summary

Here is a detailed technical summary of the paper "Sim2Sea: Sim-to-Real Policy Transfer for Maritime Vessel Navigation in Congested Waters."

1. Problem Statement

Autonomous navigation in congested maritime environments (e.g., ports, coastal fairways) is critical for safety and efficiency but remains a significant challenge due to:

• Complex Dynamics: Vessels are underactuated, possess high inertia, and are subject to unpredictable environmental forces like currents.
• Multi-Modal Perception: Agents must synthesize asynchronous data from AIS, radar, GNSS, and nautical charts.
• Sim-to-Real Gap: Existing Reinforcement Learning (RL) methods often fail in real-world deployment due to discrepancies in simulation fidelity, sensor noise, and actuation delays.
• Safety Constraints: Traditional rule-based systems (COLREGs) and Velocity Obstacle (VO) methods can be indecisive or overly conservative in dense, mixed-traffic scenarios.

The core problem is developing an RL agent that can be trained efficiently in simulation and deployed zero-shot (without fine-tuning) on a physical vessel in real-world congested waters.

2. Methodology: The Sim2Sea Framework

The authors propose Sim2Sea, an integrated framework consisting of three core pillars:

A. High-Performance Parallel Simulator

To address the lack of suitable training environments, the authors built a GPU-accelerated simulator using the Taichi language.

• Physics Models: Supports multiple fidelity levels, including a 3-DOF Maneuvering Modeling Group (MMG) model, a Nomoto yaw-response model, and a lightweight kinematic model.
• Parallelization: Utilizes an agent-centric parallelization strategy (executing a single kernel over $N \times M$ agents) to achieve massive throughput (up to 65,536 agents simultaneously).
• Safety & Interaction: Features Continuous-Time Collision Detection (CCD) to prevent tunneling errors and a hash-grid method for efficient broad-phase collision detection between vessels and complex coastlines.

B. Dual-Stream Spatiotemporal Policy Network

The agent architecture is designed to handle dynamic obstacles and vessel inertia:

• Temporal Encoder: Uses a Transformer to process a sequence of historical observations ($k$ time steps). This allows the agent to infer underlying environmental dynamics (e.g., currents) and vessel momentum, which single-step observations cannot capture.
• Spatial Encoder: Generates a Bird's-Eye-View (BEV) image by fusing data from radar, AIS, and nautical charts. This is processed by a lightweight CNN to provide situational awareness of static and dynamic obstacles.
• Fusion: The temporal and spatial features are concatenated and passed through an MLP decoder to output action logits.

C. VO-Guided Active Action Masking

To ensure safety and improve sample efficiency, the framework employs an Active Action Masking mechanism:

• Instead of relying solely on reward shaping, the system dynamically prunes unsafe actions before the policy selects one.
• It uses an extended Velocity Obstacle (VO) method to check candidate headings against circular obstacles (other ships) and polyline obstacles (coastlines).
• Actions leading to a Time-to-Collision (TTC) below a safety horizon are masked (set to zero probability). This forces the agent to explore only safe regions of the action space.

D. Targeted Domain Randomization

To bridge the sim-to-real gap, the simulation introduces controlled variability:

• Ocean Currents: Modeled as a combination of a low-frequency dominant flow and high-frequency random disturbances. The direction and amplitude are randomized per episode.
• Sensor/Actuation Noise: Random perturbations are applied to observations and command transmissions.
• Goal: This forces the policy to learn invariant features and adapt to unmodeled dynamics rather than overfitting to a specific simulation environment.

3. Key Contributions

1. High-Throughput Simulator: A specialized, GPU-accelerated maritime simulator supporting complex hydrodynamics (MMG) and massive parallel training, filling a gap in open-source maritime RL tools.
2. Novel Agent Architecture: A dual-stream network combining Transformer-based temporal reasoning with BEV spatial perception, augmented by VO-guided active action masking. This ensures safe exploration and faster convergence.
3. Successful Zero-Shot Transfer: The first demonstration of a learning-based policy, trained purely in simulation, successfully navigating a 17-ton unmanned surface vessel in real-world congested waters without fine-tuning.

4. Experimental Results

Simulation Performance

• Efficiency: The parallel simulator achieved a 700x speedup on an NVIDIA A100 GPU compared to CPU-based baselines.
• Ablation Studies: Removing any component (Action Masking, BEV, or Temporal Sequence) significantly degraded performance.
  • Without Action Masking: Higher collision rates and slower convergence.
  • Without BEV: Reduced situational awareness in complex geometries.
  • Without Temporal Sequence: The agent failed to account for inertia, leading to erratic maneuvers.
• Baseline Comparison: Sim2Sea outperformed baselines (VO-RL, COLREG-RL, and pure VO controllers) in both Mini Coastline and Mini Port scenarios, achieving:
  • 93% success rate (Coastline) and 90% (Port).
  • Significantly fewer "unsafe actions" per step compared to reward-shaping baselines.

Real-World Deployment (Sim-to-Real)

• Platform: Deployed on a 17-ton unmanned vessel with twin jet propulsion, running on an onboard NVIDIA RTX 2080.
• Results:
  • Sim2Sea (Full): Successfully navigated to goals in both scenarios with smooth, collision-free trajectories.
  • Ablated Models (Real-World):
    • Without Domain Randomization: Exhibited high-frequency oscillations and brittle behavior, failing to handle real-world currents.
    • Without Temporal Sequence: Failed catastrophically due to an inability to control the vessel's inertia, resulting in collisions.
• Conclusion: The synergy between domain randomization (robustness to uncertainty) and temporal modeling (dynamic awareness) was essential for successful transfer.

5. Significance

This work represents a major advancement in maritime autonomy by solving the "Sim-to-Real" bottleneck.

• Practical Applicability: It proves that complex, learning-based navigation policies can be deployed on full-scale, underactuated vessels in real-world conditions without expensive real-world data collection or fine-tuning.
• Safety Mechanism: The integration of Active Action Masking with RL provides a rigorous safety guarantee that pure reward shaping lacks, making it suitable for high-stakes environments.
• Scalability: The proposed parallel simulator offers a scalable foundation for future research in multi-agent maritime systems and complex hydrodynamic modeling.

In summary, Sim2Sea demonstrates that by combining high-fidelity simulation, robust temporal-spatial architectures, and targeted domain randomization, autonomous vessels can achieve reliable, safe, and efficient navigation in the world's most challenging maritime environments.