WaterVideoQA: ASV-Centric Perception and Rule-Compliant Reasoning via Multi-Modal Agents

This paper introduces WaterVideoQA, a large-scale video question answering benchmark for all-waterway environments, and NaviMind, a multi-agent neuro-symbolic system that enables Autonomous Surface Vessels to transition from passive perception to regulation-compliant, interpretable cognitive reasoning through adaptive semantic routing and self-reflective verification.

The Gist

Imagine you are teaching a robot to drive a boat.

The Problem: The "Passive Observer" vs. The "Smart Captain"
Right now, most autonomous boats are like passive observers. They have great eyes (cameras) and can tell you, "Hey, there's a red buoy!" or "There's a big ship over there!" But they are terrible at thinking. They don't understand why that buoy is there, or what they should do about the other ship.

It's like having a passenger who can describe the scenery but doesn't know the rules of the road. If a car is coming toward them, the passenger might say, "Oh, a car is coming," but they won't say, "We need to steer right because the rules say so!" This is dangerous in the real world, where waterways are messy, weather changes fast, and ships have strict laws (like traffic rules, but for water) to prevent crashes.

The Solution: Two Big Innovations
The authors of this paper built two things to fix this: a giant practice test and a smart thinking team.

1. The Practice Test: "WaterVideoQA"

Think of this as the SATs or the Driver's License Exam for boat robots.

• Before: Robots were only tested on static pictures (like a photo of a tree). But boats move! You need to see a video to understand if a ship is coming toward you or going away.
• The New Test: They created a massive library of 3,000+ video clips from all kinds of water: rivers, lakes, oceans, canals, and harbors.
• The Questions: Instead of just asking "What is that?", the test asks complex questions like:
  • "Is that ship going to hit us?" (Prediction)
  • "Who has the right of way?" (Rules)
  • "Why should we turn left?" (Reasoning)
• The Levels: The test has 5 levels of difficulty, starting from "I see a boat" (Level 1) all the way to "I know the international laws and can explain exactly why we must yield" (Level 5).

2. The Smart Team: "NaviMind"

This is the robot's brain. Instead of one giant, slow computer trying to do everything at once, the authors built a team of specialized agents (like a small office staff) that work together.

Imagine a Maritime Law Firm inside the boat:

• The Receptionist (Router Agent): When you ask a question, this agent decides who handles it.

  • Simple question: "Is the water calm?" -> Sends it to the Fast Vision team (instant answer).
  • Complex question: "Do we need to yield?" -> Sends it to the Lawyer team (takes time to think).
  • Analogy: It's like a doctor triaging patients. A cold gets a quick check; a broken leg gets a specialist. This saves time and energy.

• The Librarian (Knowledge RAG): This agent has a massive digital library of maritime laws (the "Rulebook"). It doesn't guess; it looks up the specific rule for the situation.

  • Analogy: If you ask a human, "What's the speed limit?" they might guess. The Librarian opens the book and says, "Page 42, Section B: 15 knots."

• The Detective (Reasoner Agent): This is the main thinker. It combines what the camera sees (the video) with what the Librarian found (the rules).

  • Analogy: The Detective looks at the video, sees a boat on the left, checks the rulebook, and concludes: "The rule says if a boat is on the left, we must turn right. So, we are turning right."

• The Quality Control Inspector (Self-Reflective Agent): Before the boat moves, this agent double-checks the Detective's work.

  • Analogy: It's like a spell-checker, but for safety. If the Detective says, "We should crash into that rock," the Inspector screams, "Wait! That violates the rules! Let's re-think this!" This prevents the robot from "hallucinating" (making up crazy answers).

Why This Matters

The paper shows that this new system is smarter, faster, and safer than previous models.

• It follows the law: It doesn't just guess; it cites the rules.
• It understands time: It watches the video flow, not just a single frame.
• It admits mistakes: If it's unsure, it checks its work before acting.

In a Nutshell:
Previous boats were like tourists with cameras who could describe the view but didn't know how to drive. This new system, NaviMind, is like a professional captain who has a team of experts, a library of laws, and a safety inspector, all working together to navigate safely through any storm or crowded harbor.

Technical Summary

1. Problem Statement

Autonomous Surface Vessels (ASVs) have achieved significant success in passive perception tasks (e.g., object detection, segmentation) but remain fundamentally limited by a lack of knowledge-driven, interactive cognitive reasoning.

• The Gap: Current systems act as static observers, relying on superficial pattern matching. They fail to decode underlying causalities, dynamic interactions, and complex maritime regulations (e.g., COLREGs) necessary for safe navigation.
• The Challenge: Maritime environments are highly dynamic, featuring irregular waterways, volatile weather, and unpredictable obstacle movements. Merely detecting an object (e.g., "a ship ahead") is insufficient; the system must understand why a maneuver is needed (e.g., "give way because we are in a narrow channel under Rule 14").
• Existing Limitations: Prior datasets and models are often restricted to single frames (lacking temporal context), confined to specific waterway types (inland vs. open sea), or lack integration with professional navigation rules, leading to hallucinations and non-compliant decisions.

2. Methodology

The authors propose a two-pronged solution: a comprehensive benchmark dataset and a novel neuro-symbolic multi-agent system.

A. WaterVideoQA Benchmark

The first large-scale Video Question Answering (VideoQA) dataset designed specifically for all-waterway environments.

• Scale & Diversity: Contains 3,029 video clips and 3,673 QA pairs covering six distinct waterway categories: River, Lake, Canal, Moat, Harbor, and Sea.
• Five-Tier Cognitive Hierarchy: The dataset evaluates models across a progressive cognitive framework:
  1. Perception: Basic object identification.
  2. Scene Understanding: Contextual analysis.
  3. Action/Interaction: Immediate maneuver decisions.
  4. Causal & Predictive: Reasoning about future states and collision risks.
  5. Knowledge-Driven: Rule-compliant reasoning based on maritime laws.
• Annotation Quality: A rigorous human-machine collaborative pipeline involving PhD annotators and external experts, ensuring high objectivity (Fleiss' Kappa $\kappa = 0.86$) and strict adherence to visual evidence and regulations.

B. NaviMind: Multi-Agent Neuro-Symbolic System

A framework designed to transition ASVs from pattern matching to regulation-compliant reasoning. It consists of five specialized agents orchestrated by three core mechanisms:

1. Adaptive Semantic Routing (ASR):

   • A lightweight Router Agent analyzes user intent to dynamically select the inference path, balancing latency and reasoning depth.
   • Paths: Fast Vision (for simple perception), Fast RAG (for direct knowledge queries), and Complex Reasoning (for high-order causal tasks).

2. Situation-Aware Hierarchical Reasoning (SAHR):

   • Adaptive Temporal Standardization (ATS): Normalizes variable-length video streams into fixed key frames to handle temporal heterogeneity.
   • Context Fusion: Synthesizes visual tokens, scene captions, and retrieved maritime regulations.
   • Rule-Constraint Injection: Uses a Retrieval-Augmented Generation (RAG) module to fetch specific regulations (e.g., COLREGs, IALA buoyage) based on the visual scene and query, ensuring the reasoning is legally grounded.

3. Autonomous Self-Reflective Verification:

   • A Grader Agent evaluates the initial response against retrieved rules.
   • If the confidence score is low, a Self-Reflective Loop triggers: the system expands the retrieval scope and regenerates the answer to mitigate hallucinations and ensure logical consistency.

3. Key Contributions

1. WaterVideoQA Dataset: The first comprehensive VideoQA benchmark for maritime navigation, spanning diverse waterways and covering a five-level cognitive hierarchy with 3,029 videos.
2. NaviMind Framework: The first multi-agent neuro-symbolic system for open-ended maritime reasoning, integrating visual perception with professional rule retrieval.
3. Novel Mechanisms:
   • SAHR: A reasoning engine that anchors visual evidence to dynamic regulatory constraints.
   • Self-Reflective Verification: A closed-loop mechanism that detects and corrects hallucinations before outputting guidance.
4. Efficiency & Safety: Demonstrates that complex reasoning can be achieved with high efficiency (via adaptive routing) and strict safety compliance (via rule grounding).

4. Experimental Results

The system was evaluated on WaterVideoQA and the generalization benchmark LingoQA (autonomous driving).

• Performance on WaterVideoQA:
  • NaviMind (7+14B parameters) achieved a GPT-Score of 0.602, significantly outperforming state-of-the-art baselines like InternVL3-38B (0.524) and VideoAgent-24B (0.471).
  • It excelled in Causal Prediction and Knowledge-Driven Reasoning, domains where pure MLLMs typically fail due to hallucinations.
  • Efficiency: The adaptive routing allowed NaviMind-11B to run 2x faster (9.74s) than larger agents while maintaining superior accuracy.
• Generalization (LingoQA):
  • In a zero-shot setting, NaviMind outperformed specialized video agents on road-based driving tasks, proving the universality of its neuro-symbolic reasoning logic.
  • After fine-tuning, it set new SOTA records with a CIDEr score of 93.77.
• Qualitative Analysis: Visualizations showed NaviMind providing precise, regulation-cited maneuvers (e.g., "Turn starboard per COLREGs Rule 14") whereas baselines offered vague or incorrect suggestions.

5. Significance and Impact

• Paradigm Shift: Moves maritime AI from "passive perception" to "active, rule-compliant cognition," addressing the critical safety gap in autonomous navigation.
• Trustworthiness: By grounding decisions in retrieved professional regulations and employing self-reflection, the system offers interpretable and verifiable guidance, essential for high-stakes safety-critical operations.
• Scalability: The multi-agent architecture allows for efficient resource allocation, making it feasible for deployment on edge devices with limited compute power.
• Future Directions: The authors identify limitations in extreme weather (fog/night) and multi-vessel game-theoretic interactions, proposing future integration of multi-sensor fusion (LiDAR/Radar) and multi-agent reinforcement learning.

In conclusion, WaterVideoQA and NaviMind establish a new paradigm for intelligent maritime systems, proving that combining large-scale video understanding with neuro-symbolic rule reasoning is the key to achieving safe, reliable, and autonomous surface vessel navigation.