🤖 AI

CmIVTP: Cross-modal Interaction-based Vessel Trajectory Prediction for Maritime Intelligence

This paper proposes CmIVTP, a cross-modal interaction framework that fuses sparse AIS data with CCTV environmental features via a target-aware scene encoder and cross-modal transformer to enhance vessel trajectory prediction accuracy, supported by a new large-scale synchronized dataset (Maritime-MmD $^+$ ) and a scalable trajectory generation method.

Original authors: Yuxu Lu, Dong Yang, Xiaoyu Li, Mengwei Bao, Congcong Zhao

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuxu Lu, Dong Yang, Xiaoyu Li, Mengwei Bao, Congcong Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict where a large ship will go next in a busy harbor. In the real world, this is a bit like trying to guess where a friend walking through a crowded, foggy park will end up, but with much higher stakes.

This paper introduces a new system called CmIVTP (Cross-modal Interaction-based Vessel Trajectory Prediction) designed to solve the problem of guessing a ship's future path more accurately than ever before. Here is how it works, broken down into simple concepts:

The Problem: One Eye is Not Enough

Traditionally, ships rely on a single source of information to track movement, much like trying to navigate a dark room using only a flashlight.

The Flashlight (AIS): Ships carry a system called AIS (Automatic Identification System) that broadcasts their location. However, this signal can be weak, get lost, or sometimes the ship turns it off (like a "dark vessel"). It's like a flashlight that flickers or goes out completely.
The Eyes (CCTV): Ports also have cameras (CCTV) watching the water. These provide a clear visual of what is happening right now, but they can't see through fog, and they don't know the ship's speed or history as well as the AIS does.

If you only use the flashlight, you might lose the ship in the fog. If you only use the eyes, you might miss the ship's speed or intentions. The authors realized that to get a perfect prediction, you need to combine both.

The Solution: A Super-Brain for Ships

The authors built a "super-brain" (an AI framework) that acts like a detective who uses both the flashlight and the camera simultaneously. Here are the four main tools this detective uses:

1. The Scene Detective (Visual Scene Target-aware Encoder)
Before guessing where the ship goes, the system looks at the "scene." It doesn't just see a dot on a screen; it understands the environment.

Analogy: Imagine a driver looking at the road. They don't just see the car in front; they see the curve of the road, the other cars, and the traffic lights. This module does the same for ships, understanding how the water, other boats, and the harbor shape limit where a ship can go.

2. The Translator (Cross-modal Interaction Transformer)
This is the core magic. The system takes the "flashlight" data (AIS) and the "camera" data (CCTV) and forces them to talk to each other.

Analogy: Think of two people speaking different languages trying to solve a puzzle. One knows the history (AIS), and the other sees the current picture (CCTV). This module acts as a perfect translator, merging their stories so that if the flashlight flickers out, the camera fills in the gaps, and vice versa. It ensures the prediction makes sense both physically and visually.

3. The "What If" Generator (Uncertainty-aware Variational Decoder)
Ships are huge and heavy; they don't turn on a dime. But they also have human captains who might make sudden decisions. The system knows it can't be 100% sure of the future.

Analogy: Instead of drawing one single line for where the ship will go, this module draws five or ten different possible paths at once. It's like a weather forecast that says, "There's a 60% chance of rain, but it could also be sunny." It generates a "cloud" of possibilities to account for uncertainty, ensuring that even if the ship does something unexpected, the prediction covers it.

4. The Memory Bank (Vessel Group Trajectory Bank)
The system has a massive library of past ship movements.

Analogy: Before making a guess, the system asks, "Has a ship ever done this exact maneuver before?" It looks up similar patterns in its history book. If it sees a ship turning sharply near a bridge, it checks its library to see how other ships handled that same turn. It uses these "historical ghosts" to refine its guess, making it smarter and faster.

The New Dataset: The Training Ground

To teach this system, the authors created a new, massive dataset called Maritime-MmD+.

Analogy: Imagine a driving school that finally got a simulator that records both the GPS data and the video feed at the exact same time. Before this, researchers had to guess how to match the two. This new dataset provides the perfect "training ground" where the GPS and the video are perfectly synchronized, allowing the AI to learn the true relationship between the ship's signals and what the cameras see.

The Results: Why It Matters

The authors tested their system against many other methods (like standard AI models that only use GPS or only use video).

The Flashlight Test: When they simulated the AIS signal disappearing (turning off the flashlight), the old systems got lost and made wild, inaccurate guesses. The new CmIVTP system, however, kept its cool. Because it was still looking at the camera, it could still predict the ship's path accurately.
The Crowd Test: In very busy waters with many ships, the new system handled the chaos better than anyone else, predicting paths that avoided collisions more effectively.

In Summary

This paper presents a smarter way to predict where ships will go by fusing two different types of data (radio signals and video) so that if one fails, the other saves the day. It uses a "scene detective" to understand the environment, a "translator" to mix the data, a "what-if generator" to handle uncertainty, and a "memory bank" to learn from the past. The result is a system that is much safer and more reliable, even when the data is messy or incomplete.

Technical Summary: CmIVTP – Cross-modal Interaction-based Vessel Trajectory Prediction

1. Problem Statement

Maritime Intelligent Transportation Systems (MITS) require accurate, real-time vessel trajectory prediction to ensure navigation safety and efficiency. However, current approaches face significant limitations due to reliance on single-source data:

AIS Limitations: Automatic Identification System (AIS) data, while providing macroscopic kinematics, is often sparse, suffers from transmission delays, and can be unavailable for small or non-cooperative vessels (e.g., "dark vessels").
CCTV Limitations: Closed-Circuit Television (CCTV) provides rich local semantics and environmental context but struggles with occlusion, imaging interference, and cannot fully capture dynamic vessel behavior in isolation.
Domain Gap: Existing multimodal methods often transfer land-based models (designed for lane-constrained vehicles) to the maritime domain, failing to account for the unstructured, open-water environment where vessel movements are governed by complex hydrodynamics and implicit interactions.
Data Scarcity: There is a lack of synchronized, large-scale datasets combining AIS and CCTV data, hindering the training of robust multimodal models.

2. Methodology: The CmIVTP Framework

The authors propose CmIVTP (Cross-modal Interaction-based Vessel Trajectory Prediction), a framework designed to fuse heterogeneous AIS and CCTV data to model intricate vessel-environment interactions. The architecture consists of four primary modules:

A. Visual Scene Target-aware Encoder (VSTaE)

This module extracts spatiotemporal representations from CCTV scene data to capture vessel-environment interactions (e.g., proximity to hazards, other vessels).

Target-aware Spatial Feature Extraction: Uses a ResNet-18 backbone to generate global feature maps. It fuses three feature types: target-specific embeddings (via RoI Align), global context (via global average pooling), and bounding box positional encodings.
Temporal Context Modeling: Processes the sequence of feature maps using a ConvLSTM to capture temporal evolution. An exponential weighting mechanism is applied to prioritize short-term dynamics (sudden maneuvers) while retaining long-term trends (high inertia).
Spatiotemporal Fusion: Concatenates spatial and temporal features via an MLP to produce a unified scene representation.

B. Cross-modal Interaction-based Encoder (CmIE)

To avoid naive feature concatenation, CmIE employs a Cross-modal Interaction Transformer (CMIT) to deeply align vessel dynamics with visual constraints.

Asymmetric Architecture: It treats AIS-derived motion features as the primary query and CCTV/scene features as memory.
Mechanism: The module sequentially performs intra-modal refinement (Self-Attention) to ensure trajectory smoothness, followed by cross-modal interaction (Cross-Attention) to inject environmental constraints into the motion representation. This allows the model to capture both intra-modal semantics and inter-modal interactions.

C. Uncertainty-aware Variational Decoder (UaVD)

Moving beyond deterministic mapping, this module models trajectory prediction as a probabilistic distribution problem to account for stochastic vessel behaviors.

Latent Space Modeling: Based on a Conditional Variational Autoencoder (CVAE) framework, it samples latent variables ( $z$ ) representing unobserved factors (e.g., human intent, sudden disturbances).
Multi-modal Generation: By sampling different $z$ values, the decoder generates diverse, physically plausible future trajectories (modes), such as maintaining course, turning, or slowing down.
Architecture: Uses an MLP to decode the entire future sequence simultaneously based on the fused features and sampled latent variables.

D. Vessel Group Trajectory Bank (VGTB)

To enhance efficiency and accuracy, the framework constructs a bank of representative motion patterns from historical AIS data.

Clustering: Historical AIS trajectories are normalized and clustered (using K-means) into representative motion patterns.
Retrieval and Refinement: For a target vessel, the system retrieves the most similar historical trajectory (prior) from the bank. This prior is fused with the base network prediction using a learnable offset and a dynamic gating mechanism, allowing the model to leverage historical priors while adapting to current conditions.

E. Dataset: Maritime-MmD+

The authors introduce Maritime-MmD+, a large-scale, synchronized dataset containing AIS data and CCTV video from the Huangpu River in Shanghai. It covers diverse scenarios including bridge areas and curved waterways with varying vessel densities, providing a robust benchmark for multimodal trajectory prediction.

3. Key Contributions

Novel Framework: Proposal of CmIVTP, which integrates AIS dynamics and CCTV context via cross-modal interaction to address data sparsity and missingness.
Architectural Innovations:
- VSTaE: Captures fine-grained vessel-environment interactions.
- CmIE: Uses asymmetric cross-attention to fuse kinematic and visual features.
- UaVD: Models trajectory uncertainty via variational inference for multi-modal predictions.
- VGTB: Utilizes historical motion priors to refine predictions.
Dataset Creation: Release of Maritime-MmD+, a synchronized AIS-CCTV dataset to support multimodal research.
Performance: Demonstration of superior accuracy and robustness compared to state-of-the-art baselines across various prediction horizons, vessel densities, and AIS missing rates.

4. Experimental Results

Extensive experiments were conducted on the Maritime-MmD+ dataset, comparing CmIVTP against deterministic (GRU, LSTM, TCNN, Transformer) and uncertainty-aware (VAE, GAN) baselines.

Prediction Accuracy: CmIVTP achieved the lowest Average Displacement Error (ADE) and Final Displacement Error (FDE) across all prediction steps ( $\Delta t = 12, 24, 36$ ). For instance, at $\Delta t = 36$ , CmIVTP achieved an ADE of $0.65 \times 10^{-2}$ , significantly outperforming the next best baseline (Trans., $0.86 \times 10^{-2}$ ).
Robustness to Data Loss: The model demonstrated exceptional resilience when AIS data was missing (simulated rates $\rho = 0.1, 0.2, 0.3$ ). While baselines suffered severe performance degradation, CmIVTP maintained stable convergence and low error margins, effectively compensating for missing AIS signals using CCTV visual features.
Density Handling: In high-density scenarios ( $\phi_h$ ), where vessel interactions are complex, CmIVTP maintained the lowest error rates, successfully capturing non-linear deviations caused by collision avoidance maneuvers.
Ablation Studies:
- Removing any single modality (AIS, CCTV, or Scene Reps) or module (VSTaE, CmIE, UaVD, VGTB) resulted in performance degradation, confirming the necessity of the full multimodal architecture.
- The VGTB module was shown to effectively refine predictions, particularly for low-similarity (rare) trajectory cases, without introducing harmful biases.
Uncertainty Analysis: Visualizations of the latent space showed structured organization corresponding to motion attributes (e.g., tortuosity, displacement). The model produced compact distributions for complete data and appropriately diverse distributions for missing data, ensuring ground truths remained within the predicted range.

5. Significance and Claims

The paper claims that CmIVTP represents a significant advancement in maritime intelligence by overcoming the inherent limitations of single-source data. Its primary significance lies in:

Reliability in Signal-Denied Environments: By leveraging the redundancy of multimodal data, the system ensures continuous and reliable trajectory forecasting even when AIS signals are lost or unavailable.
Context-Aware Modeling: The framework successfully models the complex interplay between vessel dynamics and environmental constraints, which is critical for safety in congested and restricted waterways.
Foundational Benchmark: The introduction of Maritime-MmD+ addresses a critical gap in the field, providing a standardized resource for future research in multimodal maritime behavior analysis.

The authors conclude that while challenges remain regarding extreme environmental conditions (e.g., heavy fog) and generalization to new waterways, CmIVTP provides a robust, scalable solution for enhancing maritime safety and supporting the evolution of Maritime Autonomous Surface Ships (MASS).