🤖 machine learning

A Novel Evaluation Metric for Unsupervised Learning in AIS-Based Maritime Anomaly Detection: MADQI

This paper proposes the Maritime Anomaly Detection Quality Index (MADQI), a novel unsupervised evaluation framework that integrates four metrics to systematically assess anomaly detection performance in maritime AIS data without requiring labeled datasets.

Original authors: Ismet Gocer, Zakirul Bhuiyan, Raza Hasan, Shakeel Ahmad

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ismet Gocer, Zakirul Bhuiyan, Raza Hasan, Shakeel Ahmad

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world's oceans as a giant, busy highway where thousands of ships travel every day. To keep everyone safe, these ships wear "digital name tags" called AIS (Automatic Identification System) that constantly shout out their location, speed, and direction to the world.

However, sometimes things go wrong. A ship might suddenly teleport across the map (a "position jump"), vanish from the radar for hours (a "time gap"), or spin in circles at impossible speeds. These are anomalies. Finding them is like looking for a single red car in a sea of white ones, but with a twist: no one knows exactly which cars are supposed to be red. There is no "answer key" or list of bad ships to compare against.

This paper introduces a new way to check if our computer programs are actually doing a good job of finding these "red cars" without needing an answer key.

The Problem: The "Black Box" of Judgment

The researchers used a popular computer program called Isolation Forest to find these weird ships. Think of Isolation Forest as a very efficient librarian who sorts books by throwing them into different piles until the weird ones end up alone in their own tiny pile.

The problem? The librarian can tell you, "This book is weird," but there was no standard ruler to measure how weird it is or if the librarian is actually doing a good job. Before this paper, checking the librarian's work was mostly a matter of guessing or asking an expert, "Does this look right to you?" which is subjective and unreliable.

The Solution: MADQI (The "Report Card")

The authors created a new scoring system called MADQI (Maritime Anomaly Detection Quality Index). Think of MADQI as a report card for the computer program. Instead of just giving a single grade, it breaks the grade down into four specific subjects to see exactly where the program is excelling or struggling:

ARC (Anomaly Rate Consistency):
- The Analogy: Imagine a teacher expecting 1% of the class to be late. If the teacher flags 50% of the class as late, something is wrong.
- What it checks: Does the computer flag roughly the right amount of ships as suspicious? If it flags too many or too few, this score drops.
PPS (Physical Plausibility Score):
- The Analogy: If a car claims to drive 500 mph, that's physically impossible.
- What it checks: Do the flagged ships actually do things that break the laws of physics? For example, did the ship jump 100 miles in one second? If the computer flags a ship that just moved normally, this score goes down.
SDS (Score Distribution Separation):
- The Analogy: Imagine a classroom where the "bad" students and "good" students all get the same test score. You can't tell them apart.
- What it checks: Is there a clear gap between the "weird" ships and the "normal" ships? The computer needs to give very different scores to the two groups so they don't get mixed up.
ECE (Extreme Case Evidence):
- The Analogy: If you are looking for a criminal, you want to be sure you caught the ones who did the most heinous crimes, not just the ones who jaywalked.
- What it checks: Did the computer successfully catch the most extreme weird behaviors (like massive position jumps or huge time gaps)? This ensures the system is catching the real dangers.

How They Tested It

The researchers took a massive chunk of real ship data from the US coast (over 1 million ship reports) and ran their program on it. To make sure the test was fair, they didn't just look at the whole pile at once. They chopped the data into five smaller "chunks" (like tasting a soup from five different spoons) to ensure the score wasn't just lucky on one specific day.

They used a clever math trick (exponential normalization) to combine these four scores into one final number, much like calculating a GPA.

The Results

The computer program got a MADQI score of 80.37%.

The Good News: It was perfect at catching the "extreme" cases (ECE score of 100%) and very good at flagging the right number of suspicious ships (ARC score of 90%).
The "Okay" News: It was decent at separating normal from weird ships and checking if the physics made sense, but there was still a little room for improvement in those areas.

The Bottom Line

This paper doesn't just say, "We found some weird ships." It built a ruler to measure how well we found them.

The authors also built a visual map tool (using a library called Folium) that lets humans click on the flagged ships and see why the computer thought they were weird (e.g., "This ship jumped 50 miles in a second"). This turns a confusing list of numbers into a clear picture for maritime authorities.

In short, the paper says: "We built a new, objective way to grade how good our AI is at spotting suspicious ships, and it proved to be a reliable tool for finding the most dangerous anomalies in the ocean."

Technical Summary: A Novel Evaluation Metric for Unsupervised Learning in AIS-Based Maritime Anomaly Detection (MADQI)

Problem Statement
Maritime anomaly detection using Automatic Identification System (AIS) data is critical for safety, security, and environmental protection. However, the field faces a significant challenge: the lack of labelled ground truth data for anomalous vessel behaviours. Consequently, researchers rely on unsupervised learning algorithms, such as Isolation Forest, to detect irregularities. While these methods are effective at identifying outliers, the paper highlights a critical gap in the literature: the absence of objective, quantitative, and domain-specific evaluation metrics. Current assessments often rely on subjective visual inspection or expert judgment, making it difficult to benchmark model performance, compare different algorithms, or validate results systematically.

Methodology
The study proposes a comprehensive framework centered on the Maritime Anomaly Detection Quality Index (MADQI), a composite metric designed to evaluate unsupervised anomaly detection without labelled data.

Data and Feature Engineering:
- The framework utilizes AIS data (specifically US data from March 31, 2022) containing spatio-temporal trajectories.
- Raw data (latitude, longitude, Speed Over Ground [SOG], Course Over Ground [COG], and timestamps) are processed to extract derived features using the Haversine formula for spatial distance.
- Key engineered features include:
  - Spatial: Distance between consecutive points.
  - Temporal: Time gaps between messages.
  - Kinematic: Implied speed (derived from distance/time) and the difference between reported SOG and implied speed.
  - Behavioural: Turn rates and composite indicators.
- Anomalies are categorized into five types: Speed Anomaly, Position Jump Anomaly, Temporal Gap Anomaly, Turn Rate Anomaly, and Composite Behavioural Anomaly.
Anomaly Detection Model:
- The Isolation Forest algorithm is employed as the core unsupervised model. It isolates anomalies by constructing binary trees based on random feature splits; anomalous points typically require fewer splits to be isolated, resulting in shorter path lengths and higher anomaly scores.
The MADQI Framework:
- To evaluate the model's output, the authors introduce MADQI, which aggregates four distinct components:
  - Anomaly Rate Consistency (ARC): Measures the alignment between the observed anomaly ratio and the expected contamination level.
  - Physical Plausibility Score (PPS): Evaluates whether detected anomalies exhibit meaningful physical deviations (in speed, distance, and time) compared to normal behaviour.
  - Score Distribution Separation (SDS): Quantifies the statistical separation between the anomaly scores of normal and anomalous observations.
  - Extreme Case Evidence (ECE): Assesses the presence of extreme behavioural deviations, particularly in spatial displacement.
- Normalization Strategy: To ensure comparability across these heterogeneous metrics, the framework employs a multi-chunk evaluation strategy (partitioning data into non-overlapping subsets) and adaptive scaling. It uses lower-quartile statistics to determine scaling parameters and applies an exponential normalization function ( $1 - e^{-x/\tau}$ ). This approach avoids the "centralization bias" of traditional soft normalization, allowing high-performing components to approach unity naturally.

Key Contributions
The paper outlines four primary contributions:

Unified Unsupervised Approach: A robust framework using Isolation Forest and Haversine-based spatial analysis to detect multi-feature maritime anomalies without labels.
Structured Anomaly Taxonomy: A classification system defining five specific types of maritime anomalies (speed, position jump, time gap, turn rate, and composite) to structure the detection process.
MADQI Metric: A novel, domain-specific, label-free quality index that integrates spatial, temporal, and behavioural dimensions into a single interpretable score.
Operational Tool: A practical implementation that projects detected anomalies onto interactive geospatial maps (using the Folium library), bridging the gap between unsupervised detection and actionable intelligence for maritime authorities.

Experimental Results
The framework was tested on a dataset of over 1 million AIS observations, with a specific focus on a 500,000-observation subset for evaluation.

Detection Performance: The Isolation Forest model identified 498 anomalies (approx. 0.1% of the data) and 10 extreme anomalies based on domain-specific rules.
MADQI Scores: The framework achieved a total MADQI score of 80.37%.
- Extreme Case Evidence (ECE): Scored 1.000 (normalized), indicating excellent capability in detecting extreme spatial deviations.
- Anomaly Rate Consistency (ARC): Scored 0.907, showing strong alignment between observed and expected anomaly rates.
- Physical Plausibility (PPS): Scored 0.672, reflecting reasonable physical validity.
- Score Distribution Separation (SDS): Scored 0.636, indicating clear statistical separation between normal and anomalous cases.
Visualization: The results were visualized on interactive maps, allowing for qualitative validation of the spatial distribution of anomalies.

Significance and Claims
The authors claim that MADQI addresses a fundamental open problem in the field: the lack of a standardized, objective evaluation metric for unsupervised anomaly detection in maritime contexts.

Interpretability: By decomposing performance into ARC, PPS, SDS, and ECE, the metric provides diagnostic insights into why a model performs well or poorly, rather than offering a single opaque score.
Robustness: The multi-chunk evaluation and data-driven adaptive scaling reduce sensitivity to specific dataset fluctuations and eliminate arbitrary parameter tuning.
Scalability and Applicability: The framework is computationally efficient and designed for large-scale AIS datasets, offering a "plug-and-play" solution for real-time maritime surveillance systems.
Model Agnosticism: While demonstrated with Isolation Forest, the authors state the metric is designed to be model-agnostic, facilitating the benchmarking of various unsupervised techniques.

The paper concludes that MADQI provides a reliable, meaningful, and scalable approach to evaluating anomaly detection in maritime AIS data, moving the field away from subjective analysis toward quantitative, domain-aware assessment.

The Problem: The "Black Box" of Judgment

The Solution: MADQI (The "Report Card")

How They Tested It

The Results

The Bottom Line

More like this