💻 computer science

From Open Waters to Enclosed Cabins: ProteusVPR for Cross-Scene Visual Place Recognition in Maritime Perception and Cabin Inspection

This paper introduces ProteusVPR, a two-stage retrieval-refinement framework designed to overcome cross-scene perceptual challenges in maritime environments by significantly improving visual place recognition accuracy for autonomous robots transitioning between open decks and enclosed cabins, supported by the newly proposed XHZ dataset.

Original authors: Zexi Chena, Zitai Huang, Qiwen Gu, Zhiqi Li, Shengli Dong, Chenlei Wang, Junqiao Zhao, Hongdong Wang, Bing Han

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zexi Chena, Zitai Huang, Qiwen Gu, Zhiqi Li, Shengli Dong, Chenlei Wang, Junqiao Zhao, Hongdong Wang, Bing Han

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Getting Lost on a Ship

Imagine you are a robot tasked with inspecting a massive ship. Your job is to know exactly where you are at all times.

The ship is a tricky place to navigate because it has two very different "worlds":

The Open Deck: This is like standing on a beach. It's wide open, the sky and water look the same everywhere, and the sun might be blinding you one minute and gone the next. It's hard to find landmarks here.
The Inside Cabins: This is like walking through a hotel with hundreds of identical hallways. Every door looks the same, every wall is painted the same color, and the corridors are narrow. It's easy to get confused because every room looks exactly like the one before it.

The Challenge: Current robot navigation systems are great at finding their way in cities (where buildings look different) or inside houses (where rooms are unique). But they get completely lost when a robot tries to jump from the open deck into these repetitive cabins. They can't tell if they are in the "kitchen" or the "bathroom" because the pictures look too similar.

The Solution: ProteusVPR

The researchers created a new system called ProteusVPR. They named it after the Greek sea god Proteus, who could change his shape. Just like the god, this system is designed to adapt to the ship's changing environments.

The system works in two steps, like a detective solving a mystery:

Step 1: The "Gross" Guess (Retrieval)

First, the robot takes a picture and asks a standard database: "Which picture in my memory bank looks most like this one?"

The Analogy: Imagine you are looking for a specific house in a city. You look at a photo of a red door and ask a friend, "Which house has a red door?" The friend points to a house that looks similar.
The Problem: In the ship's cabins, the friend might point to the wrong house because every house has a red door. This is called "retrieval ambiguity." The robot gets a "good guess," but it might still be wrong.

Step 2: The "Fine-Tune" Correction (Refinement)

This is where ProteusVPR shines. Instead of just accepting the friend's guess, the robot looks at the two pictures it took just before the current one.

The Analogy: Imagine you are walking down a hallway. Even if you can't tell which specific room you are in just by looking at the door, you know you just turned left, then walked forward three steps.
How it works: The system combines the "best guess" picture with the two previous pictures. It uses geometry (math about shapes and distances) and the direction the camera is facing to calculate: "If I was at the 'guess' location, and I moved forward like I did in the last two photos, where should I actually be?"
The Result: It corrects the mistake. If the robot thought it was in the "Kitchen" but the math says it's actually in the "Bathroom" based on its movement, it fixes the location instantly.

The New Map: The XHZ Dataset

To prove this works, the researchers couldn't just use old maps. They had to build a new one.

They went onto a real, working ship (the Xinhongzhuan).
They used a special 360-degree camera to take thousands of photos of the decks and the inside cabins.
They carefully labeled every single photo with its exact GPS-like coordinates.
Why this matters: This dataset is like a "final exam" for robots. It forces them to deal with the confusing, repetitive hallways and the blinding open decks, proving that ProteusVPR can handle the real deal.

What Did They Find?

The researchers tested their system against many other popular robot navigation tools.

The Result: ProteusVPR was a clear winner. It reduced the average error in knowing where the robot was by over 60%.
The "Magic" Ingredient: The most important part of their system was the geometric math (Step 2). When they removed the math that calculates the relationship between the three photos, the system got lost again. When they kept it, the system became incredibly precise, even in the most confusing, repetitive hallways.

Summary

Think of ProteusVPR as a robot that doesn't just rely on recognizing a landmark (which can be confusing on a ship). Instead, it remembers how it got there. By combining a "best guess" with the memory of its recent steps and the direction it's facing, it can pinpoint its location with high accuracy, whether it's standing in the blinding sun on the deck or lost in a maze of identical cabins.

Technical Summary: ProteusVPR for Cross-Scene Visual Place Recognition in Maritime Perception

1. Problem Statement

Autonomous robotic inspection in maritime environments faces unique challenges for Visual Place Recognition (VPR) due to extreme cross-scene perceptual shifts. Robots navigating ship-borne environments must transition between two visually distinct domains:

Open Decks: Characterized by sparse textures, repeated sky-water backgrounds, and severe illumination changes caused by weather and reflections.
Enclosed Cabins: Defined by highly repetitive structures, narrow corridors, and multi-floor layouts with strong appearance similarity.

Existing VPR methods, primarily designed for urban or indoor scenes (e.g., NetVLAD, CosPlace, MixVPR), fail to generalize reliably across these starkly different scenarios. They often optimize for image-level retrieval rather than precise localization, struggling with the ambiguity of repetitive indoor structures and the lack of texture in open decks. Furthermore, simply fine-tuning existing models on maritime datasets is often insufficient because their underlying feature representations are designed for generic large-scale retrieval rather than repetitive indoor localization.

2. Methodology: ProteusVPR

The authors propose ProteusVPR, a two-stage retrieval-refinement framework designed to adapt existing VPR systems to maritime cabin environments without redesigning the retrieval backbone.

Stage 1: Initial Image Retrieval

The first stage employs a standard, pre-trained VPR model (e.g., MutualVPR, CosPlace, SALAD) to retrieve the most visually similar reference image ( $I_{r1}$ ) from a geo-tagged database. This stage outputs the geographic location ( $\mathbf{p}_{r1}$ ) and the field-of-view center azimuth ( $y_{r1}$ ) of the retrieved image.

Stage 2: Geometric-Visual Estimation Network

The second stage refines the initial retrieval using a relative displacement estimation network. It takes the retrieved image ( $I_{r1}$ ) and the two temporally preceding frames of the query video sequence ( $I_{r2}, I_{r3}$ ) as input. The network fuses multiple modalities to estimate the precise location:

Visual Feature Extraction: A frozen DINOv2 ViT-B/14 encoder processes the query and three reference images. Features are enhanced via concatenation of CLS tokens and average patch tokens, followed by inter-reference and query-reference attention mechanisms.
Geometric Descriptor Encoding: The network computes geometric relationships (displacements, distances, and angles) between the three reference points. These are encoded into a global geometric descriptor ( $\mathbf{f}_{geo}$ ).
Local Affine Coordinate System: To achieve rotation and translation invariance, a local 2D affine coordinate system is constructed using the three reference points. The query's location is estimated as a displacement ( $\boldsymbol{\delta}_{local}$ ) relative to the centroid of this local system, rather than in global coordinates.
Azimuth Encoding: The field-of-view center azimuths (yaw angles) of the query and reference images are encoded and fused to provide directional context.
Regression and Transformation: An MLP regresses the local displacement vector. This is transformed back to global coordinates using the constructed affine basis.

Loss Function: The network is trained end-to-end using a combination of:

Absolute Position Loss ( $\mathcal{L}_{abs}$ ): Euclidean distance between predicted and ground-truth positions.
Geometric Consistency Loss ( $\mathcal{L}_{geo\_cons}$ ): Ensures the predicted position remains geometrically plausible relative to the reference points (barycentric constraints), penalizing predictions that deviate significantly from the reference cluster.

3. The XHZ Dataset

To support rigorous evaluation, the authors introduce the XHZ (Xinhongzhuan Visual) Dataset, collected from an operational vessel.

Scope: Covers interior cabins (multi-floor structures), deck transition zones, and exterior deck areas.
Characteristics: Features highly repetitive spatial layouts, strong visual similarity, and constrained motion trajectories.
Annotation: Contains 2,996 panoramic images (Version 1) with strict query-database separation. The dataset employs a specific coordinate offset strategy to distinguish different decks (L0–L3) in 2D projections, preventing false associations between floors.
Evaluation Protocol: Queries are selected from specific yaw angles (0°), while the database includes other angles and temporal frames to simulate discrete observation and dense mapping scenarios.

4. Experimental Results

Extensive experiments were conducted on the XHZ dataset using multiple VPR backbones (BoQ, CosPlace, EigenPlaces, MixVPR, SALAD, MutualVPR) with both pre-trained and fine-tuned weights.

Retrieval Performance: Fine-tuning significantly improved baseline performance. MutualVPR (fine-tuned) achieved the highest recall (R@1 up to 97.3% on Layer 3) and the lowest 90% quantile localization errors across all layers.
Refinement Performance: The ProteusVPR refinement stage consistently improved localization accuracy.
- On average, the framework reduced the mean localization error by over 60% across multiple backbones.
- For MutualVPR (fine-tuned) on Layer 3, the mean error dropped from 1.519 m (Stage 1) to 0.242 m (Stage 2), an 84.1% reduction.
- For CosPlace (fine-tuned) on Layer 3, the mean error dropped from 4.416 m to 0.275 m (93.8% reduction).
Robustness: The refinement stage was particularly effective in correcting "long-tail" errors (catastrophic failures) where the initial retrieval was inaccurate, often correcting cross-floor retrieval errors by leveraging the temporal continuity of the preceding frames.

Ablation Study

Geometric Descriptors: Removing geometric descriptor encoding caused a massive performance drop (e.g., Layer 3 90% quantile error increased from 0.432 m to 2.974 m), confirming its critical role.
Local Affine Coordinate System: Removing this system (performing regression in global coordinates) significantly degraded performance, validating the need for invariance.
Azimuth Encoding: Removing explicit azimuth encoding resulted in only marginal performance loss, suggesting that visual features in ship interiors implicitly encode directional information.

5. Significance and Claims

The paper claims that ProteusVPR offers an effective and robust solution for precise visual localization in challenging, cross-scene maritime environments. Its primary significance lies in:

General Adaptability: It serves as a general second-stage refinement framework that can be applied to existing VPR systems, enhancing their performance in maritime settings without requiring a complete architectural overhaul.
Bridging the Gap: It successfully bridges the gap between open-water scenarios (e.g., autonomous berthing) and enclosed environments (e.g., AR-based cabin inspection), addressing the specific perceptual aliasing found in repetitive ship interiors.
Benchmarking: The introduction of the XHZ dataset provides a rigorous, fine-grained benchmark for maritime VPR, addressing the lack of suitable datasets for evaluating cross-scene navigation on vessels.

The authors acknowledge limitations, noting that the framework depends on the quality of the initial retrieval (severe retrieval failures cannot be fully corrected) and that the dataset is currently limited to a single vessel. Future work is suggested to explore uncertainty-aware retrieval and larger, more diverse maritime datasets.