Improved Vision-to-Chart Buoy Association with Learned World-to-Image Projection

Technical Summary: Improved Vision-to-Chart Buoy Association with Learned World-to-Image Projection

Problem Definition
The paper addresses the "Vision-to-Chart" data association challenge (MaCVi 2026), which involves detecting buoys in maritime camera images and matching each detection to its corresponding entry in a nautical chart. The core difficulty lies in the geometric projection: the challenge baseline receives per-buoy queries encoding only world-space distance and bearing. Consequently, the transformer decoder must implicitly learn the complex mapping from these world coordinates to image pixels, a process heavily dependent on the vessel's instantaneous pitch, roll, and heading (captured via IMU). The baseline approach forces the model to discover this geometric relationship from scratch, relying solely on appearance-based refinement.

Methodology
The proposed solution introduces a lightweight modification to the DETR-based fusion transformer baseline by explicitly providing a spatial prior to the decoder. The methodology consists of two main stages:

1. QueryMLP (Learned World-to-Image Projection):
   A dedicated Multi-Layer Perceptron (MLP) is trained independently to predict the buoy's "waterline contact point" in the image. This point is chosen as the prediction target because it corresponds directly to the object's position on the water plane and is approximated by the bottom-center edge of the bounding box ($[c_x, c_y + h/2]$).

   • Inputs: The MLP takes six features derived from chart data and IMU orientation: normalized distance, inverse distance (clipped to handle near-field nonlinearity), normalized bearing, pitch, roll, and heading.
   • Architecture: A four-layer network ($6 \to 128 \to 128 \to 128 \to 2$) with BatchNorm, ReLU, Dropout, and a Sigmoid output layer.
   • Training: It is trained on paired (query, IMU, label) examples using AdamW and SmoothL1 loss, converging at epoch 585 with a median validation error of 18.4 pixels.

2. Integration into the DETR Pipeline:
   Once trained, the QueryMLP is frozen. During DETR training and inference, the dataset loader runs the MLP on every chart query to generate predicted pixel coordinates. These coordinates are concatenated with the original normalized distance and bearing to form an augmented 4-dimensional query vector ($q = [dist_{norm}, bearing_{norm}, c_x, c_y + h/2]$).

   • This vector widens the input dimension of the baseline decoder's embedding MLP from 2 to 4.
   • The transformer decoder then cross-attends these enriched embeddings against ResNet-50 image features.
   • All other architectural components (ResNet-50 backbone, transformer layers, loss functions) remain identical to the baseline.

Key Contributions

• Explicit Spatial Priors: The work replaces the implicit learning of geometric projection with an explicit, learned mapping that provides the decoder with an approximate location for each buoy before it processes image features.
• Waterline Contact Point Target: Inspired by prior maritime georeferencing work, the method specifically targets the waterline contact point as the most geometrically meaningful image-space location for buoy association.
• Lightweight Modification: The approach requires no changes to the heavy transformer backbone or the main training loop of the DETR model, only an additional pre-trained MLP and a modification to the query construction pipeline.

Results
The method was evaluated on the ATON dataset (5,189 samples) and the challenge's private held-out test set.

• Performance: The proposed approach achieved an Overall score of 0.7386 on the test set (calculated as the mean of F1-score and mIoU), placing second among all submissions.
  • Test Set Metrics: F1 = 0.8055, mIoU = 0.6718.
  • Validation Set Metrics: Overall = 0.7462 (F1 = 0.8171, mIoU = 0.6753).
• Comparison: When re-trained under identical conditions, the baseline (using only distance and bearing) achieved an Overall score of 0.7193 on the validation set. The proposed method demonstrated consistent generalization between validation and test sets.
• Qualitative Analysis: Visual comparisons indicate that the proposed method suppresses false positive detections on non-buoy objects and successfully detects buoys missed by the baseline, attributed to the improved precision from the spatial prior.

Significance and Claims
The paper claims that the primary significance of this work is the reduction of the "geometric reasoning burden" on the transformer decoder. By explicitly predicting where a buoy should appear based on world coordinates and vessel orientation, the model can focus its capacity on appearance-based refinement rather than learning the projection geometry from scratch.

The authors modestly note limitations and future directions:

• Frozen Design: The current implementation keeps QueryMLP frozen during DETR training, preventing the decoder from providing feedback to refine pixel predictions. End-to-end joint training is suggested as a future improvement.
• Environmental Robustness: While the waterline contact point is geometrically stable in calm conditions, the authors acknowledge it may be less reliable in rough seas where wave action or spray obscures the buoy's base—precisely the conditions where robust association is most critical.

The code for the implementation is publicly available.