Learning Cross-View Object Correspondence via Cycle-Consistent Mask Prediction

1. Problem Definition

The paper addresses the challenge of cross-view object correspondence, specifically establishing object-level visual alignment between egocentric (first-person, wearable camera) and exocentric (third-person, external camera) viewpoints in videos.

• Core Difficulty: The task involves mapping an object mask from a source view (e.g., a shaky, cluttered, first-person view) to a target view (e.g., a stable but potentially distant third-person view) and vice versa.
• Challenges:
  • Appearance Variation: Drastic changes in lighting, resolution, and occlusion between views.
  • Spatial Disparity: The spatial layout and background context differ significantly, making background cues unreliable.
  • Temporal Dynamics: Objects may deform or move differently across camera angles.
  • Data Scarcity: Traditional correspondence models often rely on co-visible or static scenes, failing in these extreme viewpoint shifts. Furthermore, obtaining ground-truth annotations for the target view is difficult, limiting supervised learning.

2. Methodology

The authors propose a simple yet effective framework based on Conditional Binary Segmentation enhanced by a Cycle-Consistency training objective and Test-Time Training (TTT).

A. Architecture

The model leverages the DINOv3 vision foundation model as a backbone and consists of three main components:

1. Source Feature Extractor:
   • Takes a source image ($I_s$) and its object mask ($M_s$).
   • Extracts feature maps and computes a compact, object-specific feature representation ($z_s$) by performing a weighted average of the feature map using the normalized mask as weights.
   • This representation is projected into a Condition Token (CDT).
2. Transformer Encoder:
   • The target image ($I_t$) is patchified into visual tokens.
   • The input sequence includes a Class Token ($\mathit{CLS}$), the Condition Token (CDT), and the visual tokens.
   • The CDT conditions the transformer via cross-token attention, guiding the encoder to focus on the corresponding object in the target view.
3. Multi-task Decoder:
   • Mask Head: Generates a binary segmentation mask ($\hat{M}_t$) for the target view using the visual tokens.
   • CLS Head: Predicts object visibility (binary classification) to determine if the object exists in the target view.

B. Training Objectives

The model is trained using a composite loss function:
$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{mask}} + \lambda_{\text{aux}} \mathcal{L}_{\text{aux}} + \lambda_{\text{cycle}} \mathcal{L}_{\text{cycle}}$$

1. Mask Loss ($\mathcal{L}_{\text{mask}}$): A combination of Binary Cross-Entropy (BCE) and Dice Loss to supervise the predicted target mask against ground truth.
2. Auxiliary Loss ($\mathcal{L}_{\text{aux}}$): Deep supervision applied to intermediate transformer layers to improve gradient flow.
3. Cycle-Consistency Loss ($\mathcal{L}_{\text{cycle}}$):
   • Mechanism: The predicted target mask ($\hat{M}_t$) is fed back into the system as a source mask to reconstruct the original query mask ($\hat{M}_s$).
   • Constraint: The reconstructed mask $\hat{M}_s$ is compared to the original $M_s$ using BCE loss.
   • Significance: This enforces view-invariant representations and provides a self-supervised signal that does not require ground-truth annotations for the target view.

C. Test-Time Training (TTT)

A key innovation is the application of TTT during inference:

• Since $\mathcal{L}_{\text{cycle}}$ does not require ground-truth target masks, it can be computed on unlabeled test pairs.
• For each test pair, the model performs a few gradient update steps (fine-tuning only the last $K$ transformer layers) to minimize the cycle-consistency loss.
• This allows the model to adapt dynamically to specific domain shifts or distribution variations in the test data, significantly boosting performance.

3. Key Contributions

1. Novel Framework: A modular, end-to-end framework for cross-view correspondence that requires minimal architectural changes to pretrained vision foundation models (DINOv3).
2. Cycle-Consistency Objective: A self-supervised training strategy that enforces semantic alignment between source and target views, enabling robust learning without target-view annotations.
3. Test-Time Training Strategy: The first successful application of TTT to this specific cross-view correspondence task, demonstrating that inference-time adaptation significantly improves robustness.
4. State-of-the-Art Performance: Achieved top results on challenging benchmarks, proving the efficacy of the proposed optimization and TTT strategy.

4. Experimental Results

The method was evaluated on two major benchmarks: Ego-Exo4D and HANDAL-X.

• Ego-Exo4D Benchmark:

  • Exo Query (Third-person to First-person): Achieved an IoU of 47.18%, outperforming the previous SOTA (O-MaMa) by 7.0% relative improvement.
  • Ego Query (First-person to Third-person): Achieved an IoU of 41.95%, closely approaching the SOTA (42.57%) while surpassing all other baselines.
  • Overall mIoU: 44.57%, a +2.9% improvement over the previous best.
  • Visibility Accuracy: Achieved 98.92% (Ego) and 99.86% (Exo), indicating excellent capability in detecting object presence.

• HANDAL-X Benchmark (Zero-Shot & Fine-tuning):

  • Zero-Shot: Without any training on HANDAL-X, the method achieved 78.8% IoU, surpassing the previous best (ObjectRelator) by 36.0% relative improvement.
  • Fine-tuned: After fine-tuning, it reached 85.0% IoU, outperforming all competitors.

• Ablation Studies:

  • Removing the cycle-consistency loss caused a significant drop in mIoU, confirming its role in providing the self-supervisory signal necessary for TTT.
  • TTT consistently improved performance across all settings.
  • The framework's success was attributed more to the architectural design and loss functions than just the choice of backbone (outperforming baselines even when swapping backbones).

5. Significance

This work is significant for several reasons:

• Robustness to Extreme Viewpoints: It effectively solves the "ego-exo" gap, a critical requirement for embodied AI, human-robot interaction, and autonomous navigation where agents must understand instructions from one perspective and act from another.
• Self-Supervised Learning: By utilizing cycle-consistency, the method reduces reliance on expensive, pixel-level annotations for the target view, making it scalable to diverse datasets.
• Inference-Time Adaptation: It demonstrates that models can be dynamically refined during inference to handle specific domain shifts, a paradigm shift from static model deployment.
• Simplicity and Efficiency: The approach achieves SOTA results with a relatively simple architecture and minimal architectural modifications to foundation models, suggesting a highly efficient path forward for cross-view tasks.