3AM: 3egment Anything with Geometric Consistency in Videos

1. Problem Statement

Video Object Segmentation (VOS) aims to identify and track objects across video frames with consistent masks. While recent foundation models like SAM2 excel at 2D segmentation using memory-based architectures, they suffer significantly under large viewpoint changes (wide-baseline motion).

• Limitation of 2D Methods: Purely 2D approaches rely on appearance features. When an object reappears after a significant camera rotation or translation, the visual appearance changes drastically, causing the model to lose track or confuse the object with distractors.
• Limitation of 3D Methods: Traditional 3D instance segmentation methods achieve better geometric consistency but require explicit 3D inputs (camera poses, depth maps, point clouds) and expensive preprocessing (e.g., Structure-from-Motion). They are often offline, computationally heavy, and struggle with dynamic scenes or incomplete reconstructions.

The Core Challenge: How to achieve 3D-aware, viewpoint-robust object tracking that maintains identity across large camera motions, without requiring camera poses, depth maps, or explicit 3D supervision during inference?

2. Methodology: 3AM

The authors propose 3AM, a training-time enhancement that integrates 3D-aware geometric features from the MUSt3R (Multi-view Stereo 3D Reconstruction) model into the SAM2 architecture.

A. Architecture Overview

The pipeline consists of three main components:

1. Dual Feature Extraction:
   • Appearance Stream: The input frame is processed by the SAM2 vision encoder to extract 2D appearance features ($F_{2D}$).
   • Geometry Stream: The same frame is passed through MUSt3R, which utilizes its internal cross-attention mechanism to attend to multi-view memory, producing geometry-aware features ($F_{3D}$) that encode implicit geometric correspondence.
2. Feature Merger:
   • A lightweight Feature Merger fuses multi-level MUSt3R features with SAM2's appearance features.
   • Multi-level Fusion: The authors observe that early MUSt3R layers preserve semantic correspondence, while deeper layers capture 3D structure. They sample features from both shallow and deep layers.
   • Fusion Mechanism: The merger uses hierarchical cross-attention to align information from different depths, followed by convolutional refinement to merge the geometry-aware features with SAM2's high-resolution 2D features. This creates a representation grounded in both spatial position and visual similarity.
3. Memory & Decoding: The merged features ($F_{merged}$) are processed by SAM2's memory attention module and mask decoder. The memory encoder updates the memory bank with these geometry-consistent features for future frames.

B. Training Strategy: Field-of-View (FoV) Aware Sampling

A critical innovation is the FoV-aware sampling strategy used during training to ensure the model learns valid 3D correspondences:

• The Problem: Naïve random sampling might select two frames showing the same object instance but from spatially disjoint regions (e.g., the headboard vs. the footboard of a bed). Treating these as the same object forces the model to match inconsistent geometry, leading to ambiguous learning.
• The Solution: For each training iteration, a reference frame is selected. Candidate frames are filtered based on whether their masked 3D points (back-projected using depth) fall within the reference camera frustum.
• Result: This ensures that the model only learns associations between frames that observe overlapping physical regions of the object, enforcing true geometric consistency while preserving natural occlusion and pose variations.

C. Dynamic Object Fallback

Since 3AM relies on relative geometric consistency, it may struggle with highly dynamic objects that change shape or location unpredictably.

• Mechanism: The system detects substantial motion (using a motion detection module). If significant motion is detected, it reverts to the original SAM2 pipeline for that sequence.
• Implementation: This is a sequence-level switch rather than a mid-sequence fusion to avoid corrupting the temporal memory bank with conflicting mask distributions.

3. Key Contributions

1. Bridging the 2D-3D Gap: The paper identifies that 2D tracking fails under large viewpoint changes while 3D methods are too rigid. 3AM solves this by injecting implicit geometric cues into a 2D foundation model.
2. Geometry-Aware Feature Integration: A novel Feature Merger that fuses multi-level MUSt3R features (encoding implicit 3D correspondence) with SAM2's appearance features, enabling robust tracking using only RGB input at inference.
3. FoV-Aware Sampling: A training strategy that filters frames to ensure spatial overlap, preventing the model from learning spurious associations between distant parts of the same object.
4. Generalized Framework: The method maintains the promptability and efficiency of SAM2, automatically falling back to standard 2D tracking for dynamic scenarios, making it practical for real-world applications.

4. Experimental Results

The method was evaluated on challenging datasets with wide-baseline motion: ScanNet++ and Replica.

• Performance on ScanNet++ (Selected Subset):
  • IoU: 90.6% (vs. SAM2Long's 74.7% and DAM4SAM's 76.5%).
  • Tracking Recall: 71.7% (vs. SAM2Long's 41.3%).
  • Improvement: Outperforms state-of-the-art VOS methods by +15.9 points in IoU and +30.4 points in Tracking Recall.
• Performance on Replica:
  • Achieved the best performance across all metrics (IoU: 0.8119, Tracking Recall: 0.6381), significantly surpassing SAM2 and its extensions.
• 3D Instance Segmentation:
  • When used for class-agnostic 3D instance segmentation on ScanNet200, 3AM achieved an AP of 47.3, outperforming other online methods and even some offline methods, demonstrating that robust 2D tracking can yield reliable 3D instances without explicit 3D merging.
• Ablation Studies:
  • Confirmed that both MUSt3R (geometry) and SAM2 (appearance) features are necessary.
  • Showed that FoV-aware sampling significantly outperforms random or continuous sampling.
  • Demonstrated that MUSt3R provides superior object alignment compared to other 3D backbones like CUT3R.

5. Significance

3AM represents a paradigm shift in video object segmentation by proving that geometric consistency can be learned and integrated into 2D foundation models without the heavy computational cost of explicit 3D reconstruction pipelines.

• Practicality: It requires no camera poses or depth maps at inference, making it deployable on standard video feeds and casual multi-view photo collections.
• Robustness: It solves the "reappearance" problem where objects disappear and reappear from drastically different angles, a common failure point for current SOTA methods.
• Efficiency: By leveraging the pre-trained SAM2 backbone and a lightweight merger, it maintains streaming capabilities suitable for real-time applications like robotics and AR/VR.

In summary, 3AM successfully combines the best of 2D promptable segmentation and 3D geometric reasoning, offering a robust solution for object tracking in dynamic, unconstrained 3D environments.