OAMVOS:2nd Report for 5th PVUW MOSE Track

Technical Summary: OAMVOS (Occlusion- and Reappearance-Aware Extension)

1. Problem Statement

The paper addresses the limitations of SAM-based (Segment Anything Model) dense trackers in Video Object Segmentation (VOS). While these models excel at short-term mask propagation, they are highly susceptible to failure in complex scenarios involving:

• Long-term occlusion and reappearance: When an object disappears and returns, trackers often lose the target.
• Fast motion and viewpoint changes: Rapid changes can lead to "compounding drift," where one incorrect prediction contaminates the model's memory.
• Small objects: Small targets are particularly vulnerable because minor localization errors or noisy object pointers can effectively "erase" the target from the model's memory.
• Distractors: In cluttered scenes, the tracker may mistakenly update its memory with a similar-looking background object.

The authors identify that the core issue is not the segmentation backbone itself, but rather inefficient memory governance—specifically, the "single-path" nature of existing pipelines where every prediction is immediately committed to memory, regardless of its quality.

2. Methodology

The proposed method is an extension of the DAM4SAM tracker. Instead of changing the backbone, the authors introduce a control layer that manages how information is propagated and stored. The system operates in three distinct modes: stable, ambiguous, and recovery.

Key Technical Components:

• Reliability Estimation (The Decision Engine): To switch between modes, the model calculates four scores for every frame:
  • Appearance Score: Cosine similarity against an anchor bank of high-confidence frames.
  • Motion Score: Penalizes implausible jumps using constant-velocity estimation.
  • Geometry Score: Monitors stability in mask area and aspect ratio.
  • Candidate Margin: Measures the confidence gap between the top two predicted masks.
• Branch-Based Recovery: When reliability drops (entering ambiguous or recovery modes), the tracker stops following a single path. Instead, it maintains a branch pool of multiple hypotheses (e.g., the primary mask, alternative masks, or an "absent object" hypothesis). Each branch evolves independently. A branch is only "reconfirmed" and returned to the main path after it demonstrates consistent wins and high appearance agreement.
• Memory Governance (Delayed DRM Promotion): To prevent "memory contamination," the authors implement a delayed promotion policy for the Dynamic Anchor Memory (DRM). A frame is only promoted to the high-level DRM if it meets strict criteria (high IoU, sufficient time gap from the last update, and stable size) and remains stable for a set number of consecutive frames ($N_{drm}$).
• Conditional Memory Selection Bypass: Native SAM3 memory selection tends to favor recent frames. During small-object disappearance, recent frames are often empty or noisy. The authors implement a bypass mechanism ($\gamma_t$) that disables native selection for small objects during uncertainty, allowing the model to access older, more reliable anchors.
• Conditioning-Memory Attention: To aid long-gap reappearance, the model explicitly preserves the first conditioning frame (the initial target definition) in the attention mechanism, ensuring the model never "forgets" what the original target looked like, even if recent DRM entries are poor.

3. Key Contributions

1. A Reliability-Aware State Machine: Transitions the tracker from a rigid single-path system to a flexible multi-hypothesis system based on real-time confidence.
2. Decoupling Prediction from Commitment: By using a branch pool and delayed DRM promotion, the method ensures that only high-quality, verified information enters the long-term memory.
3. Small-Object Robustness: Specifically addresses the "memory erasure" problem of small objects through selective memory bypass and the preservation of the initial frame.
4. Efficiency-Preserving Design: The method remains computationally efficient by only activating complex branching and multiple hypotheses when the tracking state becomes uncertain.

4. Results

The method was evaluated in the 5th PVUW MOSE Track (CVPR 2026 challenge), which focuses on complex, cluttered scenes with small, occluded, and reappearing objects.

• Performance: The proposed method achieved 2nd place on the leaderboard.
• Metrics: It demonstrated high robustness in $J\&F$ (Jaccard and F-measure) and temporal accuracy, proving its effectiveness in handling the specific failure modes (occlusion and reappearance) targeted by the challenge.

5. Significance

This work shifts the focus of robust video object segmentation from segmentation quality (the "what") to memory management (the "when" and "how"). It demonstrates that for long-term tracking, the ability to "doubt" a prediction and maintain alternative hypotheses is more critical than the raw power of the segmentation backbone. This approach provides a blueprint for making foundation-model-based trackers more reliable in real-world, unpredictable video environments.