Technical Summary: GMOS – Grounding Moving Object Segmentation in 3D Space and Time

1. Problem Definition and Limitations of Current Approaches

Moving Object Segmentation (MOS) aims to discover, segment, and track objects that move independently of the camera. This capability is foundational for applications such as autonomous driving, camouflaged object detection, and video surveillance, as well as for 3D/4D scene reconstruction methods that require separating dynamic objects from static backgrounds.

The authors identify two fundamental limitations in existing MOS methods:

1. Lack of 3D Awareness: Current methods rely heavily on pre-computed 2D auxiliary modalities (e.g., optical flow, point trajectories, or depth maps). These inputs lack 3D geometric information, making methods fragile under large camera motions or heavy depth parallax. Furthermore, generating these auxiliary inputs incurs substantial pre-processing overhead.
2. Coarse Temporal Granularity: Existing benchmarks and methods typically treat motion as a sequence-level attribute. They mask every object that moves at any point in the video sequence, regardless of whether the object is currently in motion. This overlooks the instantaneous motion state of objects, precluding online, real-time deployment, and failing to distinguish between objects that are temporarily static and those that are truly stationary.

2. Methodology: The GMOS Framework

The authors propose GMOS (Grounding Moving Object Segmentation), a feed-forward framework that operates directly on RGB video to produce 3D-aware, temporally fine-grained segmentation without auxiliary inputs. The framework follows a proposer–propagator design.

2.1 The Proposer (Per-Frame Prediction)

The proposer operates on a short temporal window (approximately 0.5 seconds) centered on a query frame. It fuses geometric and segmentation features to generate object proposals and motion states for the middle frame.

• Geometric Encoder: A frozen $\pi^3$ encoder (a feed-forward 3D reconstruction model) processes the short temporal window to extract geometric features. This allows the model to disentangle camera-induced parallax from independent object motion. Only the feature of the middle frame is retained, as inter-frame correspondence is encoded via the model's cross-frame attention.
• Segmentation Encoder: A frozen SAM2 encoder processes the query frame to provide strong objectness priors and high-resolution segmentation features.
• Feature Fusion & Decoding: Geometric and segmentation features are fused and passed through a transformer decoder with $N$ learnable object queries. The decoder outputs:
  • Segmentation masks for moving objects.
  • A binary motion state ($m=1$ if moving, $0$ if static).
  • Estimated IoU and confidence scores.
• Training Objective: The authors introduce a confidence-based loss that automatically down-weights noisy frame-level motion annotations near static–moving transitions. This mitigates label ambiguity where an object's state switches abruptly.

2.2 The Propagator (Temporal Consistency)

To link per-frame proposals into coherent object tracks across the full video, GMOS employs a propagator based on a SAM2 video predictor.

• Mechanism: The propagator takes the high-quality moving-object proposals from the proposer (filtered by motion probability and estimated IoU) and uses them as prompts to update or initialize tracks in the SAM2 state.
• Modes:
  • Online Procedure: A single causal forward pass starting from the first frame. It supports MOS-I (Instantaneous) by streaming masks and motion states as the video is processed. It cannot recover masks for objects that are initially static and only move later.
  • Offline Procedure: A two-stage process. It first runs the online procedure, selects an anchor frame with the highest aggregate predicted IoU, and then performs bi-directional propagation (forward and backward) to recover dense masks for all frames, supporting both conventional MOS and MOS-I.

2.3 GMOS-S (Foreground-Background Variant)

For efficiency-critical deployment, the authors introduce GMOS-S, a variant that predicts all moving objects as a single binary foreground mask. It removes the query-based transformer decoder and the propagator, resulting in an end-to-end model with significantly lower inference latency.

3. Dataset and Evaluation Protocol

3.1 GMOS-2K Dataset

To support training and evaluation in this regime, the authors curated GMOS-2K, a dataset of 2,210 real-world videos drawn from five established Video Object Segmentation (VOS) benchmarks (DAVIS17, YTVOS19, OVIS, MoCA-Mask, HOI4D).

• Key Feature: Unlike standard VOS datasets, GMOS-2K includes per-object temporal motion annotations. It specifies the exact time intervals during which each object is actively moving, distinguishing them from static objects or objects that are temporarily stationary.
• Curation: The dataset was filtered to ensure all moving objects have ground-truth masks. Videos where objects are not moving throughout the entire sequence underwent Temporally Fine-grained Annotation (TFA) to label motion intervals.

3.2 MOS-I Evaluation Protocol

The authors formalize MOS-I ("I" for Instantaneous), a temporally fine-grained evaluation protocol.

• Definition: Unlike conventional MOS, which scores masks for all frames if an object moves anywhere in the sequence, MOS-I scores segmentation masks only on frames where the object is actively moving.
• Metrics:
  1. Moving-object Jaccard ($J_{mov}$): Segmentation accuracy on actively moving frames.
  2. False-positive count (FP count): Penalizes predicting motion for static or background objects.
  3. Mean temporal IoU (mtIoU): A joint measure of detection accuracy and false-positive suppression.

4. Experimental Results

GMOS was evaluated on MOS, MOS-I, and Unsupervised Video Object Segmentation (UVOS) benchmarks.

• MOS Benchmarks: GMOS achieves state-of-the-art (SOTA) results across all conventional MOS benchmarks (DAVIS16-M, DAVIS17-M, SegTrackv2, FBMS-59, MoCA, YTVOS19-M). It significantly outperforms prior methods, particularly on datasets with dense object interactions (YTVOS19-M) and camouflaged objects (MoCA).
• MOS-I Benchmarks: On the new MOS-I benchmarks (DAVIS17-IM, YTVOS19-IM), GMOS surpasses previous SOTA by substantial margins (e.g., +13.4 in $J_{mov}$ on YTVOS19-IM). The results demonstrate that treating motion as a sequence-level attribute is a significant bottleneck for instantaneous segmentation.
• UVOS Benchmarks: GMOS also generalizes well to unsupervised VOS (DAVIS16/17), achieving SOTA on DAVIS17 and competitive results on DAVIS16, despite being trained primarily for moving object segmentation.
• Efficiency: GMOS runs approximately three times faster than prior multi-object MOS methods (e.g., 0.38s/frame vs. 1.79s/frame for FlowSAM). GMOS-S is even faster (0.26s/frame).

5. Significance and Claims

The paper claims that GMOS represents a shift in how moving object segmentation is approached by grounding the task in 3D space and time:

1. 3D Grounding: By leveraging feed-forward 3D reconstruction models ($\pi^3$) directly on RGB video, GMOS achieves robustness to camera motion and depth parallax without relying on error-prone 2D auxiliary inputs like optical flow.
2. Temporal Grounding: By introducing the MOS-I protocol and the instantaneous motion state prediction, GMOS enables fine-grained, online segmentation that accurately reflects the dynamic state of objects frame-by-frame.
3. Efficiency: The framework supports online inference and streaming deployment, making it suitable for real-time applications.
4. Resource Contribution: The release of GMOS-2K and the MOS-I protocol provides a unified resource and evaluation standard for developing temporally fine-grained MOS methods, addressing the lack of datasets that distinguish between moving and temporarily static objects.

The authors conclude that GMOS sets a new state-of-the-art while running significantly faster than prior art, effectively bridging the gap between high-accuracy segmentation and real-time deployment requirements.