Technical Summary: FeVOS - Foresight Expression Video Object Segmentation

Problem Definition

Existing Referring Video Object Segmentation (RVOS) tasks are limited to grounding language expressions within observed frames, focusing on static attributes, appearances, or events that have already occurred. This paradigm lacks the capability for pre-decisive spatio-temporal reasoning, which is critical for proactive decision-making in real-world applications like autonomous driving and robotic planning.

The authors propose a novel task: Foresight Expression Video Object Segmentation (FeVOS). In this task, a model is given a video clip and a "foresight expression" (a query about a future event, e.g., "What tool will be used?"). The model must reason over the observed frames (prior to the event) to predict which object(s) will be involved in the upcoming action and output pixel-level segmentation masks for those objects in the observed frames. This requires bridging the gap between current visual cues and future outcomes using world knowledge and causal reasoning.

Methodology

1. The FeVOS Dataset

To support this task, the authors constructed the FeVOS dataset through a rigorous five-stage pipeline:

• Video Collection: Sourced from diverse benchmarks (COIN, STAR, CLEVR, OOPS, EPIC-KITCHENS-VISOR) covering egocentric activities and instructional videos.
• Automatic Filtration: Used Qwen2.5-VL to filter videos based on clear cause-and-effect narratives, multiple interacting objects, and observable actionable clues.
• Manual Splitting: Expert annotators split videos into "observed" and "future" segments, ensuring the observed part contains implicit cues for the future event.
• Expression Annotation: A two-stage process where annotators designed foresight expressions (e.g., "the object that will be picked") and validators confirmed they could identify the target via predictive reasoning without seeing the future frames.
• Mask & CoT Annotation: Generated pixel-level masks using SAM2 and created 2,904 synthetic Chain-of-Thought (CoT) annotations. These CoT annotations explicitly detail the reasoning steps (analyzing temporal context, spatial cues, and causal relationships) required to reach the prediction.

The final dataset comprises 968 video clips, 14,525 foresight expressions, and 2,904 CoT annotations.

2. FeVOS-R1 Model Architecture

The authors developed FeVOS-R1, a Multimodal Large Language Model (MLLM) based on the Sa2VA framework (integrating InternVL2.5 and SAM2). The model is trained via a two-stage pipeline:

• Stage 1: Supervised Fine-Tuning (SFT) with CoT

  • The model is fine-tuned on the synthetic CoT dataset.
  • It learns to generate step-by-step reasoning processes (analyzing visual cues and causal relationships) before outputting the special [SEG] token.
  • Loss functions include pixel-wise cross-entropy, Dice loss, and text generation loss.
  • This stage establishes the reasoning patterns and output formatting.

• Stage 2: Reinforcement Learning (RL) with GRPO

  • The model undergoes Group Relative Policy Optimization (GRPO).
  • Reward Design: Unlike prior works that rely on complex format rewards (e.g., JSON structure), FeVOS-R1 uses a single IoU-based accuracy reward ($R_{IoU}$) derived from the intersection-over-union between predicted and ground-truth masks.
  • The authors found that format constraints were redundant after SFT and that focusing solely on segmentation accuracy via IoU yielded better performance.
  • This stage optimizes the reasoning chain to directly maximize segmentation quality.

Key Contributions

1. Task Formulation: Introduction of Foresight Expression Video Object Segmentation, shifting RVOS from grounding observed events to predicting future events based on implicit visual cues.
2. Dataset (FeVOS): A curated dataset with 968 videos, 14,525 foresight expressions, and 2,904 CoT annotations, providing the first benchmark for predictive reasoning in video object segmentation.
3. Model (FeVOS-R1): A two-stage training framework combining CoT-guided SFT and RL with end-to-end rewards, achieving state-of-the-art performance on the new task.

Experimental Results

Performance on FeVOS

• Zero-Shot Baselines: Existing RVOS models (e.g., ReferFormer, LMPM) and video-based MLLMs (e.g., VideoLISA, VRS-HQ) struggle significantly, with J&F scores below 31.0, indicating a lack of predictive reasoning capability.
• Fine-Tuned Baselines: Standard fine-tuning of Sa2VA (without CoT/RL) improves performance to 35.8 J&F.
• FeVOS-R1: The proposed method achieves 42.3 J&F, a +6.5 improvement over the SFT-only baseline. This demonstrates the necessity of explicit reasoning chains and reward-guided optimization.

Generalization

• ReVOS & MeViS: FeVOS-R1 generalizes well to traditional RVOS benchmarks. On ReVOS, it achieves 60.3 J&F, outperforming the fine-tuned Sa2VA baseline (58.1) and zero-shot models, particularly in the reasoning subset. On MeViS, it reaches 49.5 J&F, surpassing comparable-sized models.
• Ablation Studies:
  • The two-stage pipeline (SFT + RL) is superior to SFT-only or RL-only training.
  • Using IoU reward alone in the RL stage outperforms combinations with format rewards, suggesting that once the format is learned in SFT, the model benefits from focusing purely on task accuracy.

Qualitative Analysis

Qualitative comparisons show that FeVOS-R1 successfully analyzes temporal context and causal relationships (e.g., predicting a cork will fly out due to pressure, or a laundry bag will fill based on ongoing actions), whereas baseline models often fail to distinguish between observed objects and future targets.

Significance and Claims

The paper claims that FeVOS advances pixel-level video understanding from observation to anticipation. While the absolute performance on FeVOS (42.3 J&F) remains lower than on traditional RVOS benchmarks (e.g., ReVOS at 60.3), this gap underscores the inherent difficulty of predictive reasoning. The work highlights that current models lack the spatio-temporal reasoning and world knowledge required for proactive decision-making.

The authors position this work as a foundation for future research in anticipatory visual understanding, noting that while FeVOS-R1 shows strong potential, challenges remain in scaling context, modeling transient cues, handling uncertainty, and enabling real-time inference.