Technical Summary: An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentation

1. Problem Definition

The paper addresses a gap in Large Vision–Language Models (LVLMs): while they excel at static visual understanding and language-guided grounding, their capacity for multi-temporal visual reasoning remains underexplored. Existing language-guided segmentation tasks (Open-Vocabulary, Referring, and Reasoning Segmentation) typically operate on single-time images, assuming the target can be identified from a static observation.

However, real-world scenarios often require identifying changes over time based on natural language. For instance, locating a vehicle that "disappeared" between two observations or segmenting a forested region that "vanished." This necessitates a task that jointly requires:

1. Temporal Correspondence Reasoning: Comparing temporally related images to identify changes.
2. Language Grounding: Understanding specific natural language instructions describing the change.
3. Pixel-Level Mask Prediction: Segmenting the specific region corresponding to the described temporal variation.

The authors define this new task as Multi-temporal Referring Segmentation (MTRS). In its primary formulation, MTRS takes a pair of temporally separated images ($I_{t1}, I_{t2}$) and a natural language expression ($x$) describing a specific change (e.g., "appeared," "disappeared," "newly placed") as input, and outputs a binary mask ($M$) for the referred changed region.

2. Methodology

A. Dataset Construction: CRAFT-Agent and MTRefSeg-21K

To facilitate systematic study, the authors constructed MTRefSeg-21K, the first large-scale benchmark for MTRS.

• CRAFT-Agent: An automated data construction pipeline with human auditing. It operates in three stages:
  1. Grid-Aware Change Perception: A multimodal LLM examines bi-temporal image pairs to identify differences, generating initial referring expressions with spatial cues and change categories (appear/disappear/state change) alongside rough masks.
  2. Iterative Refinement: A vision-language segmentation model progressively improves mask boundary accuracy.
  3. Expression Beautification: Mechanical descriptions are rewritten into natural, contextually coherent referring expressions.
• Dataset Statistics: MTRefSeg-21K contains 21,000 high-quality bi-image–text–mask triplets across diverse domains, including Normal Scenes (NS) and Remote Sensing (RS). It covers various viewpoints (aerial, normal), scenes (urban, industrial, seasonal), and resolutions (224² to 2048²).

B. Baseline Adaptations

The authors adapted existing VLMs and LVLMs to the MTRS setting to establish baselines:

• VLM Adaptation: Replaced single-image inputs with paired temporal images. Features are encoded by a shared visual encoder and fused via a temporal fusion operator (concatenation, differencing, etc.) before being fed to the segmentation decoder.
• LVLM Adaptation: Reformulated prompts to explicitly state temporal order (e.g., " is the earlier image..."). The fused visual representation and the prompt are processed by the multimodal LLM, with the hidden state of a special [SEG] token guiding the mask decoder.

C. Proposed Model: MTRefSeg-R1

To overcome the limitations of direct inference and simple fine-tuning, the authors propose MTRefSeg-R1, a change-aware LVLM framework trained via a two-stage strategy:

1. Stage 1: Multi-temporal Vision Pretraining (Vision-Only):

   • The model is trained on ~20K vision-only bi-temporal change samples (without language).
   • Goal: Learn general temporal-change perception, cross-temporal correspondence, and change localization before exposure to language instructions.
   • Architecture: Uses a shared multi-scale vision encoder and a Change-Aware Fusion Block. This block explicitly models temporal differences by computing both signed differences (preserving direction: appearance vs. disappearance) and absolute differences (magnitude of change), fusing them with a gating mechanism to control information injection.

2. Stage 2: Referring Multi-temporal Fine-tuning:

   • The model is fine-tuned on MTRefSeg-21K using bi-temporal image–text–mask triplets.
   • Goal: Learn to align fine-grained language instructions with specific temporal-change regions.
   • Mechanism: The LVLM processes the temporal image tokens and the referring expression. The hidden state of the [SEG] token is projected into a language-conditioned segmentation query, which guides the mask decoder to output the specific region described by the text, rather than all changed regions.

Training Objective: A joint loss function combining language modeling loss, classification loss, mask prediction loss (BCE), and Dice loss (to handle foreground-background imbalance).

3. Key Contributions

1. Task Definition: Introduction of Multi-temporal Referring Segmentation (MTRS), unifying temporal change understanding, natural language grounding, and pixel-level segmentation.
2. Benchmark: Construction of MTRefSeg-21K via the CRAFT-Agent, providing the first large-scale dataset with 21K high-quality triplets across normal and remote sensing domains.
3. Methodology: Proposal of a two-stage training strategy (vision-only pretraining followed by language-guided fine-tuning) and the MTRefSeg-R1 framework, which explicitly models cross-temporal visual differences.
4. Evaluation: Comprehensive benchmarking of existing VLMs and LVLMs, demonstrating that direct inference fails and that task-specific adaptation is crucial.

4. Experimental Results

Extensive experiments were conducted on MTRefSeg-21K across three settings: Overall (Train→Val), Normal Scene (NS→NS), and Remote Sensing (RS→RS).

• Baseline Performance: Direct inference of existing LVLMs (e.g., LISA, GSVA) without fine-tuning performed poorly (mIoU ~2–18%), indicating that single-temporal pretraining is insufficient for temporal reasoning.
• Fine-tuning Impact: Fine-tuning existing LVLMs on MTRefSeg-21K significantly improved performance (mIoU ~50–60%), validating the necessity of the benchmark.
• MTRefSeg-R1 Performance: The proposed model achieved state-of-the-art results, outperforming the strongest fine-tuned LVLM baseline (SegEarth-R1) by 6.08 mIoU in the overall setting (65.68 vs. 59.60) and achieving superior precision at high thresholds (Pr@80, Pr@90).
  • NS Domain: 59.87 mIoU.
  • RS Domain: 68.92 mIoU.
• Ablation Studies:
  • Fusion: The proposed "change_aware" fusion block outperformed simple averaging or pure differencing, particularly in the fine-tuning stage.
  • Pretraining: Stage-1 vision-only pretraining significantly boosted performance compared to end-to-end MLLM pretraining or no pretraining, confirming the value of learning change perception before language grounding.
  • Fine-tuning: Full fine-tuning outperformed parameter-efficient LoRA tuning, suggesting MTRS requires substantial adaptation of the visual encoder and fusion modules.

5. Significance and Claims

The paper claims that MTRS represents a significant step forward in interactive and scenario-driven intelligence, moving language-guided perception from static understanding to temporal reasoning in dynamic scenes.

• Limitation of Current Models: The work highlights that current LVLMs struggle to distinguish language-relevant temporal changes from irrelevant visual differences or to localize specific changed objects without explicit training.
• Necessity of Two-Stage Learning: The results suggest that acquiring general temporal-change perception (via vision-only pretraining) is a prerequisite for effective language-guided temporal localization.
• Benchmark Utility: MTRefSeg-21K serves as a unified yet domain-aware benchmark for both general-scene and remote-sensing communities, enabling systematic analysis of cross-domain generalization.

The authors conclude that while MTRefSeg-R1 demonstrates strong performance, challenges remain regarding severe viewpoint changes, extremely small objects, and ambiguous expressions, pointing to future directions for longer temporal sequences and improved robustness.