Can Unsupervised Segmentation Reduce Annotation Costs for Video Semantic Segmentation?

1. Problem Statement

Video Semantic Segmentation (VSS) requires dense, pixel-level annotations for every frame to train deep neural networks effectively. However, creating these fine-grained annotations is prohibitively expensive and time-consuming (e.g., ~90 minutes per image for Cityscapes).
While large-scale datasets often contain:

1. Unannotated raw frames: Abundant and free.
2. Coarse annotations: Rough masks that cover objects approximately but lack precise boundaries (e.g., ~7 minutes per image).

Current deep learning models predominantly ignore these cheaper resources, relying solely on the small subset of fine-grained data. The paper investigates whether Foundation Models (specifically SAM and SAM 2) can be leveraged to generate pseudo-labels from unannotated frames or refine coarse annotations, thereby significantly reducing manual annotation costs without sacrificing model performance.

2. Methodology

The authors propose two primary strategies to augment training data using foundation models:

A. Pseudo-Labeling Unannotated Frames (Using SAM 2)

Since SAM 2 is a video segmentation foundation model, the authors use it to propagate labels from a manually annotated "keyframe" to surrounding unannotated frames.

• Process:
  1. Take a video clip where the 20th frame is manually annotated.
  2. Use the 20th frame mask as a prompt for SAM 2.
  3. Run SAM 2 forward to generate masks for future frames (up to frame 30).
  4. Run SAM 2 backward to generate masks for past frames (down to frame 10).
• Strategy: Instead of using all generated frames, the authors found that selecting frames with high temporal diversity (e.g., the 10th and 30th frames, furthest from the keyframe) yields better results than using sequential frames.

B. Refining Coarse Annotations (Using SAM)

For frames with coarse, rough masks, the authors use the image-based SAM to refine specific object boundaries.

• Process:
  1. Identify instance-level classes (e.g., car, person, traffic sign) within the coarse mask.
  2. Randomly select two points inside the coarse mask.
  3. Prompt SAM to generate a precise mask based on these points.
  4. Iterate this refinement process twice.
• Scope: Refinement is applied only to instance-level classes (8 classes in Cityscapes). Background classes (road, sky) and ambiguous boundaries (vegetation) are left as coarse annotations because SAM struggles with them in this context.

3. Experimental Setup

• Datasets:
  • Cityscapes: Used for both strategies (provides 2,975 fine frames, 20,000 coarse frames, and unannotated frames).
  • IDD (Indian Driving Dataset): Used to validate generalizability on unannotated frames.
• Backbones:
  • TMANet: A video segmentation model using temporal memory attention.
  • TDNet: A model using temporal feature propagation.
• Evaluation Metric: Mean Intersection over Union (mIoU).

4. Key Results

A. Reducing Annotation via Unannotated Frames

• 33% Annotation Reduction: By training on only one-third of the manually annotated frames and replacing the rest with SAM 2-generated pseudo-labels (specifically frames 10 and 30), the model achieved an mIoU of 74.48%, compared to 75.65% when using 100% manual data. This represents a ~1.2% drop in performance for a 67% reduction in annotation effort.
• 50% Reduction: Using 50% manual data + 50% pseudo-labels resulted in nearly identical performance (75.75% vs 75.65%).
• Failure of Sequential Frames: Using all generated frames (sequential) resulted in significantly lower performance. This highlights that frame diversity is more critical than the sheer quantity of data.

B. Utilizing Coarse Annotations

• Refinement Benefits: Refining coarse masks for instance classes using SAM improved performance compared to using raw coarse masks.
• Optimal Mix: A 50/50 mix of fine and coarse (refined) annotations yielded reasonable performance (73.39% mIoU for TMANet) at a fraction of the cost.
• Limitations: Refining all classes (including background) or using iterative "dog-feeding" (training a model to refine its own outputs) did not yield consistent improvements and sometimes degraded performance due to noise accumulation.

C. Cross-Dataset Validation

• Experiments on the IDD dataset confirmed the findings: using 33% manual data + SAM 2 pseudo-labels (B3) achieved 61.31% mIoU vs. 62.15% with 100% manual data, demonstrating the method's generalizability.

5. Key Contributions

1. Annotation Cost Reduction: Demonstrated that manual annotation requirements can be reduced by one-third while maintaining comparable VSS performance by leveraging foundation models for pseudo-labeling.
2. Diversity over Quantity: Identified that the variety of frames (temporal diversity) is more important than the number of frames when selecting pseudo-labels. Sequential frames introduce redundancy that harms learning.
3. Coarse Annotation Refinement: Established a pipeline to refine coarse instance-level masks using SAM, making them viable for training when mixed with fine data.
4. Negative Results Analysis: Provided a detailed analysis of failed approaches (e.g., using all sequential frames, refining non-instance classes, iterative self-training), offering valuable guidance for future research.

6. Significance

This work challenges the prevailing notion that high-quality VSS requires 100% fine-grained manual annotation. By treating foundation models (SAM/SAM 2) as "black box" pseudo-label generators, the authors show a practical pathway to data-efficient video segmentation.

The findings suggest that for practitioners, the most effective strategy is to ignore coarse annotations and focus on generating pseudo-labels for unannotated frames to reduce manual effort by ~66%. This approach democratizes the creation of large-scale video segmentation datasets, making it feasible to train robust models with significantly lower financial and time resources.