Understanding Image2Video Domain Shift in Food Segmentation: An Instance-level Analysis on Apples

Technical Summary: Understanding Image2Video Domain Shift in Food Segmentation

1. Problem Statement

Current food segmentation research is heavily focused on static image benchmarks (e.g., FoodSeg103, UECFoodPix). While models trained on these datasets achieve high pixel-level accuracy (mIoU), they often fail catastrophically when deployed in real-world video streams.

The authors identify a fundamental "Image-to-Video Domain Shift." In video, food appearance is not static; it evolves due to illumination changes, specular reflections (common on shiny food surfaces like apples), motion blur, and occlusions. Standard evaluation metrics (frame-wise mIoU) fail to capture these temporal dynamics, leading to a massive overestimation of model reliability. In practice, this results in "mask flickering" and "identity fragmentation," making downstream tasks like automated food counting or portion estimation highly inaccurate.

2. Methodology

To diagnose this gap, the researchers moved away from frame-independent analysis and reformulated the task as an Instance Segmentation and Tracking problem.

• Experimental Setup: Models (CNN-based, Transformer-based, and Foundation-based) were trained exclusively on static images to simulate a realistic deployment scenario where video-level supervision is unavailable.
• Tracking-by-Matching Framework: To evaluate temporal stability without ground-truth video labels, the authors applied a tracking-by-matching strategy. They used the segmentation masks from consecutive frames and associated them based on Spatial Overlap (IoU) and appearance similarity.
• Evaluation Metrics: The study introduced a dual-layered evaluation approach:
  • Spatial Metrics: Frame-wise mIoU (to measure per-frame accuracy).
  • Temporal Metrics: ID Switches (how often an object's identity changes) and Track Fragmentation (how often a single object is split into multiple tracks).
  • Downstream Task: Apple Instance Counting, to measure the practical impact of temporal errors.
• Remedies: The authors tested two lightweight, non-supervised methods to mitigate failures:
  1. Post-hoc Temporal Regularization: Smoothing predictions using short-term temporal windows.
  2. Self-supervised Temporal Consistency: Training objectives that encourage adjacent frames to have consistent predictions.

3. Key Contributions

• Identification of the Evaluation Gap: Proved that high frame-wise mIoU does not correlate with temporal stability or reliable instance tracking.
• Instance-Level Analysis: Shifted the focus from pixel-level accuracy to object-level temporal consistency using apples as a representative case study.
• Failure Mode Taxonomy: Categorized the primary video failures into mask flickering, identity fragmentation, and identity switching.
• Practical Mitigation Strategies: Demonstrated that while temporal consistency can be improved via self-supervision or smoothing, the root cause is the lack of temporal awareness in the initial training objectives.

4. Results

• The Discrepancy: All models performed well on image tests. However, in video, even "Foundation-based" models (which had the highest mIoU) suffered from significant identity switches and fragmentation.
• Counting Errors: The temporal instability led to severe over-counting. For example, in a scenario with 12 true apples, models predicted counts as high as 18 due to identity fragmentation (one apple being treated as multiple objects over time).
• Model Comparison: Interestingly, the ranking of models based on image accuracy did not predict their performance in video, proving that image-centric benchmarks are misleading for video deployment.
• Effectiveness of Remedies: Self-supervised temporal consistency was more effective than post-hoc smoothing, reducing ID switches and counting errors, though it did not fully bridge the gap.

5. Significance

This paper serves as a critical wake-up call for the food analysis community. It argues that model capacity is not the bottleneck; training objectives are.

The work highlights that for food segmentation to be useful in real-world applications (like automated dietary monitoring or industrial food processing), the field must move toward temporally-aware learning and video-centric evaluation protocols. It shifts the research paradigm from "how well can we segment a frame?" to "how reliably can we track an instance through time?"