💻 computer science

Temporally Consistent Label Interpolation for Robust Surgical Multi-Task Learning under Challenging Conditions

The paper proposes FAROS, a flow-guided label interpolation framework that generates temporally consistent dense pseudo-labels from sparse annotations to overcome supervision imbalances and enhance robust multi-task learning for surgical scene understanding under challenging conditions.

Original authors: Garam Kim, Juyoun Park

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Garam Kim, Juyoun Park

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand a complex surgery video. You want the robot to do two very different jobs at the same time:

The "Storyteller" Job: It needs to tell you what "chapter" of the surgery is happening right now (e.g., "cutting," "stitching," "tying"). This is easy to teach because you can just label every single second of the video with the current chapter.
The "Spotter" Job: It needs to point out exactly where every single surgical tool is on the screen, pixel by pixel. This is incredibly hard to teach because it takes a human expert hours to draw a perfect outline around a tool in just one frame.

The Problem: The "Labeling Mismatch"
The paper explains that trying to teach the robot both jobs at once usually fails. Why? Because the robot gets confused by the imbalance. It has thousands of examples of "chapters" (dense labels) but only a handful of examples of "tool outlines" (sparse labels). It's like trying to learn a language where you have a dictionary for every word, but only one picture for every object. The robot learns to be great at guessing the chapter but terrible at finding the tools, and because the two jobs are connected, the whole system gets messy.

The Solution: FAROS (The "Smart Time-Traveler")
The authors created a system called FAROS (Flow-guided Annotation for Robust Operating Scenes). Think of FAROS as a smart assistant that fills in the missing pictures for the robot.

Here is how it works, using a simple analogy:

The Starting Point: Imagine you have a few "keyframes" (like snapshots) where an expert has perfectly drawn the tools.
The "Guessing" Engine (SAM2): The system uses a powerful AI called SAM2 to guess what the tools look like in the frames between those snapshots. It's like a time-traveler who looks at a photo and tries to guess what happens next.
The Problem with Guessing: In surgery, things get messy. Smoke from a tool, blood, or a tool moving too fast can confuse the time-traveler. It might lose track of a tool or draw it in the wrong place.
The "Motion Detective" (Optical Flow): This is the secret sauce. While the time-traveler guesses based on what things look like (color, shape), the Motion Detective watches how things move (geometry). Even if a tool is covered in smoke or looks blurry, the Motion Detective knows exactly where it should be based on its movement path.
The Fix (Reprompting): If the Motion Detective sees that the time-traveler is getting lost (e.g., the tool disappeared or the mask is drifting), it steps in. It takes the perfect snapshot from the nearest "keyframe," warps it to the current moment using the movement path, and says, "Hey, try again! Here is the correct shape."

The Result: A Balanced Team
Once FAROS fills in all the missing tool outlines for every single frame, the robot has a complete, balanced dataset. It now has:

Dense labels for the "Story" (chapters).
Dense labels for the "Spots" (tools).

The paper shows that when they train the robot with this balanced data, it becomes much better at both jobs. The robot learns that "if I see a needle here, I'm probably in the 'stitching' chapter," and conversely, "if I'm in the 'stitching' chapter, I should expect to see a needle."

Why This Matters (According to the Paper)
The authors tested this on real surgical video datasets (GraSP, MISAW, and AutoLaparo). They found that:

Without FAROS, trying to learn both jobs at once actually made the robot worse than if it learned them separately.
With FAROS, the robot improved significantly across the board, getting better at recognizing steps, predicting future steps, and spotting tools, even in difficult conditions like smoke or fast motion.

In short, the paper claims that by using "motion clues" to fix "visual guesses," they created a way to teach robots to understand surgery videos more holistically, without needing humans to draw every single tool outline by hand.

Technical Summary: Temporally Consistent Label Interpolation for Robust Surgical Multi-Task Learning under Challenging Conditions

1. Problem Statement

The paper addresses a fundamental barrier in unified surgical multi-task learning (MTL): annotation granularity mismatch.

Temporal Tasks: Workflow-related tasks (phase recognition, step recognition, anticipation) typically benefit from dense, frame-level supervision available for nearly every video frame.
Spatial Tasks: Pixel-level tasks (instrument segmentation, action recognition) require labor-intensive expert annotation involving specialized clinical knowledge. Consequently, these are often only sparsely annotated on selected keyframes due to prohibitive labeling costs.

This imbalance creates a supervision gap that undermines shared representation learning. Joint optimization of heterogeneous tasks is limited because the model lacks dense spatial cues to ground temporal predictions, while temporal context fails to regularize sparse spatial predictions effectively. Furthermore, directly propagating sparse labels in surgical videos is challenging due to complex intraoperative conditions (occlusion, smoke, motion blur, illumination variation), which cause appearance-based propagation methods (like SAM2) to fail, resulting in temporally inconsistent pseudo-labels.

2. Methodology

The authors propose FAROS (Flow-guided Annotation for Robust Operating Scenes), a framework designed to generate temporally consistent, dense pseudo-labels from sparse keyframe annotations, followed by a unified multi-task learning architecture.

A. FAROS: Flow-Guided Label Interpolation
FAROS combines zero-shot segmentation-based mask propagation with optical flow estimation to overcome the limitations of appearance-based methods in surgical environments.

Bidirectional SAM2 Prompting: Instead of propagating unidirectionally from a single keyframe, the system registers ground-truth masks from both boundary keyframes ( $k_0$ and $k_1$ ) into the SAM2 memory bank. This allows the model to reference anchors from both ends of a segment, improving temporal stability.
Flow Consistency-Based Anomaly Detection: To detect propagation failures (e.g., due to occlusion or instrument exit), the system utilizes the forward-backward consistency property of optical flow (estimated via RAFT). It computes a per-pixel consistency error. If the error exceeds a threshold ( $\tau$ ) for a consecutive number of frames ( $n_{confirm}$ ), the propagation is flagged as anomalous.
Flow-Guided Reprompting: Upon detecting an anomaly, the system computes the backward flow from the nearest verified keyframe to the current frame. It warps the ground-truth mask from that keyframe to generate a geometrically grounded "corrective prompt." This prompt is fed back into the SAM2 prompt encoder to re-initialize propagation, effectively recovering the mask where appearance-based attention failed.

B. Unified Multi-Task Learning Framework
The densified pseudo-labels are integrated into a unified Transformer-based framework that jointly learns five tasks: surgical phase recognition, step recognition, anticipation, instrument segmentation, and action recognition.

Architecture: The framework employs a Mask2Former-based Region Proposal Network (RPN) as a baseline for instrument segmentation. In Stage 1, the RPN is trained on ground-truth and pseudo-masks to produce region embeddings. In Stage 2, the RPN is frozen, and these precomputed embeddings are reused.
Spatio-Temporal Coupling: An MViT backbone extracts spatio-temporal features. Dedicated CLS tokens are assigned to frame-level tasks (phase, step).
Cross-Task Conditioning: The framework explicitly models dependencies between tasks via lightweight conditioning modules:
- Phase $\to$ Step: Phase features inform step prediction.
- Spatial $\to$ Temporal: Pooled region embeddings (instrument presence/configuration) are injected into phase and step prediction heads, grounding workflow recognition in pixel-level instrument understanding.
Optimization: The model uses a weighted sum of task-specific losses, activating them only when corresponding annotations (or pseudo-labels) are available, enabling balanced optimization across heterogeneous supervision signals.

3. Key Contributions

Identification of Annotation Granularity Mismatch: The paper identifies the disparity between dense temporal and sparse spatial supervision as a critical barrier to unified surgical MTL and proposes FAROS to bridge this gap.
FAROS Framework: A novel flow-guided label interpolation method that augments zero-shot segmentation (SAM2) with optical flow guidance. It specifically addresses failure modes in challenging surgical conditions (occlusion, smoke, motion blur) by using geometric motion priors to detect anomalies and reprompt the segmentation model.
Unified Spatio-Temporal MTL: A unified Transformer-based framework that jointly models workflow-level and region-level tasks, leveraging the densified spatial supervision to enable balanced cross-task representation learning.
Empirical Validation: Extensive experiments on GraSP, MISAW, and AutoLaparo benchmarks, along with validation on the DAVIS 2017 dataset under a sparse ground-truth protocol.

4. Results

DAVIS 2017 Validation: Under a sparse ground-truth protocol (keyframes every 30 frames), FAROS outperforms the SAM2 baseline in region similarity ( $\mathcal{J}$ ) and contour accuracy ( $\mathcal{F}$ ), demonstrating robust propagation even with long inter-keyframe intervals and challenging motion.
GraSP Benchmark: The full model with flow-guided interpolation ("All w/ inter") significantly improves workflow-level tasks compared to single-task baselines and multi-task models without interpolation. Notably, step recognition shows strong gains. While a marginal quantitative decline in mIoU was observed compared to the non-interpolated multi-task model, qualitative analysis indicated visually superior segmentation with fewer false positives and better temporal consistency.
MISAW Benchmark: The proposed method achieved the best results across nearly all metrics, with particularly strong improvements in workflow recognition and step anticipation. Instrument segmentation also recovered and surpassed single-task baselines.
AutoLaparo Benchmark: Used to isolate the effect of label interpolation, results showed that models trained with interpolated pseudo-labels consistently improved segmentation performance (IoU, mIoU) over baselines without interpolation, even without joint multi-task training.
Efficiency: The framework achieves real-time capability (approx. 44 clips/s on a single GPU) by decoupling the computationally heavy mask generation (Stage 1, frozen) from the multi-task inference (Stage 2).

5. Significance and Claims

The paper claims that flow-guided label interpolation is essential for realizing the benefits of multi-task learning in surgical scene understanding. Without interpolation, joint training often degrades performance relative to single-task baselines due to the annotation imbalance.

The authors assert that:

Spatial-Temporal Coupling: Instrument spatial configurations serve as structured priors for workflow recognition. By densifying spatial supervision, the framework provides richer cues for phase and step transitions.
Robustness: The combination of appearance-based segmentation and geometry-based flow correction effectively handles clinically challenging conditions (e.g., instrument exit/re-entry, smoke, blood occlusion) where pure appearance-based models fail.
Generalizability: The approach is validated across diverse surgical domains (robot-assisted human surgery, microsurgical training, laparoscopic surgery) and general video scenarios, suggesting the method is robust to varying annotation densities and visual complexities.

The work positions itself as a foundation for annotation-efficient holistic surgical scene understanding, demonstrating that bridging the supervision gap allows for balanced cross-task optimization that improves performance across all tasks simultaneously.