Multimodal Optimal Transport for Unsupervised Temporal Segmentation in Surgical Robotics

This paper introduces TASOT, an unsupervised multimodal optimal transport framework that leverages visual and text-based cues to achieve state-of-the-art surgical phase and step segmentation without relying on costly large-scale pre-training.

The Gist

Imagine you are watching a very long, complex movie of a surgeon performing an operation. The movie is hours long, the camera moves around a lot, and the "actors" (the surgical tools and organs) look very similar to each other.

The Problem:
If you wanted to break this movie down into a script with clear chapters (like "Cutting," "Stitching," "Cleaning"), you would normally need a team of expert surgeons to watch every single second and write down what is happening. This is incredibly expensive, slow, and boring.

Recently, scientists tried to teach computers to do this by showing them thousands of these movies first (pre-training). But this is like trying to teach a student to drive by making them sit in a simulator for 10,000 hours before they ever touch a real car. It works, but it's a huge waste of time and computing power.

The Solution (TASOT):
The researchers behind this paper asked a simple question: "Do we really need to force the computer to memorize thousands of hours of surgery first? Can't it just figure it out by looking at the video and reading the 'story' as it goes?"

They created a new tool called TASOT. Here is how it works, using a simple analogy:

The "Translator and Matchmaker" Analogy

Imagine you are trying to organize a messy library where the books have no titles, and you don't know the order they should go in.

1. The Visuals (The Pictures): You have a camera that takes a photo of every book cover.
2. The Text (The Story): You have a smart AI assistant that looks at the video and writes a running commentary, like a sports announcer. It says, "Okay, now the surgeon is cutting the tissue," then "Now they are sewing," then "Now they are cleaning up."
3. The Matchmaker (Optimal Transport): This is the magic part. TASOT acts as a super-smart matchmaker. It tries to pair every single photo (visual) with the perfect sentence from the commentary (text).

How TASOT is different:
Old methods tried to memorize the library first. TASOT says, "I don't need to memorize the library. I just need to match the picture of a red book with the sentence 'This is a red book'."

It uses a mathematical concept called Optimal Transport. Think of this as a delivery service.

• The Packages: The video frames (pictures).
• The Destinations: The surgical steps (like "Cutting" or "Sewing").
• The Cost: The system calculates the "cost" of moving a picture to a step.
  • If the picture looks like a knife, the cost to move it to the "Cutting" step is low.
  • If the picture looks like a needle, the cost to move it to the "Sewing" step is low.
  • The Twist: TASOT also checks the "commentary." If the text says "sewing," it lowers the cost of moving the picture to the "Sewing" step, even if the picture is a bit blurry.

By combining the picture and the text description, the system creates a perfect timeline of the surgery without ever needing a human to label the video beforehand.

Why is this a big deal?

• No Heavy Lifting: It doesn't need to be trained on massive, expensive datasets of labeled surgeries. It works "out of the box" on new types of surgeries.
• Better than "Zero-Shot": "Zero-shot" methods are like a student who has read a textbook but never seen a real surgery. They guess based on general knowledge. TASOT is like a student who has the textbook and a live commentary, allowing them to guess much more accurately.
• The Results: When they tested it on real surgical videos (like gallbladder removals and bypass surgeries), TASOT beat the best existing "guessing" methods by a huge margin. In some cases, it was 23% more accurate.

The One Catch

The system is currently a bit rigid. It's like a train that must stop at exactly 10 stations, even if the trip only has 8 stops. The researchers found that if they let the system decide how many stops (steps) are actually in the video, it gets even better. But even with this small limitation, it's a massive leap forward.

The Bottom Line

This paper proves that we don't need to build giant, expensive "surgery brains" to understand surgical videos. Instead, we can use a clever combination of what we see (the video) and what is being said (the text description) to automatically break down complex surgeries into clear, understandable steps. It's a smarter, cheaper, and faster way for robots to understand what they are doing in the operating room.

Technical Summary

1. Problem Statement

The paper addresses the challenge of unsupervised temporal action segmentation in surgical videos.

• Context: Recognizing surgical phases and steps is critical for intraoperative guidance, skill assessment, and autonomous robotics.
• Limitations of Current Methods:
  • Fully Supervised Methods: Require dense, frame-level annotations by medical experts, which are prohibitively expensive and time-consuming.
  • Zero-Shot/Transfer Methods: Recent approaches rely on large-scale pre-training on thousands of labeled videos (e.g., SurgVLP, HecVL). While effective, these methods incur massive computational costs and require complex, domain-specific pre-training pipelines.
• Core Question: Is large-scale surgical pre-training truly necessary, or can competitive performance be achieved using a fully unsupervised approach that leverages existing visual and textual representations?

2. Methodology: TASOT

The authors propose TASOT (Text-Augmented Action Segmentation Optimal Transport), a fully unsupervised framework that extends the Action Segmentation Optimal Transport (ASOT) method by integrating textual cues.

A. Pipeline Overview

1. Captioning Pipeline:
   • Long surgical videos are divided into non-overlapping temporal windows (e.g., 5-minute clips).
   • A Large Language Model (Gemini 2.0 Flash) generates dense, natural language descriptions (captions) for these windows, creating a temporally aligned text sequence.
2. Feature Extraction:
   • Visual: Frames are sampled at 1 fps and encoded using DINOv3 to extract visual features ($x_{img}$).
   • Textual: Generated captions are encoded using CLIP to extract text features ($x_{text}$).
   • Alignment: Text features are temporally aligned with video frames based on the caption timestamps.
3. Multimodal Optimal Transport (OT) Formulation:
   • TASOT learns $K$ normalized prototypes ($A$) in a latent space.
   • It introduces modality-specific projection heads to map visual and textual features into a shared latent space ($z_{img}$ and $z_{text}$).
   • Cost Matrix Construction: A weighted multimodal cost is defined to measure the distance between features and prototypes:
     $$C_{i,k} = \beta C_{img}^{i,k} + (1 - \beta) C_{text}^{i,k}$$
     Where $C_{img}$ and $C_{text}$ are cosine distances, and $\beta$ controls the trade-off between visual and text cues.
   • Regularization: The problem is formulated as an unbalanced Gromov-Wasserstein Optimal Transport problem. This ensures temporal consistency and monotonic alignment between the video sequence and the action prototypes without requiring predefined action ordering.
4. Self-Training:
   • The resulting transport plan provides pseudo-labels.
   • These pseudo-labels are used to optimize the representation learning via cross-entropy loss, refining the embeddings and prototypes iteratively.

3. Key Contributions

1. First Multimodal OT Framework for Surgery: TASOT is the first method to integrate visual and textual cues within an unsupervised Optimal Transport objective for surgical temporal segmentation.
2. Elimination of Heavy Pre-training: The method achieves state-of-the-art performance without requiring large-scale surgical-specific pre-training or massive backbone architectures tailored to surgical data. It relies on off-the-shelf encoders (DINOv3, CLIP).
3. Superior Performance: The approach consistently outperforms existing zero-shot and supervised baselines across multiple datasets.
4. Insight into Cluster Flexibility: The authors demonstrate that while a fixed number of clusters is standard, adapting the number of clusters ($k$) to match the specific variability of individual videos significantly boosts performance, particularly for fine-grained step segmentation.

4. Experimental Results

The model was evaluated on three public datasets: Cholec80, AutoLaparo, and MultiBypass140 (Bern and Strasbourg centers). The primary metric was Segmental F1 Score.

• Comparison with Zero-Shot Methods: TASOT significantly outperformed the strongest zero-shot baselines (e.g., PeskaVLP, HecVL).
  • StrasBypass70: +23.7% improvement in Phase F1.
  • BernBypass70: +4.5% improvement.
  • Cholec80: +16.5% improvement.
  • AutoLaparo: +19.6% improvement.
• Ablation Studies:
  • Multimodal Fusion: The weighted cost matrix approach (TASOT) outperformed using visual-only, text-only, or simple feature concatenation, proving the efficacy of the OT-based fusion.
  • Encoders: The combination of DINOv3 (visual) and CLIP (text) yielded the best results, outperforming variants using Gemma for text encoding.
  • Cluster Adaptation: When the number of clusters was adapted to the specific video content (rather than fixed globally), performance on BernBypass70 jumped from 23.0 to 48.8 (Step F1), surpassing even some supervised baselines (TeCNO).

5. Significance and Conclusion

• Paradigm Shift: TASOT challenges the prevailing assumption that effective surgical video understanding requires massive, expensive pre-training datasets. It demonstrates that fine-grained understanding can be achieved by exploiting information already present in standard visual and textual representations.
• Efficiency: By avoiding heavy pre-training and using standard encoders, the method offers a computationally efficient alternative for surgical robotics applications.
• Generalizability: While focused on surgery, the framework is applicable to any domain involving long, untrimmed procedural videos where aligned textual cues can be generated.
• Future Work: The authors identify the adaptive estimation of the number of clusters as a promising direction to further close the gap with supervised methods.

In summary, TASOT provides a robust, unsupervised solution for surgical phase and step recognition, leveraging multimodal optimal transport to achieve state-of-the-art results without the computational and data burdens of current large-scale pre-training strategies.