💻 computer science

On the Robustness of Temporal Vision-Language Models for Surgical Endoscopy Videos

This paper introduces the Endo-C6 benchmark and the RobustEndoCLIP model to demonstrate that while off-the-shelf temporal vision-language models suffer severe performance collapse under endoscopy-specific video corruptions, lightweight few-shot adaptation can significantly enhance their robustness without altering the prompt-based interface.

Original authors: Darakshan Rashid, Raza Imam, Ufaq Khan, Muhammad Bilal, Shazad Ashraf, Dwarikanath Mahapatra, Mohammad Yaqub, Muhammad Haris Khan, Imran Razzak, Brejesh Lall, Lena Maier-Hein, Yutong Xie

Published 2026-08-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Darakshan Rashid, Raza Imam, Ufaq Khan, Muhammad Bilal, Shazad Ashraf, Dwarikanath Mahapatra, Mohammad Yaqub, Muhammad Haris Khan, Imran Razzak, Brejesh Lall, Lena Maier-Hein, Yutong Xie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a super-smart robot to watch videos. You don't want to teach it one specific game at a time; instead, you give it a general "brain" that understands both what it sees and what words mean. This is called a Vision-Language Model. Think of it like a student who has read every book in the library and watched every movie ever made, so you can just ask them, "What is happening in this video?" and they answer based on what they know. These robots are becoming very popular for watching surgery videos, where doctors need to quickly identify tools, phases of an operation, or safety issues without spending years labeling every single frame.

However, there is a catch. The robot was trained on perfect, high-definition, studio-quality videos. But real-life surgery videos are messy. They are like watching a movie through a foggy window, with the camera shaking, smoke from a laser cutting tool, or the signal freezing up because of a bad internet connection. The big question scientists are asking is: If you show this super-smart robot a messy, real-world surgery video, does it still work? Or does it get confused and start hallucinating nonsense? This paper dives into that exact problem, testing how well these AI models hold up when the video quality goes "off the rails."

The Robot's Reality Check: When Surgery Videos Get Messy

Meet the Temporal Vision-Language Model (TVLM). You can think of these as the "multitasking geniuses" of the AI world. Instead of being trained to recognize just one thing (like "is this a scalpel?"), they are trained to understand the whole story of a video by matching moving images with text descriptions. Surgeons love them because they can be asked questions like, "What phase of surgery is this?" or "What tool is being used?" without needing a custom-built brain for every single question.

But here is the plot twist: these geniuses were raised in a sterile, perfect world. The videos they learned from were clean, steady, and bright. Real surgery, however, is a chaotic adventure. The camera might get foggy from the heat, the lens might get blurry, smoke from a cautery tool (a device that burns tissue) might fill the screen, or the video might glitch due to packet loss (like when your video call freezes).

The authors of this paper, a team of researchers from places like MBZUAI and the German Cancer Research Center, decided to put these AI models to the ultimate stress test. They asked: If we throw these messy, real-world problems at the models, will they crumble?

The "Endo-C6" Obstacle Course

To answer this, the team built a special obstacle course called Endo-C6. Imagine a video game level where you have to navigate six different types of disasters:

Defocus: The camera loses its sharpness, like a blurry photo.
Fog (Haze): The view gets cloudy, like looking through a steamy bathroom mirror.
Shot Noise: The image gets grainy, like an old TV with static.
Motion Blur: The camera moves too fast, smearing the image.
Packet Loss: The video skips or freezes, like a buffering internet stream.
Cautery Smoke: Thick smoke obscures the view, common when surgeons burn tissue.

They took three popular AI models (SurgVLP, HecVLP, and PeskaVLP) and ran them through this obstacle course using real surgery videos from three different datasets: laparoscopic surgery (inside the belly), GI endoscopy (looking down the throat or gut), and colorectal surgery.

The Shocking Results: The "Collapse"

The results were a bit scary for the AI community. When the videos were clean, the models did okay. But as soon as the "disasters" hit, the models suffered what the authors call a "worst-case collapse."

Think of it like a student who aced a math test in a quiet library but, when asked to solve the same problems while standing in a hurricane with a foggy visor, forgot how to count.

On one dataset (CholecT50), the models' accuracy dropped from about 40% on clean videos to as low as 7% when smoke or blur was introduced.
On another dataset (TEMSET-24K), the models were even worse, dropping to a terrifying 0.4% accuracy on some corrupted clips. That's basically random guessing.

The paper explicitly rules out the idea that these models are ready for prime time in their current "off-the-shelf" form. They are too fragile. A small amount of smoke or blur can make them completely useless, which is dangerous in a surgery where safety is everything.

The Hero: A Lightweight "Tuning" Trick

But don't panic! The paper doesn't just point out the problem; it offers a solution. The researchers developed a new model called RobustEndoCLIP.

Instead of retraining the entire giant brain of the AI (which takes forever and needs massive amounts of data), they used a clever trick called VeRA (Vector-based Random Matrix Adaptation). Imagine the AI model is a massive, heavy library. Retraining it is like rebuilding the whole library. VeRA is like adding a tiny, smart index card system to the front desk. It's incredibly small and efficient.

They took a tiny slice of clean data (just 16% of the available training clips) and used this "index card" trick to gently nudge the model to understand the messy world better.

The result? RobustEndoCLIP didn't just survive the obstacle course; it thrived.

While the old models dropped to 7% accuracy in the worst smoke scenarios, the new model held steady at 16.83%.
On the hardest dataset (TEMSET-24K), it improved the worst-case accuracy from a dismal 0.40% to a much more reliable 19.98%.

The paper shows that this lightweight tuning makes the model much more robust, especially in the "worst-case" scenarios, without changing how doctors interact with it. You still ask it questions in plain English, and it gives you better answers even when the video is messy.

What This Means for the Future

The authors are careful not to say they have "solved" surgery AI. They suggest that their new benchmark, Endo-C6, should become the standard way to test these models. Before we trust an AI to help in an operating room, we need to know how it handles smoke, blur, and fog.

The study concludes that while current models are fragile, a little bit of smart, efficient tuning can make them much tougher. It's a reminder that in the messy, real world of surgery, being "smart" isn't enough; you also need to be "tough." And with tools like RobustEndoCLIP, we might be getting closer to AI that can handle the chaos of the operating room without losing its cool.

Technical Summary: On the Robustness of Temporal Vision-Language Models for Surgical Endoscopy Videos

Problem Statement

Temporal Vision-Language Models (TVLMs) offer a reusable, prompt-based interface for surgical video understanding, eliminating the need for bespoke classifiers for every new label set. However, their robustness under clinically realistic acquisition artifacts in endoscopy remains insufficiently characterized. Surgical endoscopy is intrinsically noisy, subject to structured distribution shifts caused by defocus, haze, motion blur, noise, cautery smoke, and packet loss. These artifacts, which are routine during procedures (e.g., during cautery or rapid repositioning), can compromise video-text alignment. Current robustness benchmarks often fail to isolate artifact-driven failures from shifts in procedure mix or labels, and existing studies on medical VLMs frequently omit temporal, prompt-based models or focus on specific tasks like depth estimation rather than general clip classification.

Methodology

The authors propose a comprehensive framework to stress-test TVLMs and improve their robustness through efficient adaptation.

1. Endo-C6 Benchmark:
The paper introduces Endo-C6, a compact corruption benchmark designed to evaluate TVLMs under six endoscopy-realistic perturbations:

Defocus: Simulating lens focus loss.
Fog (Haze): Mimicking fluid or lens fog.
Shot Noise: Simulating low-light sensor noise.
Motion Blur: Representing rapid camera movement.
Packet Loss: Simulating transmission errors in live streams.
Smoke: Modeling cautery smoke plumes.

These corruptions are applied at a fixed high severity (severity 5) to clips from three public datasets: CholecT50 (laparoscopic cholecystectomy phases), Kvasir (GI landmark/pathology), and TEMSET-24K (surgical actions). The evaluation protocol treats clean clips as In-Distribution (ID) and corrupted variants as controlled Out-Of-Distribution (OOD) shifts, measuring performance via Mean-C (average accuracy across corruptions) and Worst-C (lowest accuracy among corruptions).

2. RobustEndoCLIP:
To address the fragility of off-the-shelf models, the authors propose RobustEndoCLIP. This approach utilizes few-shot parameter-efficient tuning with VeRA (Vector-based Random Matrix Adaptation).

Architecture: It utilizes a pretrained temporal CLIP backbone (specifically initialized from SurgVLP with a ResNet-50 visual encoder).
Adaptation Strategy: Instead of full fine-tuning or standard LoRA, the method freezes the backbone and updates only the final fully connected (FC) projection layer. VeRA achieves this by fixing random matrices $A$ and $B$ and learning only scaling vectors ( $\Lambda_b, \Lambda_d$ ), significantly reducing the number of trainable parameters compared to LoRA.
Training: The model is adapted using a small labeled subset (up to 16%) of clean training data, preserving the prompt-based deployment interface while "anchoring" the embedding space to the target domain.

Key Contributions

Endo-C6 Benchmark: A diverse corruption benchmark covering six realistic artifact types across GI endoscopy and laparoscopic surgery, organized by domain salience and temporal structure.
Systematic Benchmarking Study: A reproducible prompt-based evaluation protocol applied to three representative surgical TVLM baselines (SurgVLP, HecVLP, and PeskaVLP) across 294 dataset-level evaluations.
RobustEndoCLIP: A specialized temporal VLM that achieves substantial robustness gains via lightweight few-shot adaptation, demonstrating that parameter-efficient tuning can improve performance without altering the prompt-based interface.

Results

The study evaluated 3 baseline models and the proposed RobustEndoCLIP across the three datasets under clean and corrupted conditions.

Baseline Fragility: Off-the-shelf TVLMs exhibit severe worst-case collapse under endoscopy-specific corruptions. For instance, on CholecT50, baseline Worst-C accuracy dropped to approximately 7% under smoke and motion blur, despite moderate clean accuracy (~40-44%). On TEMSET-24K, baselines showed Worst-C as low as 0.40%.
Corruption Sensitivity: Failure modes are domain-specific. Laparoscopic videos (CholecT50) are most sensitive to smoke, while GI datasets (Kvasir, TEMSET-24K) show heterogeneous sensitivity to motion, shot noise, and defocus depending on the model.
RobustEndoCLIP Performance: The proposed method significantly outperformed baselines in both Mean-C and Worst-C settings.
- On CholecT50, it improved Worst-C from ~7% (baselines) to 16.83%.
- On Kvasir, it improved Worst-C from ~6.5-8.2% to 11.42%.
- On TEMSET-24K, it raised Worst-C from ~0.4-1.5% to 19.98%.
VeRA vs. LoRA: Under a fixed 16% label budget, VeRA-based adaptation yielded superior robustness (Mean-C) compared to LoRA, despite having significantly fewer parameters (1024 vs. ~720k). VeRA achieved an average Mean-C of 29.13% across datasets, compared to 16.92% for LoRA.
Label Budget Scaling: Robustness generally improved with increased label budgets (0% to 16%), though returns were dataset-dependent.

Significance and Claims

The paper claims that Endo-C6 provides a standardized, reproducible protocol for reporting robustness in endoscopic video understanding, addressing the gap where rare failures in high-stakes clinical settings are often overlooked. The findings suggest that while off-the-shelf TVLMs are brittle under realistic surgical artifacts, lightweight few-shot adaptation can substantially improve both average and worst-case performance. The authors posit that Endo-C6 will serve as a practical reporting template to promote more reliable clinical vision-language systems, emphasizing that robustness must be evaluated not just on mean performance but on tail failures (Worst-C). The study remains modest, noting that the benchmark currently focuses on prompt-based classification and that extending it to other tasks (e.g., detection, segmentation) or generative models is a direction for future work.