💻 computer science

Beyond Uncertainty: Generalizable Failure Monitoring for Surgical Segmentation under Acquisition Degradation

This paper introduces TCSR-Monitor, a post-hoc framework that enhances surgical segmentation failure detection under acquisition degradation by combining confidence with observable shape, temporal, and image-quality cues, demonstrating superior generalization over confidence-only baselines while highlighting persistent challenges with false alarms and zero-shot transfer.

Original authors: Hieu D. Pham, Dang P. M. Cao, Thanh Trung Huynh

Published 2026-08-18

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Hieu D. Pham, Dang P. M. Cao, Thanh Trung Huynh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the operating room, a surgeon relies on a steady hand and sharp eyes, but increasingly, they also rely on software that can see inside the body. Modern computer systems can watch a live video feed from a camera inside a patient and automatically draw outlines around surgical tools and organs, helping the surgeon track their position or assess their skill. These systems are trained on thousands of hours of clear, high-quality video. However, real surgery is messy. Smoke from cauterizing tissue, sudden shifts in lighting, blurry motion, or even the compression of the video signal itself can degrade the image. When this happens, the computer might still confidently draw a perfect-looking outline, even though the image it is looking at is too distorted to be accurate. This is a silent failure: the system is wrong, but it does not know it, and it offers no warning to the human watching the screen.

A team of researchers at VinUniversity in Hanoi has developed a new way to catch these silent mistakes. They created a system called TCSR-Monitor, which acts as an independent watchdog for surgical video. Instead of trying to fix the computer vision model or retrain it, this new system simply watches what the model produces and checks the result against a set of common-sense rules. It looks at the shape of the drawn outline, how smoothly it moves from one video frame to the next, and the quality of the image itself. By combining these observations with the model's own confidence score, the monitor can spot when a tool is being misidentified, even if the original computer program insists it is doing a perfect job. The researchers tested this on a large dataset of surgical videos that had been artificially degraded to mimic real-world problems like blur and noise. They found that their new system could detect these failures much better than relying on the computer's confidence alone, proving that looking at the shape and movement of the result is just as important as listening to the computer's internal voice.

The core problem the researchers tackled is that deep learning models, which power these surgical tools, are often overconfident. When an image is blurry or dark, a standard model might still output a sharp, clear outline of a surgical instrument, convinced it is correct. In the real world, this is dangerous because a surgeon might trust a wrong outline and make a mistake. Previous methods tried to solve this by measuring uncertainty, essentially asking the computer, "How sure are you?" If the computer was unsure, it would raise an alarm. But the researchers found that under certain types of image degradation, the computer remains stubbornly sure even when it is completely wrong. The new approach, TCSR-Monitor, accepts that the computer might be confident but wrong, so it stops listening only to the computer's confidence and starts looking at the evidence in the video itself.

To build this watchdog, the researchers designed a system that does not need to know how the surgical computer works inside. It is a "post-hoc" monitor, meaning it sits on top of the existing model without changing its weights or requiring access to its internal code. This is crucial for hospitals, where the software running the surgery might be a "black box" that cannot be modified or retrained. The system watches the video frame by frame. For every image, it extracts twenty-two different clues. Some of these clues come from the confidence of the model, such as how much the model wavers between different possibilities. Others come from the shape of the outline: a surgical tool should look like a solid object, not a scattered collection of pixels. The system also checks how the tool moves; in a real video, a tool moves smoothly, so if the outline jumps erratically from one frame to the next, it is a sign of trouble. Finally, it measures the quality of the image itself, checking for blur, brightness, and contrast.

The researchers tested their system using a dataset called EndoVis 2017, which contains recordings of robotic surgery. They took these videos and deliberately corrupted them with six different types of problems, including Gaussian blur, motion blur, and JPEG compression, at five different levels of severity. They then asked their monitor to predict which frames contained a failure, defined as a moment when the computer's outline did not match the actual tool well enough. To ensure the system was truly learning to spot failures and not just memorizing the specific types of bad images it was trained on, they used a rigorous testing method. They trained the monitor on five types of image corruption and then tested it on the sixth type it had never seen before. This is similar to teaching a student to recognize a broken car by looking at a flat tire, a cracked windshield, and a dented door, and then asking them to identify a car with a missing wheel. The monitor succeeded, showing that it had learned the general concept of a failure rather than just memorizing specific defects.

The results showed that relying on the computer's confidence alone was not enough. While standard methods based on uncertainty could spot some errors, they missed many of the confident failures that are most dangerous. The new TCSR-Monitor, by combining confidence with shape, movement, and image quality, significantly outperformed these older methods. In tests where the system had to identify failures in images it had never seen during training, it correctly identified the problems far more often than the uncertainty-based baselines. The researchers also performed a specific check to make sure the system wasn't just flagging any blurry image as a failure. They showed that the monitor could distinguish between a blurry image where the computer was still correct and a blurry image where the computer had made a mistake. This is a critical distinction; a system that just screams "the image is bad" is not useful if it cannot tell when the computer is actually wrong.

However, the researchers were careful to note that the system is not yet perfect for immediate clinical use. While it successfully identified failures, it also raised false alarms. In some cases, when the image was moderately corrupted, the system flagged nearly forty percent of the correctly drawn outlines as failures. This means that if a surgeon were to rely on this alarm stream today, they would be interrupted by warnings even when the computer was doing its job correctly. To address this, the team used a statistical technique called Mondrian conformal calibration. This method adjusts the sensitivity of the alarm for different levels of image degradation, ensuring that the rate of missed failures stays consistent across all types of problems. While this balanced the safety, it did not eliminate the false alarms entirely, highlighting a trade-off between catching every mistake and avoiding constant interruptions.

The study also explored whether this monitoring system could work with different types of surgical computers. They tested it on a newer, more powerful model called SAM2, which is a general-purpose vision model. Interestingly, for this specific model, the simple measure of uncertainty was actually better than the complex new monitor. This suggests that the new system is not a magic bullet that works in every situation, but rather a specialized tool that shines when the underlying computer model is stubbornly confident despite being wrong. The researchers found that the features their system learned—looking at shape and movement—could be transferred to the new model, but they did not outperform the simpler method in that specific case. This nuance is important: the value of the new approach lies in its ability to generalize to unseen problems and its robustness when the computer is confidently incorrect, not in its ability to replace all other methods.

Ultimately, this work provides a credible framework for making surgical AI safer. It demonstrates that by adding a layer of external observation that checks the shape, movement, and quality of the output, we can catch errors that the computer itself fails to see. The researchers have made their code and trained configurations available to the public, allowing other scientists to build upon this foundation. While the system still needs refinement to reduce false alarms before it can be used in a live operating room, it offers a clear path forward. It shifts the focus from hoping the computer knows when it is wrong to building a system that can tell when the computer is wrong, even when the computer insists it is right. This is a vital step toward ensuring that the digital assistants in the operating room remain reliable partners in the high-stakes environment of surgery.

Technical Summary: Beyond Uncertainty: Generalizable Failure Monitoring for Surgical Segmentation under Acquisition Degradation

1. Problem Statement

Deep learning-based surgical segmentation networks are increasingly deployed to parse endoscopic video into instrument and anatomy masks. However, these models are prone to "silent failures" under acquisition degradation (e.g., smoke, defocus, motion blur, compression). In such scenarios, a network may produce a segmentation mask with low Intersection-over-Union (IoU) while simultaneously reporting high confidence. Existing deployment-time monitoring methods rely primarily on uncertainty estimates (e.g., predictive entropy, max-softmax confidence). These signals fail precisely when the network is confidently wrong, as acquisition shifts can push inputs out of the training distribution while leaving output logits sharp. Furthermore, many existing monitors cannot distinguish between detecting a corrupted image and predicting an actual segmentation failure, nor do they guarantee generalization to unseen types of degradation.

2. Methodology: TCSR-Monitor

The authors propose TCSR-Monitor (Temporal Conformal Surgical Risk Monitor), a post-hoc failure-monitoring framework designed to wrap a frozen segmentation model without requiring access to its internal weights or gradients.

Input & Features: The monitor operates without ground truth at deployment. For each video frame, it extracts 22 scalar features from the raw image, the segmentation probability map, the predicted mask, and the previous frame. These features are categorized into four groups:
- Confidence (7 features): Statistics of the probability map (e.g., mean foreground probability, entropy, margin).
- Shape (6 features): Geometric properties of the predicted mask (e.g., area fraction, number of connected components, eccentricity, solidity).
- Temporal (5 features): Consistency across frames (e.g., IoU with the previous mask, centroid displacement, rolling IoU statistics).
- Image Quality (4 features): Acquisition conditions (e.g., brightness, blur score via Laplacian variance, contrast, specular highlight ratio).
Model: A lightweight XGBoost classifier ( $g_\phi$ ) maps these 22 features to a failure risk score in $[0, 1]$ .
Calibration & Safety: The system employs Mondrian conformal calibration to set alarm thresholds. Unlike global calibration, which averages across all conditions, Mondrian calibration partitions frames into groups based on corruption severity (or blur tertiles for clean frames). This ensures that the miss-rate (the fraction of actual failures not flagged) is controlled uniformly across different degradation severities, preventing easy conditions from dominating the safety metrics.
Objective: The primary safety objective is to keep the miss-rate below a target $\alpha$ (e.g., 10%) while minimizing false alarms.

3. Key Contributions

Framework: A deployment-time failure-monitoring framework that integrates complementary observable cues (shape, temporal, quality) beyond standard uncertainty scores.
Validation Protocol: A rigorous evaluation suite including:
- Leave-One-Corruption-Out (LOCO): Tests generalization to unseen corruption types.
- Circularity Control: Evaluates whether the monitor predicts segmentation failure specifically, rather than just detecting image corruption.
- Severity-Aware Calibration: Uses Mondrian calibration to balance safety across degradation levels.
Empirical Evidence: Demonstrates that uncertainty-based confidence alone is insufficient under acquisition degradation, whereas the combination of observable cues improves robustness to unseen corruptions.
Feature Portability: Shows that the learned feature representation can transfer to foundation models (SAM2), though with limitations.

4. Experimental Results

Experiments were conducted on the EndoVis 2017 dataset using a frozen ResNet34-U-Net segmenter and six corruption types at five severity levels.

Generalization (LOCO): TCSR-Monitor significantly outperforms confidence-based baselines (Entropy, Max-Softmax) in LOCO settings. For instance, in the LOCO overall test, TCSR-Monitor achieved an AUROC of 0.814 compared to 0.481 for Entropy. It consistently won in 26 out of 30 (corruption type, severity) cells against Entropy.
Circularity Control: When evaluated strictly on corrupted frames to distinguish correct masks (IoU $\ge$ 0.75) from failed ones (IoU < 0.75), TCSR-Monitor achieved an AUROC of 0.946, whereas Entropy dropped to 0.594. This confirms the monitor detects segmentation failure, not just image degradation.
Calibration & False Alarms: While Mondrian calibration successfully equalized miss-rates across severity levels (targeting ~10%), it revealed a significant cost: at a single global threshold, the system produced false alarms on up to 40.4% of correctly segmented frames at moderate corruption (severity 3).
SAM2 Transfer: When applied to SAM2-Tiny (zero-shot with oracle prompts), the transferred TCSR-Monitor achieved an AUROC of 0.930 (at $\tau=0.75$ ). However, simple Entropy outperformed it (0.988), indicating that for models where failures manifest as high uncertainty, uncertainty-based baselines remain superior.
Feature Importance: Analysis showed that while confidence features contributed the largest share (57.9%), image quality (20.6%) and shape (16.5%) features provided critical complementary signals, with two of the top-5 individual features being image-quality metrics.

5. Significance and Claims

The paper positions TCSR-Monitor not as a universal solution that replaces uncertainty, but as a generalization-aware framework that addresses specific failure modes where uncertainty fails.

Core Claim: Reliable monitoring under acquisition degradation benefits from complementary observable signals beyond confidence alone. The combination of shape, temporal, and quality cues allows the system to detect "confident failures" that uncertainty metrics miss.
Modest Limitations: The authors explicitly state that the current system is not yet ready for direct clinical deployment as a standalone alarm stream due to high false-alarm rates (up to 40% on correct frames at moderate corruption) and the lack of certified temporal coverage.
Scope: The evidence supports TCSR-Monitor as a credible research baseline and a validation protocol for monitoring surgical segmentation under acquisition degradation. It does not claim universal transfer across all hospitals, procedures, or backbone families, nor does it replace the need for fine-tuned surgical backbones. The work highlights that while uncertainty is a strong baseline, it is insufficient in isolation for robust surgical safety monitoring.