🤖 AI

Budget-Aware Uncertainty for Radiotherapy Segmentation QA Using nnU-Net

This paper proposes a budget-aware uncertainty-driven quality assurance framework for radiotherapy segmentation based on nnU-Net, demonstrating that combining calibrated checkpoint ensembles with temperature scaling effectively generates voxel-wise uncertainty maps to guide targeted manual review of Clinical Target Volume delineations.

Original authors: Ricardo Coimbra Brioso, Lorenzo Mondo, Damiano Dei, Nicola Lambri, Pietro Mancosu, Marta Scorsetti, Daniele Loiacono

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ricardo Coimbra Brioso, Lorenzo Mondo, Damiano Dei, Nicola Lambri, Pietro Mancosu, Marta Scorsetti, Daniele Loiacono

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef preparing a massive, complex banquet for cancer patients. Your job is to carve out the exact pieces of meat (the tumors) that need to be cooked with radiation, while carefully avoiding the delicate vegetables (healthy organs) that must remain untouched. This process is called radiotherapy planning.

For decades, this "carving" has been done by hand by highly skilled doctors. It's slow, tiring, and prone to human error, especially when the "meat" is scattered all over the body (like in a treatment called TMLI, which targets the whole marrow and lymph nodes).

Recently, we've introduced a robot assistant (an AI called nnU-Net) that can do this carving automatically. It's fast and usually very good. But here's the catch: Robots can be confidently wrong. Sometimes the robot draws a perfect line around a healthy organ, thinking it's a tumor, or misses a tiny piece of tumor entirely. If a doctor has to check the robot's work on a 3D scan that is hundreds of slices thick, it would take them all day. They don't have the time to look at everything.

The Problem: How do we tell the doctor, "Hey, don't waste time checking the easy parts; look here instead"?

The Solution: This paper introduces a "Budget-Aware Uncertainty" system. Think of it as a smart spotlight that only shines on the parts of the image where the robot is confused.

The Ingredients of the Recipe

The researchers tested different ways to make this "confusion spotlight" brighter and more accurate. They combined three main techniques:

The Ensemble (The Council of Experts):
Instead of asking one robot to do the job, they asked a "council" of five robots.
- Deep Ensemble: Five completely different robots trained separately. (Expensive and slow to set up, like hiring five different chefs).
- Checkpoint Ensemble: One robot, but we ask it for its opinion at five different moments during its training (like asking a student for their answer at the start, middle, and end of a semester). This is cheaper and faster.
Test-Time Augmentation (The "What-If" Game):
Before the robot gives its final answer, the researchers slightly tilt, rotate, or brighten the image a few times and ask the robot to carve again. If the robot gives a different answer every time the image is tilted, it means it's unsure. This uncertainty is a great signal for the spotlight.
Temperature Scaling (The "Confidence Thermostat"):
AI robots often think they are 100% sure when they are actually wrong. This technique is like a thermostat for confidence. It cools down the robot's overconfidence, making its "I'm sure" signals more honest. If the robot says "I'm 90% sure," the thermostat might adjust that to "I'm actually only 70% sure," which is more realistic.

The Experiment: Who Shines the Brightest?

The researchers tested these methods on 45 patients. They wanted to see which combination created the best "spotlight" to guide the doctor's limited time (their "budget").

The Result: The segmentation (the actual carving) stayed about the same quality across all methods. The magic happened in the uncertainty map.
The Winner: The combination of the Checkpoint Ensemble (one robot, multiple snapshots) + Temperature Scaling (the confidence thermostat) + Test-Time Augmentation (the "what-if" game) was the champion.

Why This Matters: The "Spotlight" Analogy

Imagine the doctor has a budget of 5 minutes to review the robot's work.

Without the spotlight: The doctor has to scan the whole image, hoping to find the mistake. They might miss it.
With the old spotlight: The light shines everywhere, or shines on the wrong places. The doctor wastes time checking areas that are actually fine.
With the new "Budget-Aware" spotlight: The system highlights the top 0.5% to 2% of the image where the robot is most confused.
- The doctor looks only at those glowing red spots.
- Because the system is so good at spotting confusion, those glowing spots almost always contain the actual errors (missed tumors or wrong boundaries).
- The doctor fixes the error in seconds, rather than spending minutes searching.

The Takeaway

This paper proves that we don't need to make the AI perfectly accurate to make it useful. Instead, we need to make the AI honest about its mistakes.

By combining a "council" of robot snapshots with a "confidence thermostat," the researchers created a system that acts like a smart GPS for doctors. It doesn't drive the car for them; it just says, "The road ahead is foggy, pay attention here," allowing the doctor to focus their limited energy exactly where it's needed most.

This is a huge step toward making AI safe and practical for real-world cancer treatment, ensuring that no tumor is missed and no healthy tissue is harmed, all while saving the doctors' time.

1. Problem Statement

Accurate delineation of the Clinical Target Volume (CTV) is critical for radiotherapy planning but remains time-consuming and prone to human error, particularly for complex treatments like Total Marrow and Lymph Node Irradiation (TMLI). While deep learning (specifically nnU-Net) can automate segmentation, clinical adoption is hindered by the lack of reliable mechanisms to identify where the model might fail.

The Challenge: Full-body CT scans contain hundreds of slices, making exhaustive manual verification unrealistic within clinical time constraints.
The Gap: Existing uncertainty quantification (UQ) methods often focus on global calibration metrics (e.g., NLL, ECE) without assessing whether the uncertainty maps are spatially actionable under realistic, budget-limited review scenarios. Furthermore, computationally expensive methods (like Deep Ensembles) are often compared against simpler alternatives without a systematic evaluation of their cost-benefit ratio for Quality Assurance (QA).

2. Methodology

The authors propose a budget-aware uncertainty-driven QA framework built on the nnU-Net architecture. The goal is to generate voxel-wise uncertainty maps (based on predictive entropy) that guide clinicians to prioritize manual reviews of the most likely erroneous regions.

A. Uncertainty Estimation Strategies

The study compares and combines four primary strategies to generate probability distributions and compute entropy ( $H(p_i)$ ):

Baseline (BASE): A single nnU-Net model.
Deep Ensembles (DE): Aggregating predictions from 5 independently trained models (via 5-fold cross-validation).
Checkpoint Ensembles (CE): Aggregating predictions from 5 checkpoints saved during a single training run (using a cyclical learning rate strategy).
Test-Time Augmentation (TTA): Applying random small-magnitude transformations (rotations, translations, intensity scaling) to the input and averaging predictions.
Temperature Scaling (TS): A post-hoc calibration method applied to the softmax outputs to adjust prediction confidence (sharpness) without retraining.

B. Experimental Setup

Dataset: 45 patients with hematological malignancies undergoing TMLI. The target is the union of lymph nodes (CTV LN) and spleen (CTV Spleen).
Protocol: Repeated hold-out strategy (4 splits) with 5-fold cross-validation for training.
Configurations: 12 total configurations testing combinations of the above methods (e.g., BASE+TS, DE+TTA, CE+TS+TTA).

C. Evaluation Metrics

To assess clinical utility, the authors moved beyond standard accuracy metrics:

Calibration: Expected Calibration Error (ECE) and Brier Score (BS), computed specifically within a Region of Interest (ROI) (boundary zones) to avoid dominance by background voxels.
Segmentation Accuracy: Dice Similarity Coefficient (DSC).
Uncertainty-Error Alignment:
- UEO (Uncertainty-Error Overlap): Dice overlap between the set of "uncertain" voxels and actual errors (FP/FN).
- Budget-Aware AUC: Metrics calculated over the top 0–5% most uncertain voxels. This simulates a clinical scenario where only a small fraction of the image can be reviewed.
- Coverage Separation: Measures how well the method flags False Positives vs. True Positives, and False Negatives vs. True Negatives.

3. Key Contributions

Budget-Aware Framework: Shifts the evaluation of UQ from global statistical scores to spatially actionable metrics under strict review budgets (0–5% of voxels).
Systematic Comparison: Provides a head-to-head comparison of Deep Ensembles, Checkpoint Ensembles, TTA, and Temperature Scaling within the same nnU-Net pipeline for a complex radiotherapy task.
Efficiency vs. Performance Trade-off: Demonstrates that Checkpoint Ensembles (CE) combined with Temperature Scaling (TS) and TTA offer a superior balance of computational efficiency and uncertainty quality compared to expensive Deep Ensembles.
Calibration Impact: Shows that post-hoc calibration (TS) significantly improves the reliability of uncertainty estimates without altering the hard segmentation output.

4. Key Results

Segmentation Accuracy (DSC): Remained stable across all configurations (approx. 0.836–0.843). Adding UQ methods did not degrade the primary segmentation performance.
Calibration: Temperature Scaling (TS) was the most effective single intervention, drastically reducing ECE (e.g., from 0.062 to 0.031 for the baseline) and improving the Brier Score.
Uncertainty-Error Alignment:
- CE+TS+TTA achieved the best UEO AUC (0.173 vs. 0.154 for baseline), indicating the strongest alignment between uncertainty maps and actual errors.
- Deep Ensembles (DE) showed modest gains in DSC but were less effective at highlighting specific errors under tight budgets compared to CE.
- CE-based methods were particularly good at flagging False Positives, while DE+TS+TTA was slightly better at separating False Negatives.
Visual Qualitative Analysis: At low budgets (0.5%), CE+TS+TTA produced spatially coherent uncertainty maps that highlighted clinically relevant missed regions (e.g., axillary lymph nodes), whereas DE tended to highlight boundary noise or spread uncertainty too broadly.

5. Significance and Conclusion

This work establishes a practical pathway for deploying deep learning in radiotherapy by addressing the "trust gap."

Clinical Workflow: The study supports a hierarchical QA workflow where clinicians first review the top 0.5–1% most uncertain voxels. If necessary, they expand the review budget incrementally.
Deployment Strategy: Checkpoint Ensembling is identified as the most promising strategy for clinical deployment because it reuses checkpoints from a single training run, avoiding the high computational cost of training multiple independent models (Deep Ensembles) while delivering superior uncertainty localization.
Future Directions: The authors note the need for multi-center validation and prospective clinical trials to measure the real-world impact on correction time and inter-observer variability.

In summary, the paper demonstrates that integrating calibration (TS) with efficient ensembling (CE) creates a robust, budget-aware QA system that effectively guides manual review in complex radiotherapy segmentation tasks.