⚡ electrical engineering

Calibrated Selective Prediction Using Deep Ensembles for ROI-Based Thyroid Nodule Ultrasound Classification Under Dataset Shift: A Retrospective Evaluation

This study presents a calibrated deep ensemble framework for thyroid nodule ultrasound classification that achieves strong internal performance and effective selective triage but demonstrates limited external generalizability under dataset shift, highlighting the necessity for local recalibration and prospective validation before clinical deployment.

Original authors: Md. Sadibul Hasan Sadib, Md. Mohayminul Mukit, Rahmatul Kabir Rasel Sarker, Tahmid Alam Tamim, Md. Monir Hossain Shimul

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Md. Sadibul Hasan Sadib, Md. Mohayminul Mukit, Rahmatul Kabir Rasel Sarker, Tahmid Alam Tamim, Md. Monir Hossain Shimul

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot team of five detectives, all trained to look at ultrasound pictures of thyroid nodules and decide: "Is this a harmless lump, or should we poke it with a needle (FNA) to check for cancer?"

This paper is about building that robot team, teaching them how to admit when they are confused, and seeing if they can actually help doctors without making dangerous mistakes.

The Detective Team and Their "Confidence Meter"

The researchers built a team of five AI detectives using a smart architecture called ConvNeXt-Tiny. But here's the twist: they didn't just want the team to guess; they wanted them to be honest about how sure they were.

To do this, they gave the team a special "disagreement meter" called Mutual Information (MI). Think of it like this: if all five detectives look at a picture and say, "Yeah, that's definitely a bad guy," the meter stays low. But if Detective A says "Bad guy," Detective B says "Good guy," and the others are scratching their heads, the meter goes high.

When the meter goes high, the system has a strict rule: "Stop! Don't guess. Send this picture to a human doctor." This is called selective prediction. The goal isn't to make a decision for every single picture; it's to only make decisions when the team is super confident and agrees.

The Big Test: Inside the Lab vs. The Real World

The team trained on a massive dataset called TN5000, which had 5,000 images. In this controlled lab environment, the team performed very well:

They got a score of 0.9395 on a scale of 0 to 1 (where 1 is perfect) for telling good from bad.
They were highly calibrated, meaning their predicted probabilities closely matched the actual outcomes, with a very small error rate of 0.88%.
When they used their "disagreement meter" to filter out the tricky cases, they could safely suggest "No Needle" for 7.2% of the cases and "Needle" for 39.9% of the cases, while sending the remaining 52.9% to a human doctor.
Crucially, they caught 99.83% of all the cancer cases in this specific dataset.

However, the paper makes a very important point: This success was only inside the lab. The authors explicitly state that the study's objective is not to replace clinical assessment or to estimate real-world biopsy avoidance, but rather to test if the system can identify cases where automated suggestions are unreliable.

The Reality Check: When the Team Met a New City

To see if the team could work in the real world, the researchers took their frozen, unchangeable robot team (trained only on TN5000) and sent it to a completely different dataset called TN3K. This new dataset had different machines, different doctors, and different types of images.

The result? The robot team stumbled.

Their "smartness" score dropped from 0.9395 down to 0.7870.
Their honesty (calibration) got messy. They started saying "90% chance" when the real chance was much lower, with an error rate jumping to nearly 19%.
The "disagreement meter" broke down. Because the new images looked so different, the team got confused much more often.
When they tried to use the same rules from the lab, 83.7% of the new images had to be sent to a human doctor. The robot team only felt confident enough to make a decision on 16.3% of the cases.
Even worse, the "Needle" suggestions they did make were only right 76.6% of the time, far below the 95% target they had in the lab.

What the Paper Says (and What It Doesn't)

The authors are very careful not to overhype this. They explicitly state that this system is not ready to replace doctors or stop biopsies in real hospitals yet.

What it proved: They proved that a robot team can be trained to be honest about its uncertainty and can safely suggest "No Needle" or "Needle" if the images look exactly like the ones it learned from.
What it ruled out: They ruled out the idea that you can just take a model trained on one hospital's data and drop it onto another hospital's data and expect it to work. The "frozen" policy failed to transport well.
What it suggests: It suggests that using "disagreement" as a safety switch is a good idea, but you need to re-calibrate the robot's confidence meter for every new hospital you use it in.

The Bottom Line

Think of this robot team like a brilliant student who aced every test in their home classroom (TN5000) but got confused when they walked into a different school with different textbooks and teachers (TN3K).

The study shows that the student is smart enough to say, "I don't know this one, let me ask the teacher," which is a great safety feature. But until we teach the student how to handle every new school's style of teaching, we can't let them grade exams on their own. The paper concludes that while the idea of "selective prediction" is promising, we need more local testing and calibration before we can trust it to make real medical decisions.

Technical Summary: Calibrated Selective Prediction Using Deep Ensembles for ROI-Based Thyroid Nodule Ultrasound Classification Under Dataset Shift

Problem Statement
While deep learning models have demonstrated high discrimination in classifying thyroid nodules from ultrasound images, high accuracy alone is insufficient for clinical decision support. In real-world scenarios, models may produce confident but unreliable predictions for ambiguous cases, images from different acquisition protocols, or data distributions differing from the training set (dataset shift). A safe clinical system must not only classify images but also identify cases where automated triage suggestions are unreliable and defer them to human experts. This requires robust probability calibration, uncertainty estimation, and a mechanism for selective prediction (classification with a reject option) that operates conservatively under dataset shift.

Methodology
The study proposes a calibrated, deterministic, five-member deep-ensemble framework for annotation-assisted thyroid nodule region-of-interest (ROI) classification. The methodology involves the following key components:

Dataset and Architecture: The framework utilizes the TN5000 dataset (5,000 images, malignant-enriched) for development and internal cross-validation, and the TN3K dataset (614 images, different class distribution and annotation style) for external evaluation under dataset shift. A ConvNeXt-Tiny backbone augmented with a Squeeze-and-Excitation (SE) attention module was selected as the optimal architecture.
Ensemble and Calibration: Five independently trained ensemble members (using different random seeds) are generated. To ensure reliable probability estimates, member-wise vector scaling is applied post-hoc using a dedicated calibration subset (val_calib) within each cross-validation fold.
Uncertainty Estimation: Mutual Information (MI) is employed as the primary operational measure of ensemble disagreement. MI quantifies the divergence between the entropy of the ensemble mean prediction and the mean entropy of individual members. High MI indicates high epistemic uncertainty (disagreement among models).
Selective Triage Policy: A strict three-tier retrospective policy is applied:
1. Uncertainty Gate: Images with MI exceeding a fold-specific threshold are immediately assigned to radiologist review.
2. Rule-Out/Rule-In Gates: Among low-MI images, calibrated malignancy probabilities are evaluated against fold-specific thresholds.
  - Probabilities below a conservative rule-out threshold ( $\tau_{low}$ ) receive an image-based "No-FNA" (Fine-Needle Aspiration) suggestion.
  - Probabilities above a conservative rule-in threshold ( $\tau_{high}$ ) receive an image-based "FNA recommendation."
  - Probabilities falling in the intermediate "gray zone" are assigned to radiologist review.
Threshold Selection: Thresholds are derived exclusively from the validation subset (val_es) of each fold to meet strict safety criteria: a No-FNA Negative Predictive Value (NPV) of $\ge$ 98% and an FNA-recommendation Positive Predictive Value (PPV) of $\ge$ 95%.
External Evaluation: A frozen model and policy derived from TN5000 are applied unchanged to TN3K to assess the transportability of calibration, uncertainty behavior, and decision thresholds under dataset shift.

Key Contributions

Framework Development: The authors developed a calibrated deterministic deep-ensemble framework using ConvNeXt-Tiny with SE attention for ROI-based thyroid nodule classification.
Uncertainty-Driven Deferral: The study evaluates Mutual Information as a specific ensemble-disagreement score to identify images unsuitable for automated triage, distinguishing this from training-time uncertainty weighting.
Conservative Triage Policy: A three-tier policy is proposed that combines uncertainty-based deferral with probability-based rule-out and rule-in pathways, ensuring automated suggestions are issued only for low-disagreement cases with extreme probabilities.
External Transportability Analysis: The study rigorously evaluates a frozen TN5000-derived policy on an independent TN3K cohort, explicitly analyzing how discrimination, calibration, and threshold validity degrade under dataset shift.

Results

Internal Performance (TN5000): The calibrated ensemble achieved a pooled out-of-fold AUC-ROC of 0.9395 and an Average Precision (AP) of 0.9715. Calibration was excellent, with an Expected Calibration Error (ECE) of 0.0088 and a Brier score of 0.0813.
Internal Selective Prediction: At a 50% nominal MI retention rate, the system assigned 7.2% of cases to "No-FNA," 39.9% to "FNA recommendation," and 52.9% to radiologist review. This achieved a 98.3% No-FNA NPV and 99.83% malignancy capture.
External Performance (TN3K): Under dataset shift, discrimination decreased (AUC-ROC 0.7870, AP 0.7254), and calibration deteriorated significantly (ECE 0.1899, Brier score 0.2281).
External Triage Outcomes: Applying the frozen TN5000 policy to TN3K resulted in 83.7% of cases being assigned to radiologist review, 15.3% receiving an FNA recommendation, and only 1.0% receiving a No-FNA suggestion. While no malignant images were incorrectly assigned to the No-FNA pathway (100% observed NPV), this was due to the high deferral rate rather than robust rule-out capability. The FNA-recommendation PPV dropped to 76.6%, failing to meet the 95% target established during internal threshold selection.

Significance and Claims
The paper claims that the proposed calibrated deep-ensemble framework successfully demonstrates strong internal discrimination and calibration, supporting a conservative selective-prediction framework where automated suggestions are withheld for high-disagreement cases. However, the authors explicitly state that the external evaluation on TN3K reveals reduced discrimination, degraded calibration, and limited transportability of the operating thresholds under dataset shift.

Consequently, the authors conclude that these findings do not support direct clinical deployment or claims of actual biopsy reduction without local calibration, threshold validation, and prospective clinical evaluation. Instead, the study positions selective prediction as a cautious mechanism for identifying images that are unsuitable for autonomous triage, emphasizing that automated image-based suggestions should only be issued for low-disagreement cases while potentially unreliable predictions are referred to expert review. The work highlights the necessity of local validation before applying such models to new clinical environments.

The Detective Team and Their "Confidence Meter"

The Big Test: Inside the Lab vs. The Real World

The Reality Check: When the Team Met a New City

What the Paper Says (and What It Doesn't)

The Bottom Line

More like this