💻 computer science

Calibration and Threshold Selection of Machine Learning Classifiers for Breast Tumour Diagnosis: A Decision-Support Study for Al-Driven Clinical Screening

This study evaluates the performance, calibration, and threshold selection of four machine learning classifiers on the Wisconsin Diagnostic Breast Cancer dataset, revealing that while models achieve high accuracy and AUC, their probability calibration varies by algorithm and that achieving target sensitivity thresholds remains uncertain due to limited sample sizes, highlighting critical considerations for AI-driven clinical decision support.

Original authors: KARTIK PHOPASE

Published 2026-10-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: KARTIK PHOPASE

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of medical screening, the goal is often described simply as getting the right answer: is a lump in the breast a harmless growth or a dangerous cancer? For decades, scientists have built computer programs to help doctors make this distinction by analyzing images of cells taken from a biopsy. These programs, known as machine learning classifiers, look at dozens of tiny measurements—such as the size of a cell's nucleus or the texture of its surface—to make a prediction. However, for a computer to be a truly useful partner in a doctor's office, it must do more than just be right most of the time. It must also be honest about its own certainty. When a computer says there is a ninety percent chance of cancer, that number needs to reflect reality, not just a guess. Furthermore, the point at which the computer decides to sound an alarm must be carefully chosen. In medicine, missing a cancer is far more costly than raising a false alarm, so the system needs a rule that prioritizes catching every possible case, even if it means checking a few healthy people more closely.

A recent study by Kartik Phopase at Sanjivani University in India explores exactly these two critical questions using a well-known collection of medical data called the Wisconsin Diagnostic Breast Cancer dataset. This dataset contains information on 569 patients, with measurements taken from digitized images of their cells. The researchers wanted to see if different types of computer learning models produced trustworthy probability numbers and if the rules they used to decide when to flag a patient as "at risk" would hold up when tested on new, unseen people. They compared four distinct types of learning algorithms, ranging from simple statistical methods to complex systems that build hundreds of decision trees, testing them under two conditions: one using only the ten average measurements from the cells, and another using all thirty available measurements.

The study began by training these computer models on a large portion of the data and then testing them on a separate group of patients they had never seen before. The results showed that all the models were incredibly skilled at distinguishing between cancerous and non-cancerous cells. In fact, they were so good that the differences between them were too small to matter statistically. Whether the computer used just the ten average measurements or all thirty available details, the top performers were nearly identical in their ability to separate the two groups. This finding suggests that, for this specific type of data, the choice of which complex algorithm to use is less important than ensuring the system is set up correctly. The researchers found that adding more detailed measurements did improve the scores slightly, but the improvement was so small that it did not change the overall conclusion: the technology is already near its limit for this specific task.

The more surprising and practical discovery concerned how well these computers communicated their confidence. A common assumption in the field is that you can always take a computer's raw output and adjust it with a mathematical formula to make the probabilities more accurate. The researchers tested this by taking the predictions from their models and applying two different adjustment methods. They found that for some models, specifically the simpler statistical ones, the raw numbers were already trustworthy. Trying to adjust them actually made the predictions worse, introducing more errors. For the more complex models that build many decision trees, a different type of adjustment helped slightly, but the improvement was modest. The key takeaway is that there is no universal fix. A system that is already well-calibrated does not need to be "re-tuned," and applying a standard adjustment to every system can sometimes do more harm than good.

Perhaps the most important lesson from the study relates to the decision threshold, the specific point where the computer switches from saying "likely benign" to "likely malignant." The researchers tried to set a rule that would catch at least ninety-five percent of all cancer cases, a very high safety standard. They tuned this rule using the training data, where the computer had already seen the answers. When they took this same rule and applied it to the new, unseen group of patients, it failed to meet the target. Instead of catching ninety-five percent of the cancers, it caught about ninety-three percent. This gap occurred because the small number of cancer cases in the test group meant that a single missed case changed the percentage significantly. The study demonstrates that a rule which looks perfect on training data is not guaranteed to work the same way in the real world. The uncertainty inherent in small sample sizes means that doctors cannot rely on a single number to guarantee safety.

Ultimately, this research serves as a cautionary guide for building artificial intelligence tools for healthcare. It shows that while computers can be excellent at spotting patterns, the journey from a high score to a reliable medical tool is filled with subtle traps. The study confirms that for this specific dataset, the models are already highly accurate, but their true value depends on how their probabilities are reported and how their alarm thresholds are set. The author suggests that future systems should not just report a single risk number, but should also show how uncertain that number might be. They argue that setting a safety target based on a small group of training data is risky, and that any rule used to trigger a medical alert should be tested on independent data with a clear understanding that it might not perform exactly as expected. The path forward involves not just finding the smartest algorithm, but rigorously testing how that algorithm behaves when the stakes are real and the data is new.

Technical Summary: Calibration and Threshold Selection of Machine Learning Classifiers for Breast Tumour Diagnosis

Problem Statement
While machine learning classifiers for breast tumour diagnosis (specifically on the Wisconsin Diagnostic Breast Cancer or WDBC dataset) frequently achieve high accuracy, accuracy alone is insufficient for clinical decision support. A system intended to guide triage or diagnosis must provide calibrated probabilities (where the predicted probability reflects the true likelihood of malignancy) and a justified decision threshold that accounts for the asymmetric costs of errors (missing a malignant tumour is costlier than a false alarm). Existing literature often neglects calibration and relies on single-split evaluations that may produce misleading model rankings due to sampling noise. This study addresses the empirical question: on a benchmark where discrimination is near its ceiling, how do standard classifiers differ in probability calibration, and how reliably does a sensitivity-constrained threshold transfer from training data to unseen data?

Methodology
The study utilizes the WDBC dataset (569 samples, 37.3% malignant) distributed with scikit-learn. The experimental protocol strictly separates model comparison from final evaluation to avoid leakage and optimistic bias:

Data Splitting: The data is split once into a stratified training set (455 samples) and a held-out test set (114 samples).
Scenarios: Two feature sets are evaluated:
- Scenario A: 10 mean-value features only.
- Scenario B: All 30 features (mean, standard error, and "worst" values).
Models: Four classifiers are compared: Logistic Regression (L2), RBF-kernel Support Vector Machine (SVM), Random Forest, and Gradient Boosting, alongside a majority-class baseline. No hyperparameter tuning is performed; default settings are used.
Evaluation Protocol:
- Model Comparison: Repeated stratified 5-fold cross-validation (10 repeats, 50 total folds) on the training set.
- Significance Testing: A corrected resampled t-test is used to determine if differences in cross-validated ROC-AUC are statistically significant.
- Final Evaluation: The top-performing model per scenario is refit on the full training set and evaluated once on the held-out test set.
- Metrics: Accuracy, Precision, Recall (Sensitivity), Specificity, F1, ROC-AUC, Brier Score, and Expected Calibration Error (ECE).
- Calibration & Thresholding: Post-hoc calibration (Sigmoid/Platt scaling and Isotonic regression) is applied. A decision threshold is tuned on out-of-fold (OOF) predictions to maximize specificity while maintaining at least 95% sensitivity, then applied to the test set.

Key Contributions

Reproducible Comparison: A leakage-free comparison of four standard models and a baseline under a rigorous protocol involving repeated cross-validation, corrected significance testing, and bootstrap confidence intervals.
Feature Set Analysis: An identical evaluation of a reduced feature set (mean values only) versus the full feature set to assess the marginal utility of additional features.
Empirical Calibration & Threshold Analysis: An investigation into the necessity of post-hoc calibration and the stability of sensitivity-constrained thresholds when transferred to unseen data, explicitly reporting uncertainty.

Results

Discrimination: All models achieved high cross-validated ROC-AUC scores (0.982–0.985 in Scenario A; 0.988–0.994 in Scenario B). Statistical testing revealed no significant differences between the models in either scenario (all $p > 0.14$ ). The choice of the "best" model was effectively arbitrary, with margins of difference as small as 0.0003.
Held-Out Performance: The selected models (Gradient Boosting for A, SVM for B) achieved held-out accuracies of 0.947 and 0.965, respectively, with AUCs of 0.979 and 0.995.
Calibration:
- Logistic Regression and SVM: These models were already well-calibrated. Applying post-hoc Sigmoid calibration worsened their calibration error (ECE and Brier score).
- Tree Ensembles (Random Forest, Gradient Boosting): These models benefited from Isotonic calibration, which reduced their calibration error.
- Overall: Post-hoc calibration is not universally beneficial; it depends on the base model.
Threshold Selection: A threshold tuned on OOF predictions to achieve 95% sensitivity failed to meet this target on the held-out test set. The achieved sensitivity was 0.929 in both scenarios. This demonstrates that with limited sample sizes (42 malignant cases in the test set), threshold selection carries significant uncertainty, and a target met on training data is not guaranteed on new data.
Feature Importance: In Scenario A, "mean area" and "mean concave points" were the most influential features. In Scenario B, feature importance rankings were unstable due to high correlation among features (e.g., radius, perimeter, area).

Significance and Claims
The paper concludes that for AI-driven clinical decision support, discrimination (AUC) is not the sole metric of success.

Model Selection: On high-performance benchmarks like WDBC, model rankings based on single-split accuracy or small AUC differences are unreliable. The evaluation design is more critical than the specific algorithm choice.
Calibration Strategy: Calibration should not be applied by default. Well-calibrated models (like Logistic Regression and SVM with Platt scaling) may be degraded by further calibration, while tree ensembles may require isotonic calibration. Calibration must be verified on held-out data.
Threshold Stability: Setting a sensitivity target on training data does not guarantee that target on unseen data due to sampling noise. Clinical systems should incorporate explicit margins, cost models, or lower confidence bounds on sensitivity rather than relying on point estimates.
Limitations: The study is a methodological benchmark using a single, small dataset with expert-selected features. It does not claim to validate end-to-end clinical screening performance or address external validity across different populations or scanners.

Technical Summary: Calibration and Threshold Selection of Machine Learning Classifiers for Breast Tumour Diagnosis

More like this