Class imbalance correction in artificial intelligence models leads to miscalibrated clinical predictions: a real-world evaluation

This study demonstrates that while class imbalance correction techniques may improve certain metrics like recall, they severely compromise the calibration of AI models for surgical risk prediction, leading to significant risk overestimation and reduced clinical net benefit compared to models trained on natural data distributions.

The Gist

The Big Idea: Why "Fixing" the Numbers Can Break the Medicine

Imagine you are a doctor trying to predict who might get sick after surgery. You have a massive list of 1.8 million patients. But here's the catch: almost everyone is fine. Only a tiny handful (less than 2%) actually die or have serious complications.

In the world of Artificial Intelligence (AI), this is called Class Imbalance. It's like trying to teach a dog to bark only when it sees a lion, but you only show it 10 lions and 10,000 cats. The dog might just decide, "I'll never bark," because it's easier to be right 99% of the time by just saying "Cat."

To fix this, data scientists often use "correction strategies." They try to balance the scales by either:

1. Copying the rare cases (the lions) until there are more of them.
2. Deleting the common cases (the cats) until there are fewer of them.
3. Inventing fake lions to make the numbers look even.

This paper asks a scary question: Does fixing the numbers actually help the doctor, or does it break the AI's ability to tell the truth?

The Experiment: The "Natural" vs. The "Balanced"

The researchers built two types of AI models to predict surgery risks:

1. The Natural Model: This AI was fed the data exactly as it happened in the real world (99% safe, 1% risky). It learned the true rarity of the danger.
2. The "Balanced" Models: These AIs were fed "corrected" data where the risks were artificially made to look much more common (like a 50/50 split).

They tested these models to see who was better at predicting the future.

The Results: The "Balanced" Models Lied

Here is what happened, broken down into simple metaphors:

1. The "Score" Trap (The Exam Analogy)

If you look at standard test scores (like Accuracy or F1-score), the "Balanced" models looked amazing. They seemed to catch more of the sick patients.

• The Metaphor: Imagine a weatherman who predicts "Rain" every single day. On a rainy day, he is right! But on a sunny day, he is wrong. If you only look at how many times he was right on rainy days, he looks like a genius. But he is useless for planning a picnic.
• The Reality: The "Balanced" models were good at spotting some risks, but they did it by screaming "DANGER!" at almost everyone. They traded precision for panic.

2. The Broken Compass (Calibration)

This is the most important part. In medicine, doctors don't just need to know if something is risky; they need to know how risky (e.g., "Is there a 1% chance or a 50% chance?"). This is called Calibration.

• The Metaphor: Imagine a speedometer.
  • The Natural Model is a speedometer that says "60 mph" when you are actually going 60 mph. It's accurate.
  • The Balanced Models are speedometers that say "60 mph" when you are actually going 10 mph. They are "off the charts."
• The Result: The "Balanced" models were miscalibrated. They massively overestimated the risk. They told doctors that a routine surgery had a 50% chance of disaster when it was actually 1%.

3. The Panic Button (Clinical Impact)

The researchers ran a simulation: "If we use these models to decide who needs extra care in the ICU..."

• The Natural Model: Correctly identified that about 16% of surgeries were high-risk.
• The Balanced Models: Screamed that 75% to 90% of surgeries were high-risk.
• The Consequence: If you use the "Balanced" models, you would send 6 out of 7 healthy patients to the Intensive Care Unit (ICU) just in case. This wastes money, clogs up hospitals, and causes unnecessary anxiety for patients. It's like calling the fire department for every time you burn a piece of toast.

The Conclusion: Don't Fix What Isn't Broken

The paper concludes that in medical AI, truth is more important than "balance."

• The Old Way: "Let's make the data look balanced so the math works better."
• The New Finding: "If you force the data to look balanced, you break the AI's ability to tell you the real probability of danger."

The Takeaway:
When an AI is used to make life-or-death decisions, it should be trained on the real world, even if the real world is messy and unbalanced. A model that says "This is rare, but here is the exact risk" is far more useful than a model that says "Everything is dangerous!"

The authors warn that using these "corrected" models could lead to harm because doctors might make decisions based on fake, inflated risks. The best AI for doctors is the one that tells the truth, not the one that tries to be a hero by finding every single needle in the haystack, even if it means pointing at the hay too.

Technical Summary

1. Problem Statement

In clinical machine learning (ML), predictive models often face class imbalance, where the event of interest (e.g., postoperative mortality or complications) occurs much less frequently than the negative class. To address this, developers commonly apply class imbalance mitigation strategies (e.g., oversampling, undersampling, cost-sensitive learning) to force the model to learn from a balanced dataset.

However, there is a critical gap in understanding how these preprocessing steps affect model calibration (the agreement between predicted probabilities and actual observed frequencies) in real-world clinical settings. While these methods often improve threshold-dependent metrics (like recall or F1-score), they may distort the probability outputs, leading to systematic over-prediction of risk. This study investigates whether these common "fixes" actually harm clinical decision-making by miscalibrating models.

2. Methodology

Data Source & Cohort:

• Dataset: A national, population-based retrospective cohort of >1.8 million patients undergoing surgery in public hospitals in Aotearoa New Zealand (2010–2024).
• Outcomes: Two rare events were predicted:
  1. 90-day mortality (Incidence: ~1.3–1.5%).
  2. Any postoperative complication (Incidence: ~9.4–11.0%).
• Split: Temporal split with 2010–2022 as the training/validation set (87%) and 2023–2024 as the independent test set (13%).

Model Architecture:

• Algorithm: Histogram-boosted Gradient (HGB) classifier (similar to the architecture used in the widely adopted NSQIP Risk Calculator).
• Features: 12 distinct clinical features extracted per patient.

Experimental Design:
The authors compared a "Natural" model (trained on the raw, imbalanced data) against models trained with four common class imbalance mitigation strategies:

1. Random Oversampling (ROS): Duplicating minority class instances.
2. Synthetic Minority Oversampling Technique (SMOTE): Generating synthetic minority instances.
3. Random Under-Sampling (RUS): Removing majority class instances.
4. Cost-Sensitive Learning (CSL): Assigning higher penalties to minority misclassifications.

Evaluation Metrics:

• Discrimination: Area Under the Receiver Operating Characteristic (AUROC) and Area Under the Precision-Recall Curve (AUPRC).
• Calibration: Log Loss (primary metric for calibration sensitivity), Calibration Plots (Reliability Diagrams), and predicted probability distributions.
• Classification Metrics: Accuracy, Precision, Recall, and F1-score (at a fixed threshold of 0.5).
• Clinical Utility: Decision Curve Analysis (DCA) to measure net benefit across a range of probability thresholds, and a simulated clinical scenario assessing the proportion of patients classified as "high-risk."

3. Key Contributions

• Real-World Validation: Unlike previous studies using synthetic data, this study evaluates class imbalance strategies on a massive, real-world national surgical dataset (>1.8M patients).
• Calibration vs. Discrimination: It explicitly decouples discrimination (ranking ability) from calibration (probability accuracy), demonstrating that improving one does not guarantee the other.
• Quantification of Harm: It quantifies the specific clinical impact of miscalibration, showing how mitigation strategies lead to significant over-estimation of risk, potentially causing unnecessary clinical interventions.
• Guidance on Metrics: It advocates for the use of threshold-independent metrics (Log Loss, AUROC, DCA) over threshold-dependent metrics (F1, Recall) when developing clinical risk models.

4. Key Results

Discrimination (AUROC/AUPRC):

• Class imbalance mitigation strategies did not improve discrimination.
• The AUROC and AUPRC remained statistically identical across the Natural model and all mitigation methods (e.g., Mortality AUROC: 0.94 for all; Complications AUROC: 0.84 for all).

Calibration (Log Loss & Probability Distribution):

• Natural Model: Demonstrated superior calibration with the lowest Log Loss (Mortality: 0.05; Complications: 0.23).
• Mitigation Models: All mitigation strategies (ROS, RUS, SMOTE, CSL) severely degraded calibration, resulting in significantly higher Log Loss values.
• Over-Prediction: Mitigation models systematically over-predicted risk. The probability distributions shifted from being skewed toward 0 (reflecting the low prevalence) to being more uniform.
  • In the clinical scenario, the proportion of surgeries classified as "high-risk" increased drastically:
    • Mortality: From 16.1% (Natural) to up to 78.9% (RUS).
    • Complications: From 31.0% (Natural) to up to 90.2% (ROS).
  • This represents an increase in high-risk classification of up to 62.8%.

Clinical Decision Making (DCA):

• Decision Curve Analysis showed that the Natural model provided the highest net benefit across all clinically relevant thresholds.
• Models trained with imbalance correction consistently showed lower net benefit, indicating that the harms of false positives (unnecessary interventions) outweighed the benefits of detecting true positives.

Threshold-Dependent Metrics:

• Mitigation methods improved Recall and F1-scores but at the cost of Precision and Accuracy. The authors argue these improvements are artifactual, resulting from shifting the decision threshold relative to the natural prevalence rather than genuine model improvement.

5. Significance and Conclusion

Clinical Implications:
The study concludes that class imbalance correction methods are detrimental for clinical risk prediction models where accurate probability estimation is required. By artificially altering the training distribution, these methods decouple model outputs from the true baseline risk (pre-test probability), leading to:

• Systematic over-estimation of patient risk.
• Misallocation of clinical resources (e.g., unnecessary ICU beds).
• Potential patient harm due to over-diagnosis or inappropriate surgical decisions.

Recommendations:

• Do not use class imbalance mitigation (ROS, SMOTE, RUS, aggressive CSL) for clinical models predicting rare events.
• Prioritize Calibration: Model selection should be based on calibration metrics (Log Loss, Calibration Plots) and Decision Curve Analysis rather than F1-scores or Recall.
• Natural Distribution: Models trained on the natural, imbalanced distribution of real-world data generally provide the most clinically valid and actionable probability estimates.

Limitations:
The study focused on Gradient Boosting models; while results are likely generalizable, other architectures (e.g., Deep Neural Networks) might behave differently. Additionally, the findings apply specifically to scenarios requiring probability outputs, not just binary ranking.