📊 statistics

Clinical Risk-Weighted Evaluation of ICU Mortality Prediction Models on MIMIC-IV

This paper proposes the Clinical Risk-Weighted Score (CRWS), a novel metric that decouples the penalties for false negatives and false positives to better reflect clinical priorities, demonstrating through MIMIC-IV analysis that it significantly alters model rankings and reveals greater clinical advantages for top-performing algorithms compared to the traditional F1 score.

Original authors: Aman Chandra H, Abhijna Shivaprakash, Amrutha Sachin Nagvekar, Spandana Sujay, Nagaraja J

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Aman Chandra H, Abhijna Shivaprakash, Amrutha Sachin Nagvekar, Spandana Sujay, Nagaraja J

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery in a bustling city. Your job is to spot the one person in a crowd of thousands who is about to commit a crime, so you can stop them before it happens. In the world of medicine, specifically in Intensive Care Units (ICUs), doctors play a similar role. They use computer programs to look at patient data and predict who might get very sick or pass away. This is a high-stakes game of "spot the danger."

But here is the tricky part: how do you know if your detective is actually good? Usually, we judge them by how often they get the right answer overall. But in a hospital, not all mistakes are the same. Imagine two types of errors:

The False Alarm (False Positive): The computer screams "Danger!" for a patient who is actually fine. The doctors rush over, check the patient, realize everything is okay, and sigh. It's annoying and wastes time, but nobody gets hurt.
The Missed Clue (False Negative): The computer says "All clear" for a patient who is actually dying. The doctors walk away, and the patient suffers a tragedy that could have been prevented.

For a long time, scientists have used a standard score called the F1 score to grade these computer programs. Think of the F1 score like a school report card that treats a "False Alarm" and a "Missed Clue" as exactly the same grade. It says, "You made two mistakes, so you get a C." But in a hospital, a missed clue is a disaster, while a false alarm is just a nuisance. This paper argues that using a report card that treats these two very different mistakes as equal is like grading a firefighter the same way for accidentally spraying water on a garden as they would for letting a house burn down.

The New Scorecard: CRWS

In this study, a team of researchers from RV University in India decided to fix this grading system. They created a new score called the Clinical Risk-Weighted Score (CRWS). You can think of CRWS as a "Safety-First Report Card." Instead of treating all mistakes equally, it gives a massive penalty to the computer if it misses a death, while being much more forgiving if it raises a false alarm.

To test this new score, the researchers used a giant, anonymous database of real hospital records called MIMIC-IV. This database contains information on 65,366 patients who had been in the ICU. In this group, 10.84% (about 7,086 people) passed away. It's a bit like a bag of 100 marbles where only 11 are red (the ones who died) and the rest are blue (the ones who survived). The computer's job was to find the red marbles.

The researchers trained four different types of computer "detectives" (called classifiers: Logistic Regression, Random Forest, XGBoost, and a Neural Network) to spot the red marbles. They then compared how these detectives performed using the old F1 score versus the new CRWS.

The Big Surprise: The "Best" Detective Was Actually the Worst

Here is where the story gets interesting. When the researchers used the old F1 score, the Random Forest detective looked like the star student. It had the highest accuracy of 87.23%. That sounds amazing, right? It got the answer right for almost 9 out of 10 patients.

But when they looked closer, they found a scary secret. Even though Random Forest had the highest accuracy, it missed the most deaths. It failed to spot 1,220 patients who actually died. It was so afraid of making false alarms that it decided to just guess "everyone will survive" for almost everyone. This made its accuracy look great, but in a hospital, missing 1,220 deaths is a catastrophic failure. Under the new CRWS, which hates missing deaths, Random Forest fell to the very bottom of the rankings with a score of 0.147.

On the other hand, the XGBoost detective looked a bit messy on the old report card. It had a lower accuracy of 59.32% because it raised a lot of false alarms (it told doctors to check on patients who were actually fine). However, it only missed 289 deaths. It was much better at catching the people who were actually in trouble. When the researchers applied the new CRWS, XGBoost jumped to the top of the class with a score of 0.596.

Why This Matters

The paper shows that the way we measure success changes who we think is the "best" tool.

Under the old rules (F1): XGBoost was #1, but Random Forest was still in the running.
Under the new rules (CRWS): XGBoost is clearly the winner, and Random Forest is revealed to be dangerous because it misses too many critical cases.

The researchers also tested if this result was just a fluke. They ran the numbers 500 times (a method called bootstrap) and used a statistical test called McNemar's test. The results were clear: the difference between the models wasn't a lucky guess. The ranking shift was real, and the difference between the best and worst models was statistically significant (with a p-value of less than 0.001).

They even tried changing the "penalty" in their new score. They asked, "What if we care even more about missing a death?" or "What if we care a bit more about false alarms?" They found that no matter how they tweaked the settings, XGBoost stayed on top, and Random Forest stayed at the bottom. This proves that the new score is robust and reliable.

The Takeaway

This paper doesn't claim to have built the perfect hospital robot. In fact, the authors are very careful to say they only used 10 simple features (like age, gender, and how long the patient stayed in the ICU) to keep the experiment clean. They aren't saying this specific computer program is ready to save lives in a real hospital tomorrow.

Instead, they are saying: "Stop grading your medical AI with a report card that treats all mistakes the same."

If you are building a system to predict who might die, you need a score that screams "FAIL" if the system misses a death, even if the system has a high overall accuracy. The Clinical Risk-Weighted Score (CRWS) is that new tool. It helps doctors and scientists see the truth: that a model with lower accuracy but fewer missed deaths is actually the safer, better choice for saving lives.

Technical Summary: Clinical Risk-Weighted Evaluation of ICU Mortality Prediction Models on MIMIC-IV

Problem Statement
In Intensive Care Unit (ICU) mortality prediction, the clinical consequences of prediction errors are asymmetric. A False Negative (FN)—predicting a patient will survive when they actually die—represents a missed opportunity for life-saving intervention and is clinically catastrophic. Conversely, a False Positive (FP)—triggering an unnecessary alert for a surviving patient—is resource-intensive but generally less immediately harmful. However, standard evaluation metrics, particularly the F1 score, treat FN and FP errors as equally costly. This paper argues that relying on such symmetric metrics in clinical AI can lead to the selection of models that maximize overall accuracy or F1 scores while failing to detect the most critical cases (deaths), thereby misrepresenting clinical risk.

Methodology
The study utilizes the MIMIC-IV v3.1 dataset, comprising 65,366 unique adult patients with a 10.84% in-hospital mortality rate. The experimental setup isolates the behavior of evaluation metrics by training four classifiers (Logistic Regression, Random Forest, XGBoost, and a Multi-Layer Perceptron) on a fixed set of ten static tabular features (e.g., age, ICU length of stay, admission type). To address class imbalance, the training set was balanced using SMOTE, while the test set retained the original imbalanced distribution.

The core methodological contribution is the Clinical Risk-Weighted Score (CRWS), a two-parameter generalization of the F-β score. Unlike the standard F-β score, which uses a single parameter ( $\beta$ ) to weight recall against precision, CRWS decouples the penalties for False Negatives ( $w_{FN}$ ) and False Positives ( $w_{FP}$ ).

Formulation: $CRWS = \frac{\alpha TP}{\alpha TP + w_{FN}^2 FN + w_{FP} FP}$ , where $\alpha = 1 + w_{FN}^2$ .
Parameters: The authors adopt a primary setting of $w_{FN}=3$ and $w_{FP}=1$ to reflect a 3:1 cost ratio where missed deaths are three times more penalized than false alarms.
Theoretical Properties: CRWS reduces exactly to the F1 score when $w_{FN}=w_{FP}=1$ and to the standard F-β score when $w_{FP}=1$ . The inclusion of $w_{FP}$ allows for independent tuning of false alarm penalties, a capability absent in F-β.

The evaluation protocol included a comprehensive comparison of CRWS against F1, F2, MCC, Balanced Accuracy, AUPRC, and AUROC. Robustness was verified through sensitivity analysis (varying $w_{FN}$ and $w_{FP}$ ), threshold optimization, bootstrap confidence intervals ( $n=500$ ), and an ablation study comparing SMOTE against class weighting. Statistical significance of model rankings was assessed using McNemar's test.

Key Results
The application of CRWS revealed significant shifts in model rankings compared to standard metrics:

Ranking Inversion: Under the F1 score, XGBoost ranked first (0.298), followed by MLP, Logistic Regression (LR), and Random Forest (RF). Under CRWS ( $w_{FN}=3$ ), XGBoost remained first (0.596), but LR moved to second (0.480), while RF dropped to last (0.147).
The Accuracy-Safety Paradox: Random Forest achieved the highest accuracy (87.23%) but produced the highest number of missed deaths (FN = 1,220). CRWS correctly penalized this behavior, ranking RF last.
XGBoost Performance: XGBoost achieved the highest CRWS (0.596) and the lowest number of missed deaths (FN = 289), despite having a lower accuracy (59.32%) and higher false positive count. Its high recall (0.796) and AUROC (0.752) confirmed its superior sensitivity to the minority class.
Threshold Optimization: Optimizing decision thresholds specifically for CRWS (rather than F1) resulted in dramatic reductions in missed deaths. For example, adjusting the Random Forest threshold reduced missed deaths from 1,220 to 188 (an 84.4% reduction).
Statistical Significance: Pairwise comparisons via McNemar's test confirmed that all model differences were statistically significant ( $p < 0.001$ ). Bootstrap confidence intervals for CRWS were non-overlapping across all models, whereas F1 intervals showed partial overlap between LR and MLP, indicating CRWS provides better discriminative power in this context.

Significance and Claims
The paper claims that CRWS serves as a clinically interpretable, complementary metric that reveals ranking differences hidden by standard evaluation. It demonstrates that models optimized for or evaluated by symmetric metrics (like F1) may inadvertently prioritize the majority class (survivors) at the expense of the critical minority class (deaths).

The authors emphasize that CRWS is not proposed as a universal replacement for F1 but as a tool for scenarios where minimizing high-risk errors is the primary operational objective. The study validates that:

Standard metrics can misrepresent clinical risk by failing to account for asymmetric error costs.
CRWS allows practitioners to explicitly tune the trade-off between missed deaths and false alarms.
Model selection based on CRWS leads to different, clinically safer outcomes than selection based on F1 or Accuracy.

Limitations and Scope
The authors explicitly state that the results are based on ten static tabular features and should be interpreted as a demonstration of evaluation metric behavior rather than a deployment-ready clinical prediction system. The study is limited to a single dataset (MIMIC-IV), a single task (ICU mortality), and a single cost assumption (3:1 ratio). Future work is proposed to extend CRWS to multi-class settings, incorporate severity scores (e.g., SOFA, APACHE) as adaptive weights, and validate on additional tasks.

The New Scorecard: CRWS

The Big Surprise: The "Best" Detective Was Actually the Worst

Why This Matters

The Takeaway

Technical Summary: Clinical Risk-Weighted Evaluation of ICU Mortality Prediction Models on MIMIC-IV

More like this