📊 statistics

RISED: A Pre-Deployment Safety Evaluation Framework for Clinical AI Decision-Support Systems

The paper introduces RISED, a comprehensive pre-deployment safety framework that evaluates clinical AI systems across five dimensions—Reliability, Inclusivity, Sensitivity, Equity, and Deployability—to detect critical failures missed by aggregate accuracy metrics and ensure robust, equitable, and operationally feasible deployment.

Original authors: Rohith Reddy Bellibatlu

Published 2026-05-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Rohith Reddy Bellibatlu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a brilliant new car engine. On the test track, it runs perfectly. It's fast, it's quiet, and it gets great gas mileage. You're ready to sell it to the public. But before you put it on the road, you realize you haven't checked if it works in the rain, if it handles potholes, or if the brakes work when you're driving on a steep hill.

This paper introduces RISED, a new "safety inspection" for Artificial Intelligence (AI) systems that doctors plan to use. The author argues that current ways of testing medical AI are like only checking the engine on a perfect, dry test track. They look at the final score (accuracy) but miss the hidden dangers that happen when the AI actually tries to work in a real hospital.

Here is the RISED framework, broken down into five simple checks, using the paper's own findings:

1. Reliability: The "Translation" Test

The Metaphor: Imagine you give the same recipe to three different chefs. One writes "1 cup of flour," another writes "240 grams," and the third writes "a bowl full." If the AI is a good chef, it should make the same cake regardless of how the ingredients are described. If it makes a cake with a hole in it just because the recipe said "grams" instead of "cups," it's unreliable.

What the Paper Found:
The authors tested an AI model that had a very high "test score" (96.1% accuracy). However, when they slightly changed how the data was written (like changing units from milligrams to grams, or slightly shifting dates), the AI changed its mind on about 6.4% of patients.

Verdict: FAIL. Even though the model was "smart," it was too sensitive to tiny changes in how data was typed up.

2. Inclusivity: The "Fairness" Test

The Metaphor: Imagine a security guard at a club who lets everyone in, but when you look closely, they are accidentally turning away 1 out of every 20 people from a specific neighborhood, even though those people have the same ticket as everyone else. The guard looks fair on average, but not for everyone.

What the Paper Found:
The AI worked great for most people, but it struggled significantly with the oldest patients (75+). The gap in performance between the best group and the worst group was just barely over the safety limit. Because the data was a bit noisy, the inspectors couldn't say for sure if it was a total failure or just a warning sign.

Verdict: INCONCLUSIVE. It's a "maybe." The model might be unfair to the elderly, but the test wasn't big enough to be 100% sure.

3. Sensitivity: The "Tuning Knob" Test

The Metaphor: Imagine a smoke detector. If you set it to be very sensitive, it screams when you toast bread. If you set it to be less sensitive, it ignores real fires. The problem is: if you have to turn the knob just a tiny bit to make it useful in a real kitchen, does it suddenly start screaming at half the people in the house?

What the Paper Found:
In the real world, doctors often have to adjust the "sensitivity" of an AI (e.g., "Let's flag more patients to be safe"). The paper found that if they adjusted the AI's settings just a little bit, the list of patients it flagged changed by nearly 20%.

Verdict: FAIL. The model is too unstable. A tiny change in the rules causes a huge change in who gets help.

4. Equity: The "Need" Check

The Metaphor: Imagine a triage nurse who decides who gets a doctor first. If the nurse looks at who called the most often in the past, they might help the rich people who can afford to call, and ignore the sick poor people who can't. The nurse needs to look at who is actually sick, not who called the most.

What the Paper Found:
The authors tried to see if the AI helped the people who needed it most. But they found a trap: if they used "past hospital bills" to guess who needed help, the AI looked good. But if they used "actual health scores" (which are different), the AI looked bad.

Verdict: DIAGNOSTIC (Not a Pass/Fail). The paper says this isn't a simple "fail" yet. It's a warning light saying: "You are using the wrong ruler to measure need. Go find a better ruler before you deploy this."

5. Deployability: The "Speed and Clarity" Test

The Metaphor: Imagine a GPS that takes 10 minutes to tell you where to turn, or one that gives you directions in a language you don't speak. Even if the route is perfect, it's useless if it's too slow or confusing.

What the Paper Found:
The AI was incredibly fast (taking less than 2 milliseconds to process a whole group of patients) and the reasons it gave for its decisions made sense to doctors.

Verdict: PASS. It's fast and easy to understand.

The Big Picture

The most surprising thing the paper found is that you can have a "perfect" AI score and still fail the safety inspection.

The model they tested had a 96% accuracy rating, which usually means "Green Light, go ahead!" But under the RISED inspection:

It Failed the Reliability test (it breaks if data is typed differently).
It Failed the Sensitivity test (it changes its mind too easily).
It was Inconclusive on Inclusivity (it might be unfair to the elderly).
It Passed Deployability (it's fast).

The Conclusion:
RISED is a tool that says, "Don't just look at the final grade. Check if the car handles rain, if the brakes work on hills, and if the map is clear." The authors released this as a free software package so hospitals can run these specific checks before letting an AI start making decisions about real patients. They argue that without this, we risk deploying systems that look great on paper but fail in the real world.

Technical Summary: RISED – A Pre-Deployment Safety Evaluation Framework for Clinical AI Decision-Support Systems

Problem Statement
Current evaluation paradigms for clinical AI decision-support systems rely heavily on aggregate accuracy metrics (e.g., AUROC, Brier score) derived from static, held-out test sets. The authors argue that these metrics fail to detect critical deployment-phase failures, including input reliability instability due to encoding shifts, subgroup equity degradation, threshold sensitivity, and operational infeasibility. Citing documented failures such as the Epic Sepsis Model, which passed internal benchmarks but failed in external validation due to poor discrimination and alert fatigue, the paper posits that existing tools (e.g., AI Fairness 360, Fairlearn, TRIPOD+AI) are designed for static model selection or post-hoc reporting rather than determining operational readiness for live clinical environments.

Methodology: The RISED Framework
The paper proposes the RISED Framework, a five-dimension pre-deployment evaluation approach. Each dimension is operationalized through formal sub-criteria, pre-specified pass/fail thresholds, and bias-corrected accelerated (BCa) bootstrap 95% confidence intervals (CIs). A Holm-Bonferroni correction is applied to control the family-wise error rate across the multiple hypothesis tests. The framework yields three possible verdicts per dimension: PASS, FAIL, or INCONCLUSIVE (when the CI brackets the threshold).

The five dimensions are:

Reliability: Measures output stability under semantically equivalent but differently encoded inputs (e.g., unit changes, ICD granularity shifts).
- Metric: Judge Sensitivity Score (JSS), defined as the average Perturbation Flip Rate (PFR) across a battery of perturbations.
- Threshold: JSS < 0.05.
Inclusivity: Assesses performance consistency across demographic subpopulations.
- Metrics: AUC Parity Gap ( $\Delta$ AUC) and subgroup Expected Calibration Error (ECE).
- Threshold: $\Delta$ AUC $\le$ 0.05; ECE $\le$ 0.10 for all subgroups.
Sensitivity: Evaluates behavioral stability under decision-threshold shifts, crucial as thresholds are often retuned post-deployment.
- Metric: Threshold Flip Rate (TFR), measuring the fraction of patients reclassified when the threshold moves from a reference $\tau_0$ .
- Threshold: Max TFR $\le$ 0.10 across a sweep of thresholds.
Equity: Tests alignment of predictions with an independent measure of clinical need, rather than utilization proxies that may encode historical disparities.
- Metric: Need-Prediction Correlation ( $\rho_{need}$ ) and group-level need gaps.
- Status: Framed as a DIAGNOSTIC rather than a gating verdict. If the verdict changes based on the choice of need proxy (e.g., outcome label vs. CCI), it triggers a requirement for an outcome-independent need measure before deployment.
Deployability: Assesses operational feasibility, including inference latency and explanation faithfulness.
- Metrics: Mean inference latency ( $\Lambda$ ) and Explanation Faithfulness ( $F_{expl}$ ), measuring if locally important features align with globally important ones.
- Threshold: $\Lambda \le$ 500 ms per cohort; $F_{expl} \ge$ 0.50.

Experimental Results
The framework was applied to a 10,000-patient synthetic cohort and three real-world cohorts spanning 35 years of data vintage: the UCI Heart Disease cohort (1989), the UCI Diabetes 130-US Hospitals cohort (1999–2008), and the NCHS National Health Interview Survey (NHIS) 2024.

Synthetic Cohort (XGBoost, AUROC 0.961): Despite high aggregate discrimination, the model FAILED the Reliability (JSS = 0.064) and Sensitivity (Max TFR = 19.9%) dimensions. The Inclusivity dimension was INCONCLUSIVE ( $\Delta$ AUC = 0.059, CI [0.042, 0.066]). Equity was DIAGNOSTIC, showing a verdict flip between the outcome-label proxy (PASS) and the Charlson Comorbidity Index proxy (FAIL). Deployability PASSED.
Real-World Cohorts:
- UCI Heart Disease (n=303): Reliability and Inclusivity were INCONCLUSIVE due to small sample sizes; Sensitivity FAILED.
- UCI Diabetes 130 (n=99,492): Reliability PASSED (JSS = 0.0004), but Inclusivity ( $\Delta$ AUC = 0.262) and Sensitivity (Max TFR = 49.1%) FAILED decisively. Equity remained DIAGNOSTIC with proxy disagreement.
- NHIS 2024 (n=9,747): Reliability PASSED (JSS = 0.011), while Inclusivity ( $\Delta$ AUC = 0.328) and Sensitivity (Max TFR = 22.5%) FAILED.
Multi-Model Robustness: When tested with Logistic Regression and Random Forest on the synthetic cohort, Inclusivity and Sensitivity failures persisted (data-driven), while Reliability failures were model-dependent (XGBoost failed; others passed).
Comparison with Fairlearn: Fairlearn identified subgroup disparities but failed to detect input-perturbation instability, threshold sensitivity, or need-based equity misalignment, highlighting RISED's broader scope.

Key Contributions

Framework Definition: Organizes existing literature into a five-dimension pre-deployment framework specific to clinical decision support, moving beyond static model selection.
Quantitative Gates: Replaces qualitative checklists with formal sub-criteria, bootstrap 95% CIs, and explicit PASS/FAIL/INCONCLUSIVE decision rules.
Empirical Validation: Demonstrates that a model with high AUROC can simultaneously fail input-encoding stability and threshold-shift sensitivity checks, a pattern validated across synthetic and three distinct real-world cohorts.
Equity Reframing: Reframes the Equity dimension as a "proxy-dependence diagnostic," explicitly surfacing construct-validity issues when need proxies are derived from utilization data, thereby mandating the procurement of outcome-independent need measures.
Open-Source Implementation: Releases the rised Python package, providing the quantitative verdicts required by existing reporting standards (TRIPOD+AI, MI-CLAIM, FUTURE-AI).

Significance and Claims
The paper claims that RISED provides a "principled gateway" between in-silico model validation and silent-trial clinical evaluation. It argues that aggregate metrics alone are insufficient to detect deployment risks such as encoding instability and threshold sensitivity. The framework is positioned not as a regulatory mandate but as a structured numerical evidence generator that can populate FDA Predetermined Change Control Plan (PCCP) submissions, ONC HTI-1 risk-management disclosures, and EU AI Act technical files. The authors state that the framework is intended to "provoke empirical work" on threshold calibration and that while a RISED PASS is necessary for deployment, it is not sufficient, as it does not address implementation governance or clinician trust. The results provide preliminary evidence of construct validity, showing that different dimensions fail differentially across cohorts and model classes, suggesting that deployment readiness is a multi-faceted property not captured by single-metric benchmarks.