💻 computer science

Multi Armed Bandit Based Adaptive Model Selection for Clinical AI Governance An Infrastructure Approach to Safer AI Use in U.S. Healthcare

This paper proposes a Multi-Armed Bandit-based governance framework for clinical AI that dynamically selects among validated models by balancing predictive quality, cost, and safety penalties, demonstrating through MIMIC-IV simulations that while regret-minimizing policies outperform static deployment, confident retirement of suboptimal models requires significantly more data than typical clinical streams provide.

Original authors: Julian Borges

Published 2026-10-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Julian Borges

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the high-stakes environment of a hospital intensive care unit, doctors rely on computer programs to help predict which patients are most likely to die. These programs, known as clinical artificial intelligence models, are trained on vast amounts of past medical records to spot subtle patterns in vital signs and lab results. However, hospitals often find themselves with several different versions of these programs, each built by different teams and each requiring different amounts of data to run. Some are simple and cheap to operate, needing only basic heart rate and temperature readings. Others are complex and expensive, requiring a full suite of blood tests and advanced imaging. The standard practice has been to pick one model, stick with it, and use it for every patient until a scheduled review. This static approach assumes that the model chosen during a one-time test will remain the best choice forever, ignoring the reality that patient populations change, data availability shifts, and the cost of running these programs varies from day to day.

The challenge for hospital administrators is how to manage this portfolio of tools without making risky guesses. They need a system that can decide, in real time, which model to use for a specific patient, balancing the cost of running the program against the safety of the patient. If a hospital switches too quickly, they might discard a good tool based on a fluke of bad luck. If they switch too slowly, they might keep using an expensive or inaccurate tool long after a better option has emerged. This dilemma sits at the intersection of computer science and medical governance, asking a fundamental question: how much evidence is actually required to confidently retire an old model and replace it with a new one, especially when the results of a patient's outcome might not be known for days?

A researcher at Boston University, Julian Borges, tackled this question by building a digital simulation of a hospital's decision-making process. Instead of testing this on real patients, which would be ethically impossible and potentially dangerous, he created a virtual environment using a massive, publicly available database of past intensive care unit stays. In this simulation, three different computer models were treated as "arms" of a decision-making machine. One arm was a simple model using only vital signs, another was a medium-complexity model using core lab tests, and the third was a complex model using an extended panel of lab tests. Each arm had a different "cost" associated with it, representing the operational burden of gathering and processing the necessary data. The goal was to see if an adaptive system could learn to pick the best arm for each patient, or if it could confidently decide to stop using a specific arm entirely.

The study introduced a new way of thinking about model selection, treating it not as a one-time configuration but as a continuous sequence of choices. The system was given a set of rules, or "governance weights," that told it how much to value saving money versus saving lives. For instance, the system was programmed to heavily penalize missing a death, while also accounting for the cost of running the more complex models. The researchers then ran the simulation over 7,383 virtual patient stays, mimicking the flow of a real hospital where outcomes arrive with a delay, sometimes hours or days after a prediction is made. They compared their adaptive system against several other strategies, including simply sticking with the model that looked best on paper before the simulation started, and other mathematical methods designed to minimize regret, or the loss incurred by not picking the perfect option every time.

The results of this simulation revealed a surprising and sobering reality about the speed of change in clinical AI. Despite running thousands of simulations, the adaptive system designed to make confident, error-controlled decisions never managed to retire any of the three models. The differences in performance between the simple, cheap model and the complex, expensive ones were simply too small to be detected with certainty within the timeframe of the study. The system found that the "cheapest" model was actually the best choice under the specific rules set by the researchers, but the gap between it and the other models was so narrow that the system needed far more data to be sure. To confidently prove that the most expensive model was worse than the cheapest one, the simulation showed that the hospital would need to observe roughly 7,500 outcomes for that specific model. In the simulated stream of 7,383 patients, this requirement was not met.

This finding suggests that the idea of rapidly swapping out AI models based on short-term performance data may be unrealistic. The study calculated that, at the rate patients typically arrive at this specific hospital, it would take nearly five years of continuous admissions just to gather enough evidence to confidently retire the most expensive model. Even if a model was clearly broken and performing poorly, the system still needed about 3,000 outcomes per model to be absolutely certain it should be discarded, a process that would take roughly two years. The researchers noted that while other, faster methods could switch away from a bad model more quickly, they did so without a guaranteed safety net, meaning there was always a chance they might make a mistake.

The study also explored what would happen if the model chosen by the hospital was actually the wrong one to begin with. In these scenarios, where the "best" model on paper performed poorly in the simulation, the faster, regret-minimizing systems were able to switch to a better model within the stream of patients. However, the system designed to be absolutely certain about its decisions took much longer to make that switch, even when the original model was clearly failing. This highlights a fundamental trade-off: a hospital can choose to switch models quickly and risk making an error, or it can wait for overwhelming evidence to be certain, but that certainty comes at the cost of time and potentially continued use of a suboptimal tool.

Ultimately, the research demonstrates that the decision to retire a clinical AI model is not just a technical problem but a governance one. The speed at which a hospital can change its tools depends heavily on the rules it sets for itself, such as how much it values cost versus accuracy, and the inherent difficulty of the task. The study provides a clear mathematical framework for hospital committees to use, allowing them to calculate exactly how many patient outcomes they need to collect before they can safely make a change. It turns a vague feeling of "maybe we should switch" into a concrete number, showing that for many current AI models, the evidence horizon is long, and patience is a necessary part of safe, adaptive AI use.

Technical Summary: Multi-Armed Bandit Based Adaptive Model Selection for Clinical AI Governance

1. Problem Statement

Healthcare institutions increasingly possess multiple validated clinical prediction models for the same task (e.g., in-hospital mortality). Current deployment practices typically involve a static selection of a single model based on retrospective validation, which is then applied to all patients until a scheduled review. This approach fails to account for:

Sequential Decision Making: The need to choose a model for each patient under uncertainty.
Delayed Feedback: Outcomes (e.g., death) often arrive hours or days after the prediction.
Operational Constraints: Different models have varying computational footprints and data acquisition costs.
Safety Priorities: Specific errors, such as missed deaths, carry higher penalties than others.

The paper addresses the lack of a principled, auditable mechanism to decide when routine use data justifies switching or retiring a model, particularly when outcomes are delayed and governance weights (cost vs. safety) are explicit.

2. Methodology

The study employs a retrospective simulation on a temporally external stream of 7,383 ICU stays from the MIMIC-IV v3.1 database (2020–2022), using models trained on 20,000 earlier stays (pre-2020).

Decision Arms:
Three "frozen" in-hospital mortality models with distinct operational footprints serve as the decision arms:

Arm A: Ridge-penalized logistic regression (Vital signs only; Cost: 0.25).
Arm B: Gradient boosted tree (Vital signs + Core labs; Cost: 0.60).
Arm C: Random forest (Vital signs + Extended labs; Cost: 1.00).

Reward Function:
A composite reward $r_{a,t}$ combines predictive quality, cost, and safety:
$r_{a,t}^{raw} = (1 - (p_{a,t} - y_t)^2) - \lambda_c c_a - \lambda_s s_{a,t}$

Predictive Quality: One minus the Brier loss (a proper scoring rule).
Cost ( $c_a$ ): A declared operational cost constant.
Safety Penalty ( $s_{a,t}$ ): A penalty for missed deaths (false negatives) at the operating threshold.
Weights: Primary governance weights were set to $\lambda_c = 0.10$ (cost) and $\lambda_s = 0.50$ (safety).

Algorithms Evaluated:

Fixed Confidence Action Elimination: The primary focus. It eliminates arms when their upper confidence bound falls below another arm's lower bound. Two radii were tested: Hoeffding and Empirical Bernstein. The policy cycles through active arms in round-robin order until an arm is retired.
Regret-Minimizing Policies: Thompson Sampling (Beta and Gaussian variants), UCB1.
Baselines: Static deployment (best arm from validation), Uniform Random Allocation, Ensemble Averaging, and a Heuristic Escalation rule.

Constraints:

Delayed Feedback: Outcomes were simulated with stochastic delays (6–72 hours).
Fixed Confidence: The elimination policy aims to bound the probability of erroneously retiring the best arm ( $\delta = 0.05$ ).
Reproducibility: All results were generated by a single run of a public, version-controlled pipeline.

3. Key Results

Primary Simulation (Static Validation Choice):

No Retirement: Under the primary weights, the fixed confidence elimination policy did not retire any arm within the 7,383-decision horizon. The reward gaps between models were small (0.015 and 0.040), driven entirely by the cost term.
Sample Complexity: To retire the most expensive arm (Arm C) with 95% confidence, the simulation estimated a need for 7,508 outcomes per arm (approx. 22,524 total decisions) using the Empirical Bernstein radius. The stream only supplied ~2,461 outcomes per arm.
Regret vs. Certainty:
- Regret Minimization: Thompson Sampling (Gaussian variant) achieved the lowest regret (13.4 units) and converged on the preferred arm (Arm A) quickly but left exposure to inferior models uncontrolled.
- Fixed Confidence: Elimination policies incurred higher regret (134.2 units) because they maintained uniform exploration (round-robin) until a confident decision could be made.
Cost Dominance: The preferred model (Arm A) was the cheapest, despite being the least accurate. The cost term accounted for the entire separation in reward; the predictive and safety terms actually favored the more complex models.

Drift Scenarios (Validation Choice is Wrong):

When the validation-selected model (Arm A) was artificially degraded on the stream:
- Regret Minimizers: Adapted quickly, beating static deployment of the wrong model within the stream (e.g., Gaussian Thompson regret of 12.9 vs. Static regret of 624.6 for a "broken" model).
- Fixed Confidence: Required significantly more data to retire the broken model with high confidence. Even for a severely degraded model, confident retirement required ~3,060 outcomes per arm (approx. 9,180 decisions), taking roughly 2 years of admissions at the institution's accrual rate.

Extended Horizon:

Resampling the stream to 30,000 decisions confirmed the sample complexity calculations. The Empirical Bernstein policy successfully retired the inferior Arm C in 100% of replicates at the predicted decision index, while the Hoeffding policy (requiring ~100,000 decisions) failed to retire any arm.

4. Key Contributions

Governance Framework: A fully specified, reproducible framework treating model selection as a sequential decision problem with explicit governance weights for cost, safety, and alert burden.
Quantified Evidence Horizon: The study quantifies the "evidence horizon" required for confident model retirement. It demonstrates that for realistic gaps between validated models, fixed confidence elimination may require years of data, far exceeding typical deployment cycles.
Trade-off Analysis: It explicitly characterizes the trade-off between certainty (fixed confidence elimination) and speed (regret minimization). Regret minimizers adapt faster but lack a bounded error guarantee for model retirement.
Reproducibility: The paper provides a complete, auditable pipeline where every reported number is generated by a single run, correcting previous preprint errors regarding model training and policy implementation.

5. Significance and Claims

The paper claims to provide a planning instrument for clinical AI governance committees rather than a recommendation to deploy specific algorithms immediately.

Explicit Governance: It formalizes that the "best" model is a function of institutional weights (cost vs. safety) and the cost model, not an intrinsic property of the algorithms.
Realistic Constraints: It highlights that under realistic conditions (small performance gaps, delayed feedback, operational costs), confident retirement of a model is a slow process. A governance body demanding a bounded probability of error must either accept a long evidence horizon, pool data across sites, or set weights that create larger reward gaps.
Safety vs. Speed: The study concludes that when a validation choice is wrong, regret-minimizing policies stop using the wrong model quickly but without a guarantee. Fixed confidence policies provide the guarantee but may continue using a suboptimal (or even broken) model for a significant period (e.g., ~2 years for a broken model) before the data is sufficient to retire it with high confidence.
Methodological Rigor: The work emphasizes that sample complexity calculations (the number of outcomes needed) are transferable planning tools, even if the specific magnitudes depend on the specific database and cost model used.

The paper does not claim that adaptive selection is universally superior to static deployment; rather, it argues that the decision to switch from static to adaptive governance requires a clear understanding of the cost of exploration versus the cost of deploying the wrong model, and that fixed confidence guarantees come with a substantial data requirement.