📄 medicine

Assistance curve analysis: evaluating the clinical value of AI-assisted decision making

This paper introduces a new metric, the net benefit of assistance (ΔNB), to evaluate the clinical value of AI by quantifying how it changes a clinician's decisions rather than just the model's accuracy, demonstrating through a chest X-ray study that AI can improve decision-making even when it reduces raw accuracy.

Original authors: Marwan Alqumbaey, Yousef M. Nasher

Published 2026-10-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Marwan Alqumbaey, Yousef M. Nasher

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern hospital, a doctor rarely makes a critical choice in a vacuum. When a radiologist looks at a chest X-ray, they are often aided by an artificial intelligence system that highlights potential problems. The doctor sees the machine's suggestion, weighs it against their own experience, and then makes the final call to treat a patient or send them home. For years, scientists have tried to measure whether these AI assistants actually help. The standard way to do this has been to check the machine's accuracy: how often does the AI correctly spot a disease on its own? If the AI is right more than it is wrong, it is considered good. But this approach misses the most important part of the story. The AI does not admit the patient; the doctor does. The real question is not whether the machine is smart, but whether the doctor makes better decisions when the machine is whispering in their ear.

This is where the new research from Marwan Alqumbaey and Yousef M. Nasher at Sana'a University steps in. They argue that the old way of measuring AI is like judging a coach by how well they can run a lap, ignoring whether they actually help the team win the game. The authors developed a new way to look at the data, focusing entirely on the moments when the AI changes a doctor's mind. They asked a simple, direct question: when a doctor sees the AI's suggestion and decides to do something different than they would have on their own, does that change help the patient or hurt them? To find out, they applied their new method to a massive, public study involving 227 radiologists reading hundreds of chest X-rays, some with AI help and some without.

The results were surprising and counterintuitive. When the researchers looked at the raw accuracy numbers, the AI appeared to be a failure. Across thousands of decisions, the radiologists using the AI were slightly less accurate than those working alone. Their overall success rate dropped by nearly one percentage point. By the traditional standards of the medical field, this tool would be rejected immediately. It seemed to be making doctors worse. However, when the researchers switched to their new method of measuring clinical value, the story flipped completely. The very same tool that lowered accuracy actually improved the doctors' decisions. At a specific threshold of risk that doctors commonly use, the AI helped the radiologists make the right call more often, adding a measurable benefit to patient care.

How can a tool make doctors less accurate but more helpful? The answer lies in the nature of medical mistakes. In medicine, not all errors are equal. Missing a serious condition like a collapsed lung is far more dangerous than ordering an unnecessary follow-up test. The old accuracy score treats these two mistakes as the same thing: a simple error. The new method, which the authors call an "assistance curve," understands that the cost of missing a disease is much higher than the cost of a false alarm. In this study, the AI convinced doctors to act more often. While this led to more false alarms (lowering the accuracy score), it also caught more real diseases that the doctors would have otherwise missed. Because catching the real disease was so much more valuable than avoiding the false alarm, the net result was a win for the patients. The AI was effectively saying, "Be careful here," and the doctors listened, saving lives even if their scorecard looked worse.

The researchers dug deeper to understand exactly where this value came from. They found that the AI's contribution was not spread evenly across all cases. In fact, the AI did not change the doctors' minds in the vast majority of cases; for nearly 90 percent of the X-rays, the doctors would have made the same decision with or without the machine. The entire clinical value of the AI was generated by that small minority of cases where the doctor changed their mind. When the AI moved a doctor to act, it was right about 11.9 percent of the time, which was a significant improvement over the baseline. When it moved a doctor to stand down, the finding was actually present in 9.4 percent of those cases. The tool worked because it successfully nudged doctors in the right direction at the moments that mattered most, even if it occasionally nudged them in the wrong direction.

This finding challenges the way the medical community currently evaluates artificial intelligence. The authors show that a tool can be rejected by standard metrics while simultaneously helping patients. If the radiologists in this study had been judged only by their accuracy scores, the AI would have been discarded, and the patients would have lost out on a helpful assistant. The study suggests that the field needs to stop asking "Is the AI accurate?" and start asking "Does the AI help the doctor make the right choice?" The new method requires no new data collection; it simply re-examines the decisions doctors already make in paired studies, looking at the specific moments of change. It reveals that the value of an AI is not in its overall intelligence, but in its ability to guide human judgment when it counts.

The study also looked at whether this result held up across different groups of doctors and different types of lung conditions. The positive effect was consistent across three different study designs involving nearly 200 additional radiologists. However, the benefit was not the same for every condition. For some specific findings, like a type of lung collapse, the AI provided a clear and significant benefit. For others, the effect was neutral or slightly negative. This variability is crucial. It means that a single "accuracy score" for an entire AI system is misleading because it averages together helpful and unhelpful effects. A tool might be a lifesaver for one condition and a distraction for another, and only a method that looks at the specific decisions can reveal this nuance.

Ultimately, this research offers a new lens for viewing the partnership between humans and machines in healthcare. It confirms that the goal of AI is not to replace the doctor or to be perfect on its own, but to improve the final decision. The study demonstrates that when we measure the right thing—the actual impact on patient care—we can find value where the old numbers saw only failure. The authors provide free software for other researchers to use this method, hoping to shift the conversation from simple accuracy to genuine clinical usefulness. In a world where medical decisions are increasingly supported by algorithms, knowing how to measure their true worth is not just a technical detail; it is a matter of ensuring that the tools we build actually help the people who need them most.

Technical Summary: Assistance Curve Analysis for Evaluating AI-Assisted Clinical Decision Making

Problem Statement
Current evaluation metrics for clinical Artificial Intelligence (AI) primarily assess the performance of the predictive model (e.g., accuracy, sensitivity, specificity, AUC) rather than the clinical value of the decisions made by a clinician assisted by that model. The authors argue that these standard metrics fail to capture the reality of clinical practice, where a doctor interprets AI output to make a final decision. Existing literature demonstrates that AI assistance can simultaneously improve and degrade clinician performance depending on whether the AI is correct or incorrect, yet standard reporting often relies on a single average accuracy figure. This averaging obscures the fact that the clinical value of an AI is determined entirely by the specific cases where it changes a clinician's mind, not by the vast majority of cases where the clinician would have acted identically regardless of the AI. Furthermore, existing decision-analytic tools do not explicitly evaluate the difference in net benefit between assisted and unassisted decisions in a paired setting.

Methodology
The paper introduces Assistance Curve Analysis, a formal extension of Decision Curve Analysis (DCA) designed specifically for paired reader studies where clinicians make decisions with and without AI support.

Net Benefit of Assistance ( $\Delta NB$ ): The authors define the net benefit of assistance as the difference between the net benefit of decisions made with AI ( $NB(y)$ ) and without AI ( $NB(c)$ ) at a specific threshold probability $p$ :
$\Delta NB(p) = NB(y; p) - NB(c; p)$
Where net benefit is calculated as $TP/n - w \times FP/n$ , with $w = p/(1-p)$ representing the trade-off between false positives and true positives.
The "Switch" Decomposition: A core theoretical contribution is the proof that $\Delta NB(p)$ depends only on the subset of cases where the clinician changed their decision (the "switches"). Cases where the decision remained the same ( $c_i = y_i$ ) contribute zero to the net benefit of assistance. The authors derive a closed-form equation for $\Delta NB(p)$ based on four quantities:
- $r_U$ : The rate of "up-switches" (clinician moves from no action to action).
- $\pi_U$ : The true positive rate within the up-switched cases.
- $r_L$ : The rate of "down-switches" (clinician moves from action to no action).
- $\pi_L$ : The true positive rate within the down-switched cases.
The formula is:
$\Delta NB(p) = \frac{r_U(\pi_U - p) + r_L(p - \pi_L)}{1 - p}$
Key Metrics Derived:
- Break-even Threshold ( $p^*$ ): The threshold at which $\Delta NB$ changes sign, calculated as $p^* = (r_L\pi_L - r_U\pi_U) / (r_L - r_U)$ . This identifies the range of clinical questions for which the tool is beneficial.
- Number Needed to Consult (NNC): Defined as $1/\Delta NB(p)$ , representing the number of cases an AI must review to generate one additional net true positive.
- Override Profile: An analysis of which specific ranges of AI output scores lead to beneficial or harmful decision changes.
Statistical Approach: The study employs cluster bootstrap resampling (clustered by radiologist) to estimate uncertainty, acknowledging that decisions by the same clinician are not independent. A cluster sign-flip permutation test is also used as a robustness check.

Application and Results
The method was applied to the Collab-CXR dataset, a public study involving 227 radiologists reading chest X-rays with and without a CheXpert-trained AI model. The primary analysis focused on Design 2, a rigorous paired design with 33 radiologists reading 60 cases each under all conditions (27,720 paired decisions).

Accuracy vs. Clinical Value: The AI assistance decreased overall accuracy by 0.83 percentage points (from 0.8465 to 0.8382) because the increase in false positives outweighed the gain in true positives. Under standard metrics, this tool would be rejected.
Net Benefit Gain: Conversely, the Net Benefit of Assistance ( $\Delta NB$ ) was positive across clinically relevant thresholds. At a threshold of $p=0.05$ , $\Delta NB$ was $+0.0022$ (95% CI: +0.0007 to +0.0038). This represents a 7% relative gain in clinical value over the unaided radiologist, or one extra net true positive for every 455 decisions.
Simulation Validation: In simulated scenarios, the authors demonstrated that an AI could improve accuracy while harming clinical value (e.g., a "reassuring" AI that talked clinicians out of acting on true positives) and vice versa (an "alarmist" AI that reduced accuracy but improved value by catching missed cases).
Heterogeneity: The effect varied by finding. For atelectasis, the AI provided a large benefit ( $\Delta NB = +0.0131$ ) despite reducing accuracy. For cardiomediastinal abnormality, the effect was slightly negative.
Replication: Two additional designs (Design 1 and Design 3) covering 194 additional radiologists confirmed the direction of the effect, though Design 2 (the most rigorous paired design) provided the most conservative estimate.

Key Contributions

Theoretical Extension: The paper provides the first formal extension of Decision Curve Analysis to the "assisted decision" setting, proving that clinical value is generated exclusively by the cases where the AI alters the clinician's decision.
Closed-Form Solution: It reduces the evaluation of AI assistance to four intuitive quantities ( $r_U, \pi_U, r_L, \pi_L$ ) that can be calculated from standard paired reader study data.
Reconciliation of Contradictory Literature: The method explains why some studies report accuracy degradation while others report clinical benefit, showing that these metrics measure different things and can point in opposite directions.
Open Source Implementation: The authors provide free software (assistcurve package) for R and Python to facilitate the adoption of this method.

Significance and Claims
The authors claim that judging AI by accuracy alone is insufficient and potentially misleading in clinical settings. In the primary analysis, a tool that improved patient outcomes (increased net benefit) would have been rejected if evaluated solely on accuracy.

The paper asserts that the value of an AI is not an intrinsic property of the model but is contingent on:

The specific clinical question (threshold probability).
The specific cases where the AI changes the clinician's mind.
Whether those changes are correct.

The authors conclude that current reporting guidelines are missing a critical component: a decision-analytic measure of the assisted decision across a range of thresholds. They argue that the "unit of analysis" for clinical AI should be the decision, not the model, and that the field must move beyond single-number summaries (like accuracy) to understand where and how AI tools help or harm. The study does not claim that the specific AI tested is universally beneficial, but rather that the method reveals a benefit that standard metrics hide, and that this benefit varies significantly across different clinical findings.

Technical Summary: Assistance Curve Analysis for Evaluating AI-Assisted Clinical Decision Making

More like this