Selective Prediction Reduces the Negative Effects of Automation Bias Overall but Increases False Negatives

Problem
High-performing AI models can produce inaccurate predictions that, when deployed, may harm human decision-making through automation bias (overreliance on incorrect AI outputs). To mitigate this, "selective prediction" (or AI deferral) has been proposed, where an AI system withholds predictions on cases it deems potentially harmful and asks the human to decide independently. However, typical evaluations of this approach rely on simulations assuming that when an AI abstains, humans revert to their baseline performance as if no AI were involved. This assumption has not been validated in multilabel tasks, where an AI might withhold some predictions while revealing others. Furthermore, prior empirical studies on deferral have largely focused on single-output binary settings, leaving a gap in understanding how selective prediction functions in complex, multilabel clinical workflows.

Methodology
The authors conducted a pre-registered, vignette-based study involving 259 clinicians (physicians and advanced practice practitioners) managing hospitalized patients with acute respiratory failure (ARF). The study utilized 45 real-world patient cases, each with ground-truth diagnoses for three concurrent conditions: pneumonia, heart failure, and chronic obstructive pulmonary disease (COPD).

The experimental design compared clinician performance across three conditions:

1. Clinician Alone: No AI input.
2. Clinician + AI: The AI provided predictions for all three conditions, regardless of accuracy.
3. Clinician + Selective Prediction: The AI withheld predictions it identified as potentially harmful (inaccurate) and explicitly informed the clinician that it was deferring ("the model defers to you"), while showing predictions for other conditions.

The selective prediction mechanism was constructed using post-hoc ground truth to identify potentially harmful predictions. To simulate realistic deployment imperfections, the mechanism withheld all incorrect predictions but also withheld a small subset (11%) of correct predictions. Clinicians were randomized to the AI-assisted conditions and asked to diagnose the causes of ARF and select corresponding treatments (antibiotics, diuretics, or steroids). Performance was measured by treatment accuracy, false positive rates (FPR), and false negative rates (FNR), with statistical analysis using mixed-effects models to account for repeated measures.

Key Contributions
This study presents the first evaluation of selective prediction in a multilabel clinical decision-making task. The authors challenge the behavioral assumption that human decision-making remains neutral when an AI explicitly abstains. Specifically, they demonstrate that:

1. Selective prediction partially mitigates the negative impact of inaccurate AI on overall treatment accuracy.
2. Selective prediction fundamentally alters the type of errors clinicians make, specifically increasing false negatives (undertreatment) when the AI abstains.
3. These effects vary by clinician characteristics, with those lacking prior AI experience and those with less medical training (Advanced Practice Practitioners) showing stronger susceptibility to these error shifts.

Results

• Impact of Inaccurate AI: When clinicians were shown all AI predictions (including inaccurate ones), their treatment accuracy dropped significantly (from 62% to 50%) compared to the baseline, driven by increases in both false positives and false negatives.
• Impact of Selective Prediction: When the AI withheld inaccurate predictions, treatment accuracy improved relative to the "Clinician + AI" condition (55% vs. 50%) but did not fully recover to the "Clinician Alone" baseline (62%).
• Error Shift: While the false positive rate returned to baseline levels under selective prediction, the false negative rate remained significantly elevated (42% vs. 31% in the baseline). This indicates that when informed the AI is deferring, clinicians are significantly more likely to undertreat patients who require intervention.
• Subgroup Analysis: The increase in false negatives under selective prediction was particularly pronounced among clinicians who had not previously used AI in their practice (AI-inexperienced) and among Advanced Practice Practitioners (APPs) compared to physicians.
• Perceived Difficulty: Clinicians perceived cases as more difficult when the AI was present, and difficulty ratings were highest when the AI used selective prediction.

Significance
The paper argues that selective prediction should not be treated as a default "safety mechanism" without context-specific evaluation. While it reduces overreliance on harmful AI outputs (reducing false positives), it introduces a new risk: it shifts clinicians toward undertreatment (increasing false negatives). In high-stakes contexts like ARF, where undertreatment can be more harmful than overtreatment, this trade-off is critical. The findings underscore that the mere act of an AI abstaining is not a neutral event; it alters human reasoning, likely due to availability bias or misinterpretation of the deferral as a lack of diagnosis. Consequently, the authors conclude that AI systems must be evaluated within real-world workflows, paying close attention to which types of errors are most acceptable for a specific clinical domain.