Learning from Disagreement: Clinician Overrides as Implicit Preference Signals for Clinical AI in Value-Based Care

1. Problem Statement

Current Clinical Decision Support (CDS) systems suffer from high override rates (often >96%), which are traditionally interpreted as system failures (alert fatigue, poor specificity) to be minimized. The paper argues this interpretation is fundamentally flawed.

• The Core Misconception: Treating overrides as noise discards the richest source of preference data available in clinical AI.
• The Specific Challenge: In standard Reinforcement Learning from Human Feedback (RLHF), annotators are often uniform or treated as having equal weight. In clinical settings, clinician heterogeneity is critical. A "rejection" of an AI recommendation can stem from:
  1. Genuine Clinical Judgment: The AI is wrong (valid preference signal).
  2. Capability Gaps: The AI is right, but the clinician lacks the skill/confidence to execute the action (invalid preference signal for the reward model, but valid signal for a capability model).
  3. Workflow/Protocol Constraints: External factors unrelated to clinical quality.
• The Failure Mode: Naive preference learning systems conflate these signals. If a population of low-capability clinicians overrides a correct but difficult recommendation (e.g., initiating SGLT2 inhibitors for heart failure), the system learns that the recommendation is "bad" and suppresses it. The authors term this Suppression Bias.

2. Methodology & Formal Framework

The paper proposes a formal framework that reframes overrides as implicit preference signals conditioned on three variables: Patient State ($s$), Organizational Context/Contract ($c$), and Clinician Capability ($\kappa$).

A. The Override Taxonomy

The authors categorize overrides into five distinct types, each mapping to a specific model update target:

1. Context: Clinician has private info not in the AI's state (Update: Expand State Representation).
2. Judgment: Genuine disagreement with evidence (Update: Reward Model).
3. Workflow: Constraints like time/staffing (Update: Filter from training data).
4. Protocol: Institutional rules superseding AI (Update: Expand Contract Context).
5. Capability: Clinician lacks skill to execute (Update: Capability Model).

B. The Dual Learning Architecture

The framework introduces a dual learning problem solved via alternating optimization to separate the "what is right" from "who can do it."

1. Reward Model ($R_\theta$): Learns the latent clinical-economic value of actions. It is trained on preference pairs weighted by clinician capability.
   • Formula: $P(a' \succ r | s, c, \kappa) = \sigma(\beta(\kappa) \cdot [R^*(s, a', c) - R^*(s, r, c)])$
   • The weighting function $\beta(\kappa)$ ensures high-capability clinicians drive reward updates, while low-capability overrides are down-weighted.
2. Capability Model ($\kappa_\phi$): Learns the domain-specific competence of each clinician.
   • It estimates how well a clinician's behavior aligns with the current best estimate of the reward model.
   • Decomposition: $\kappa$ is split into Execution Capability (can they do it?) and Alignment Capability (do they practice in a way aligned with value-based outcomes?).

C. Alternating Optimization Loop

To prevent circular reasoning and suppression bias, the system iterates:

1. E-Step: Fix the Reward Model ($R_\theta$) and estimate Clinician Capability ($\kappa$) based on how well $R_\theta$ predicts the clinician's overrides.
2. M-Step: Fix the Capability Model ($\kappa$) and update the Reward Model ($R_\theta$) using capability-weighted preference likelihoods.
3. Outcome Anchoring: To avoid the system converging on collective clinician bias, the converged models are validated against longitudinal patient outcomes (e.g., hospitalization rates, HbA1c control) observed retrospectively.

D. The "Modify" Action Advantage

Unlike standard RLHF which uses binary pairwise comparisons, clinical overrides offer a three-valued response:

• Accept: Endorsement.
• Reject: Preference for an alternative (binary).
• Modify: The richest signal. It provides a preference pair plus a proximity signal (the alternative is close to the recommendation) and a gradient direction (how to improve the recommendation).

3. Key Contributions

1. Reframing Overrides: Shifting the paradigm from "overrides as noise/failure" to "overrides as structured preference data."
2. Capability-Conditioned Preference Learning: A novel formulation where the weight of a preference signal is a function of the annotator's capability ($\kappa$), preventing the suppression of correct-but-difficult recommendations.
3. Dual Learning Architecture: A formal solution to the coupled problem of learning a reward model and a capability model simultaneously, explicitly addressing Suppression Bias.
4. Context-Conditioned Rewards: Extending preference learning to include organizational contract structures ($c$), acknowledging that the "correct" action in Value-Based Care (VBC) differs from Fee-for-Service (FFS).
5. System Blueprint: A complete architecture for capturing, classifying, and learning from overrides, including scaffolding mechanisms to improve clinician capability based on the model's insights.

4. Results & Validation (Theoretical & Empirical Context)

• Suppression Bias Demonstration: The paper theoretically demonstrates that in a population where 70% of clinicians lack the capability to execute a guideline-recommended therapy, a naive model learns the therapy is suboptimal. The proposed $\kappa$-corrected model correctly identifies the therapy as optimal by down-weighting the low-capability overrides.
• Data Flywheel: In VBC chronic care, the system creates a compounding learning advantage:
  • Longitudinal Density: Thousands of interactions per clinician per quarter.
  • Outcome Observability: Unlike standard RLHF, clinical outcomes (e.g., hospitalization) provide ground truth to validate preferences.
  • Concentration: As the model improves, overrides concentrate on "blind spots" (high-value learning signals) rather than routine decisions.
• Operational Provenance: The framework emerged from operational work in a live VBC deployment, addressing real-world pathologies where recommendations disappeared because the population couldn't execute them.

5. Significance and Implications

• For Clinical AI:
  • Instrumentation is Critical: Systems must be designed from day one to capture structured override data (including the specific alternative chosen), not just binary accept/reject.
  • Scaffolding: The system can actively improve the workforce by identifying capability gaps and providing targeted support (e.g., checklists, protocols) to low-$\kappa$ clinicians, thereby increasing the quality of future preference signals.
• For RLHF Research:
  • Annotator Heterogeneity: The concept of weighting preferences by annotator capability ($\kappa$) is generalizable to code generation, legal review, and content moderation.
  • Context Matters: Reward functions must be conditioned on the operational context (e.g., contract type, platform policy), not just intrinsic quality.
• Policy & Economics:
  • The paper argues that Value-Based Care (VBC) with outcome-based incentives is a necessary condition for learning aligned reward models. Fee-for-Service (FFS) structures create misaligned incentives that corrupt the preference signal, regardless of the AI's sophistication.
  • AI developers and policymakers must align to create incentive geometries (like the CMS ACCESS model) that reward definitive, early interventions, enabling the AI to learn from high-quality, outcome-validated data.

Conclusion

The paper establishes that clinician overrides are not failures but the most valuable training signal in clinical AI, provided they are processed through a dual-learning framework that disentangles clinical truth (Reward) from human limitation (Capability). By leveraging longitudinal outcome data and capability-conditioned weighting, clinical AI can simultaneously improve its recommendations and the clinical workforce's ability to execute them, creating a compounding flywheel of improved patient outcomes.