The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing

Technical Summary: The Clinician's Veto

Problem Statement
The paper addresses the emerging regulatory and ethical challenge of transitioning autonomous AI systems from advisory roles to fully autonomous agents capable of prescribing medication without immediate human review. Recent policy developments, such as the proposed U.S. bill H.R. 238 and Utah's prescription-renewal pilot program, signal a shift toward granting AI "agentic" prescribing authority. However, current regulatory frameworks and technical architectures lack consensus on the conditions required for safe deployment. The authors identify a critical gap: existing guidelines focus on aggregate model performance metrics at the point of clearance but fail to address per-prediction confidence, the differentiation of uncertainty types, or the mechanisms required for liability allocation. Without these, autonomous prescribing risks creating an accountability vacuum where clinicians bear liability for outcomes they cannot control or verify, potentially leading to automation bias and catastrophic safety failures.

Methodology
To empirically test the necessity of specific technical architectures for safe autonomous prescribing, the authors conducted a survey of 136 U.S. prescribing clinicians (physicians, nurse practitioners, and physician assistants). The study was structured around three core hypotheses derived from technical and ethical arguments:

1. H1 (Calibrated Action Gating): Whether clinicians view calibrated confidence thresholds as a prerequisite for safety, specifically requiring the AI to abstain and escalate low-confidence decisions for human review.
2. H2 (Differentiated Uncertainty Communication): Whether clinicians distinguish between epistemic uncertainty (model ignorance due to sparse/biased data) and aleatoric uncertainty (irreducible clinical ambiguity) and if their preferred interface response changes based on the source.
3. H3 (Inferential Transparency and Liability): Whether liability allocation shifts based on the presence of inferential transparency (per-prediction reasoning) and whether clinicians accept personal liability only when they can substantively review the AI's decision under acknowledged uncertainty.

The survey utilized vignettes and Likert-scale questions to measure willingness to permit autonomous prescribing, perceived patient safety, preferred interface content during escalation, and responsibility allocation across five parties (AI company, regulatory body, health system, clinician, and patient) in adverse outcome scenarios.

Key Contributions and Technical Requirements
The paper argues that three architectural features constitute the minimum technical requirements for the safe and ethical deployment of autonomous AI prescribing. The authors posit that satisfying these conditions effectively "collapses" the conventional definition of autonomy, transforming the system from an independent agent into a heavily supervised decision-support tool.

1. Calibrated, Action-Gated Confidence Thresholds:
   The system must utilize calibrated confidence scores where high probability corresponds to high accuracy. Crucially, the system must possess an "action-gating" mechanism: if confidence falls below a defined threshold, the agent must abstain from prescribing and escalate the case to a clinician. The authors argue that in fully autonomous settings, where the clinician is structurally absent from the initial decision, the model's internal confidence estimate is the sole safety infrastructure. Mis-calibrated models that fail to abstain when uncertain pose a direct safety hazard.

2. Differentiated Communication of Uncertainty:
   Upon escalation, the system must distinguish between the source of uncertainty.

   • Aleatoric Uncertainty (Clinical Ambiguity): When the case is inherently ambiguous but the model has seen similar data, the system should present a "competing-options summary" to aid clinician judgment.
   • Epistemic Uncertainty (Model Ignorance): When the model lacks validated training data for the case, the system should abstain from offering a preferred recommendation entirely. Displaying a recommendation in this state risks inducing automation bias, as the clinician may defer to a suggestion the model cannot validate.

3. Inferential Transparency:
   Distinct from aggregate "documentation transparency" (e.g., model cards released pre-deployment), the system must provide "inferential transparency" at the moment of decision. This involves per-prediction traceability, exposing the reasoning and data provenance for every specific output. This is necessary for real-time auditing and for determining liability in adverse events, ensuring that responsibility is not unfairly defaulted to the clinician when the system acted outside its validated bounds.

Results
The survey of 136 clinicians yielded the following findings supporting the proposed hypotheses:

• Support for H1: 84.4% of clinicians indicated they would not permit autonomous prescribing without a calibrated confidence-based escalation mechanism. They rated calibrated agents as substantially safer (mean 3.71) than uncalibrated agents (mean 2.06) relative to a baseline with no escalation. There was no consensus on who should set the threshold, though organizational bodies (regulatory, health systems) were preferred over AI companies.
• Support for H2: Clinicians significantly differentiated their preferences based on uncertainty type. In aleatoric scenarios, 71.0% preferred a competing-options summary. In epistemic scenarios, preferences shifted significantly toward abstention (46.9%) and away from competing options (36.9%), aligning with the need to prevent automation bias when the model lacks a validated basis.
• Support for H3: In adverse outcome scenarios, clinicians consistently assigned higher responsibility to organizational actors (AI companies, regulatory bodies, health systems) than to individuals (clinicians, patients). However, when the AI provided inferential transparency (communicated confidence and escalated for review), clinicians were willing to accept more personal responsibility. Conversely, in non-escalated scenarios, clinicians assigned the least responsibility to patients, even when consent was documented, rejecting the notion that consent transfers meaningful accountability.

Significance and Claims
The paper claims that the proposed architectural requirements are not merely optional enhancements but necessary preconditions for the ethical deployment of autonomous AI in prescribing. The authors argue that these requirements fundamentally constrain the degree of autonomy that can be ethically granted. A system adhering to these standards functions less as an autonomous agent and more as a "heavily supervised decision-support tool."

The significance of this work lies in providing a technical and clinician-backed framework to guide legislation (such as H.R. 238) and regulatory policy. The authors contend that without these constraints, the current trajectory of autonomous prescribing risks creating a "foreseeable accountability pattern" where clinicians bear liability for AI actions they cannot verify. By aligning liability with the institutional actors who control system design and deployment, and by mandating technical features that enable this alignment, the proposed framework offers a pathway to safer, more adoptable autonomous prescribing systems. The paper concludes that while baseline support for fully autonomous prescribing remains low, these technical constraints are essential for any future adoption to be considered safe and just.