The Doctor Will (Still) See You Now: On the Structural Limits of Agentic AI in Healthcare

1. Problem Statement

The paper addresses a critical misalignment between the commercial narrative of "agentic AI" in healthcare and the operational reality of its deployment.

• The Promise: Industry discourse promotes agentic AI as systems capable of autonomous, goal-directed behavior, multi-step reasoning, and independent action to solve healthcare challenges (e.g., reducing administrative burden, augmenting diagnosis). The market is projected to reach $199 billion by 2034.
• The Reality: In high-stakes clinical environments, these systems operate under near-total human oversight due to safety, regulatory, and liability constraints.
• The Core Conflict: There is a "definitional crisis" where the term "agentic" is contested. Developers define it by technical capabilities (planning, tool use), while clinicians define it by operational constraints (bounded automation). This ambiguity creates an accountability vacuum: when systems fail, responsibility is diffused because stakeholders cannot agree on what the system is supposed to do or who is responsible for its actions. Furthermore, current evaluation frameworks prioritize technical benchmarks (accuracy) over sociotechnical safety (workflow integration, trust, temporal consistency).

2. Methodology

The authors employed a qualitative grounded theory approach to investigate how agentic AI is defined, evaluated, and constrained across the healthcare ecosystem.

• Data Collection: Semi-structured interviews were conducted with 20 stakeholders in the US between August and December 2025.
• Participant Demographics: The sample included three distinct groups to cover the end-to-end pipeline:
  • Technical Developers: AI/ML researchers, engineers, and startup founders.
  • Clinical Implementers: Physician-developers, clinical data scientists, and informatics fellows.
  • Healthcare End Users: Clinicians, nurses, and medical students interacting with AI tools.
• Analysis: Transcripts (approx. 173,000 words) were analyzed using open and axial coding. The process identified themes regarding definitions, autonomy, trust, evaluation metrics, and accountability.
• Goal: To move beyond technical benchmarks and understand the institutional, regulatory, and social factors shaping the deployment of agentic AI.

3. Key Contributions

The paper makes three primary contributions to the fields of HCI, AI safety, and healthcare informatics:

1. Identification of Structural Tensions: It articulates three mutually reinforcing tensions that limit agentic AI:
   • Conceptual Fragmentation: No shared definition of "agentic" exists across disciplines.
   • Autonomy Contradiction: Commercial promises of autonomy exceed operational reality (systems are "agents" in marketing but "assistants" in practice).
   • Evaluation Blind Spots: Current metrics fail to assess safety-critical dimensions like trust calibration, workflow integration, and temporal consistency.
2. Reframing Accountability: The paper argues that accountability is not just a legal issue but a sociotechnical problem rooted in definitional ambiguity. When "agency" is ill-defined, liability cannot be assigned, leading to a "distributed lack of accountability."
3. Proposal for Deployment-Centered Evaluation: The authors advocate for shifting evaluation paradigms from capability-centered (benchmark accuracy) to deployment-centered (real-world safety, workflow fit, and temporal behavior).

4. Key Results & Findings

A. Conceptual Fragmentation & The Definitional Crisis

• Moving Target: Participants described "agentic AI" as a marketing buzzword. Developers view it as systems with planning and tool-use capabilities; clinicians view it as strictly bounded workflow automation.
• Spectrum vs. Binary: Some stakeholders view agency as a spectrum (from passive documentation to interactive reasoning), while others reject the label entirely for current systems, noting they are rarely capable of chaining multiple tasks toward an outcome.
• Consequence: This instability allows stakeholders to "locate responsibility elsewhere." If a system fails, developers claim it was just a tool, while deployers claim the system was promised to be autonomous.

B. The Autonomy Contradiction

• 0% Autonomy in Practice: Despite marketing claims, nearly all deployed systems require a "human-in-the-loop." Participants noted that society lacks the framework to allow AI to operate autonomously in high-stakes environments where errors can be fatal.
• Risk Propagation: Even in low-stakes tasks (e.g., documentation), lack of oversight is dangerous because errors propagate into permanent records, affecting long-term patient care.
• The "Promissory" Gap: Systems are governed based on anticipated future capabilities rather than current operational reality.

C. Evaluation Gaps & Trust Barriers

• Metric Misalignment: 83% of current evaluations focus on technical correctness. Only 5% assess longitudinal behavior, and few assess safety/governance.
• The "Portability Trap": High accuracy on static benchmarks (e.g., MedQA) does not translate to clinical success because benchmarks ignore infrastructure dependencies (EHR integration) and the "human-in-the-loop" labor required to catch errors.
• Trust vs. Trustworthiness: Clinicians distinguish between trust (psychological reliance) and trustworthiness (system reliability). Trust is eroded by:
  • Hallucinations: Even with temperature set to zero, omissions or fabrications (e.g., wrong medication dosage) are unacceptable.
  • Inconsistency: Multi-agent systems often produce different outputs for the same input, destroying reproducibility.
  • Black Box Reasoning: Clinicians need visibility into the "chain of reasoning" to accept recommendations, especially when they remain legally liable for outcomes.

D. Operational Constraints & Viable Use Cases

• Workflow Integration: Systems fail when they cannot navigate the relational work of healthcare (e.g., ordering a wheelchair involves nurses, doctors, insurance, and multiple systems).
• Temporal Mismatch: Current benchmarks are static (single-turn Q&A), but clinical care is dynamic and longitudinal. Systems must handle timescales ranging from instantaneous triage to multi-year patient management.
• Where it Works: Agentic AI is currently viable only in low-stakes, well-defined workflows (e.g., prior authorization, billing, ambient documentation) where human oversight is preserved.
• Where it Fails: Autonomous diagnosis, complex multi-step reasoning, and real-time acute intervention remain out of reach due to liability and safety constraints.

5. Significance and Implications

• For Policy & Regulation: Regulatory bodies (e.g., FDA, EMA) must require explicit disclosure of the autonomy level at which a system is designed versus marketed. Liability frameworks must account for the gap between "promised" and "deployed" capabilities.
• For AI Design: Developers must move beyond optimizing for benchmark scores. Systems should be architected for workflow integration, interpretability, and risk-based deployment (starting with low-stakes tasks).
• For Evaluation: The field must adopt deployment-centered evaluation that measures:
  • Trust calibration (does the system know when it is unsure?).
  • Temporal consistency (does the system behave reliably over time?).
  • Sociotechnical fit (does it work within existing clinical workflows?).
• Broader Impact: The paper warns that without reconciling these divergent framings, the "agentic" label will continue to promise autonomy while systems remain constrained, leaving patient safety vulnerable and accountability ambiguous when failures occur.

In conclusion, the paper argues that the path forward for healthcare AI is not faster autonomy, but better governance of the human-AI relationship, grounded in realistic definitions, rigorous sociotechnical evaluation, and clear accountability structures.