Technical Summary: PHYSASSISTBENCH

Problem Statement

Current evaluations of Large Language Models (LLMs) in healthcare predominantly assess isolated capabilities: medical knowledge retrieval, Electronic Health Record (EHR) system interaction, or patient communication. However, the most plausible near-term deployment for medical LLMs is not as autonomous physicians, but as assistants operating under human oversight. This role requires the simultaneous coordination of three distinct modalities within a single interaction:

1. Implicit Physician Requests: Clinicians often issue elliptical, context-dependent, or multi-turn queries (e.g., "Given all this, switch the med?") rather than fully specified instructions.
2. Ambiguous Patient Information: Patients provide colloquial, incomplete, and clinically imprecise descriptions of symptoms.
3. Precise EHR Tool Use: The system must execute exact FHIR-based tool calls to retrieve or update structured records.

Existing benchmarks fail to capture this integrated setting. They typically treat these roles in isolation (e.g., static Q&A or single-turn tool use) and do not evaluate the model's ability to coordinate knowledge, communication, and system actions across evolving, multi-turn clinical scenarios.

Methodology

PHYSASSISTBENCH Benchmark

The authors introduce PHYSASSISTBENCH, a benchmark designed to evaluate LLMs as physician assistants in an interactive doctor-patient-EHR environment.

• Data Source: Built from real, de-identified patient records from MIMIC-IV.
• Scale: 324 multi-turn sessions (1,296 turns total), evaluated in both English and Chinese.
• Structure: Each session simulates a single clinical encounter with four turns (Turn 0–Turn 3).
  • Turn 0: Always an Information Lookup (IL) to anchor the session in concrete data.
  • Turns 1–3: Progress through Data Gathering (DG), Clinical Reasoning (CR), or Write/Update (WU) tasks.
• Task Types:
  • Information Lookup (IL): Retrieving a single fact via EHR.
  • Data Gathering (DG): Collecting info from multiple sources (EHR + Patient Interview).
  • Clinical Reasoning (CR): Synthesizing findings to form recommendations.
  • Write/Update (WU): Executing FHIR write operations (e.g., prescriptions).
• Implicitness Subtypes: The benchmark specifically tests the model's ability to handle underspecified queries via three linguistic mechanisms:
  • Nominal Anaphora (NA): Pronouns referring to prior entities.
  • Predicate Ellipsis (PE): Omission of verbs/arguments (e.g., "Creatinine?" instead of "What is the creatinine?").
  • Abstract Event Anaphora (AE): References to accumulated context (e.g., "Given all this...").
• Evaluation Metrics:
  • Rubric Score: Turn-level scoring based on factual accuracy, clinical interpretation, and absence of hallucinations.
  • Pass@Turn: Fraction of turns passing a threshold.
  • Pass@Session: Fraction of sessions where all turns pass the threshold (a stricter metric reflecting the multi-turn dependency of clinical practice).

Agentic Patient Environment Pipeline

To generate scalable, clinically grounded scenarios from static MIMIC-IV records, the authors developed a multi-agent synthetic data pipeline:

1. Patient Pre-filtering: Offline checks ensure records contain sufficient evidence (e.g., specific lab trends or medication pairs) to support the target scenario before LLM involvement.
2. Session Planning: An agent constructs a coherent four-turn clinical arc, mapping static data to a narrative involving specific FHIR tools and patient interview questions.
3. Turn-Level Generation: A multi-agent system (User, Planner, Patient, Answer, Checker agents) generates the dialogue.
   • Patient Agent: Simulates a patient with specific health literacy and adherence profiles, responding based strictly on the underlying EHR evidence (no counterfactual hallucinations).
   • Tool Execution: Real FHIR queries are executed against the MIMIC-IV data to ensure factual grounding.
   • Quality Gates: Three-stage validation (Plan, Observation, Answer) filters out hallucinations, structural errors, and unsafe content.

Key Contributions

1. Formulation of Interactive Assistance: The paper defines physician assistance as a coordination problem across clinical knowledge, patient communication, and EHR systems, a setting overlooked by existing evaluations of isolated roles.
2. Scalable Agentic Pipeline: A novel method for transforming static EHR records into interactive, record-grounded "agentic patients," enabling multi-turn scenarios without compromising clinical factuality.
3. PHYSASSISTBENCH Release: A manually reviewed and physician-validated benchmark of 324 sessions and 1,296 turns, demonstrating that current leading LLMs are not yet reliable physician assistants.

Experimental Results

The authors evaluated 14 state-of-the-art LLMs (5 closed-source, 9 open-weight) on PHYSASSISTBENCH.

• Overall Performance: Even the strongest models (e.g., GLM-5, Claude-Opus-4.7, GPT-5.4-high) show significant limitations. The best model (GLM-5) achieved a Mean Rubric Score (mRS) of ~69.4% (English), but Pass@Session (strict consistency) was only 21.3% at a 0.60 threshold and 6.5% at 0.75.
• The Bottleneck of Coordination: Performance drops significantly when tasks require integrating multiple sources.
  • Information Lookup is robust (~82.6%).
  • Data Gathering and Clinical Reasoning are consistent weak points (~47.5% and ~54.9% respectively), indicating that multi-tool composition and reasoning are harder than single-point retrieval.
• Impact of Implicit Queries: Models suffer a measurable penalty when queries are implicit.
  • Abstract Event Anaphora (AE) caused the largest performance drop (average -5.3% compared to explicit queries).
  • Smaller models (e.g., Qwen3.5-27B) showed extreme fragility to implicitness (up to -9.8% drop), while stronger models remained more stable.
• Patient Communication Gap: All models performed substantially worse on Patient-Interview turns compared to EHR-only turns (average drop of 26.4 points). This suggests that handling ambiguous, colloquial patient input is a distinct capability bottleneck, independent of general model capacity.
• Language Bias: Non-Chinese models performed surprisingly better on Chinese sessions (up to +11.4% Pass@Session), attributed to "language-conditioned safety priors" where English training data suppresses EHR tool calls due to safety disclaimers.

Significance and Claims

The paper claims that reliable clinical assistance requires coordination across knowledge, communication, and systems, not isolated gains in any of them.

• Current State: Leading LLMs are not yet reliable physician assistants. They struggle to resolve implicit intent, handle patient ambiguity, and integrate evidence across sources in a single workflow.
• Evaluation Paradigm: The authors argue that current benchmarks focusing on static Q&A or single-turn tool use are insufficient. The "interaction" dimension—specifically the ability to maintain context, resolve ellipsis, and coordinate tools over multiple turns—is the key bottleneck for clinical LLM deployment.
• Future Direction: The benchmark serves as a resource to stress-test multi-turn reasoning and structured tool use, highlighting that future progress must focus on agentic coordination rather than just expanding parametric medical knowledge.

The authors explicitly state that the benchmark is for research evaluation only and is not intended for clinical deployment or autonomous decision-making. They acknowledge limitations in scenario coverage (focusing on four specific clinical workflows) and the reliance on MIMIC-IV data, which may not generalize to all healthcare systems.