Technical Summary: HealthAgentBench

Problem Statement

The field of AI is transitioning from static Large Language Models (LLMs) that answer isolated prompts to agentic systems capable of executing actions within complex, interactive environments. While traditional healthcare benchmarks (e.g., MedQA) evaluate static question-answering on concise clinical vignettes, they fail to capture the requirements of real-world clinical workflows. These workflows often involve processing multimodal data (3D CT volumes, gigapixel pathology slides, longitudinal EHRs) that exceed context windows and require multi-step interactions with tools and databases. Existing agent-oriented benchmarks are often fragmented, limited to specific modalities or predefined workflows, or rely on simulated dialogue rather than direct interaction with raw data. Consequently, there is a lack of a unified, realistic, and challenging benchmark suite to evaluate the end-to-end autonomous capabilities of frontier AI agents in healthcare.

Methodology

The authors introduce HealthAgentBench, a unified suite of 54 agentic healthcare tasks across 7 distinct categories, each operating within a unique, containerized terminal-based environment.

Task Design and Creation

The benchmark was constructed using a principled four-stage workflow:

1. Selection Criteria: Tasks were selected to require agentic workflows (planning, tool use, multi-step reasoning) that naive single-shot prompting cannot solve. They cover realistic clinical stages (data management, diagnosis, research, treatment planning) and diverse modalities (2D radiographs, 3D CT, whole-slide pathology, free text, structured EHR).
2. Sourcing: Tasks were derived from existing benchmarks (e.g., EHRSHOT) re-engineered for autonomous execution, or created by injecting realistic errors into curated datasets (e.g., MIMIC-IV) or wrapping clinical pipelines.
3. Design Principles:
   • Terminal Environments: Tasks are packaged as Docker containers where agents interact via a command line, allowing them to inspect raw data, install dependencies, and query databases.
   • Minimal Instructions: Agents receive only high-level goals, forcing them to formulate their own strategies for file exploration and tool usage.
   • Anti-Cheat: Gold labels and verifier code are hidden from the agent container. Identifiers are obfuscated, and web browsing is disabled to prevent lookup of answers.
   • Verifiability: Success is defined by binary criteria (success/failure) against expert-derived labels or human performance baselines, with a low chance success rate (typically <10%) to ensure performance reflects capability rather than luck.
4. Post-Creation Checks: Tasks undergo manual and LLM-assisted review to eliminate unintended shortcuts, ambiguities, or trivial solutions.

The 7 Task Categories

1. X-ray Report Correction: Agents correct clinically significant errors in a draft radiology report by cross-referencing longitudinal chest X-ray images and prior studies.
2. Pathology Tumor Area Selection: Agents identify tumor regions in gigapixel whole-slide pathology images by selecting grid tiles, requiring multi-resolution reasoning.
3. CT Abnormality Classification: Agents classify the presence/absence of multiple abnormalities in 3D non-contrast chest CT volumes.
4. Clinical Trial Matching: Agents match patient admission notes to eligibility criteria in a corpus of ~400 free-text clinical trial protocols.
5. EHR Data Quality Auditing: Agents identify and flag injected data quality errors (impossible values, demographic conflicts) within a large relational EHR dataset (>800k rows).
6. EHR Event Modelling: Agents build a machine learning pipeline from longitudinal EHR data to predict clinical events, aiming to match or exceed human-engineered baselines.
7. EHR Format Conversion: Agents customize an ETL pipeline to transform raw EHR data into a standardized format (MEDS) with specific schema requirements.

Evaluation Protocol

The benchmark evaluates 10 frontier agents (from GPT-5 and Claude families) across three execution harnesses (Codex, Claude Code, Copilot-CLI). Agents are given a 1-hour time budget per task. Performance is measured by the overall task success rate (fraction of tasks successfully resolved) and cost-efficiency metrics.

Key Results

• Overall Difficulty: The suite remains highly challenging for current frontier agents. The strongest agent, Codex GPT-5.5, achieved an overall task success rate of only 42%. The next best agents (Copilot CLI with Opus-4.8 and GPT-5.5) scored 36% and 35%, respectively.
• Model and Harness Differences:
  • Codex vs. Claude: Codex GPT models generally outperformed Claude Code models, particularly in medical imaging tasks. Codex GPT-5.5 averaged ~35% on imaging tasks, while Claude Code agents averaged ~12%.
  • Cost-Efficiency: The Pareto frontier for cost-efficiency was traced entirely by GPT-5 agents. Claude Code agents were often more expensive and slower (due to verbose outputs and spawning sub-processes) while achieving lower success rates.
• Category-Specific Performance:
  • Strengths: Agents showed promise in EHR Event Modelling and EHR Format Conversion, where several agents achieved near-perfect success rates (up to 100% for format conversion, ~78% for event modelling), indicating capability in building ML pipelines over tabular data.
  • Weaknesses: Medical Imaging (CT, X-ray, Pathology) remained a significant bottleneck, with mean success rates as low as 17% across agents. EHR Data Quality Auditing and Clinical Trial Matching also proved difficult due to large search spaces and complex compositional reasoning; no agent exceeded 50% success on Data Quality Auditing.
• Search Space and Compositionality: Performance dropped significantly when agents had to navigate large search spaces without hints or detect multiple error types simultaneously, highlighting these as major bottlenecks.

Significance and Claims

The paper claims that HealthAgentBench provides a necessary shift from static evaluation to realistic, executable benchmarks that reflect the complexity of clinical workflows. Its significance lies in:

1. Unified Evaluation: It offers a single framework to assess agents across diverse modalities and clinical stages, filling a gap in the current fragmented landscape.
2. Realistic Challenge: The benchmark is far from saturated, revealing that even the most advanced agents struggle with end-to-end clinical reasoning, particularly in imaging and complex data retrieval.
3. Diagnostic Insight: By breaking down performance, the benchmark identifies specific failure modes (e.g., difficulty with gigapixel image analysis, large search space navigation) that guide future research directions.
4. Community Resource: The authors release the benchmark suite to enable the community to track progress, compare systems, and identify areas for improvement in agentic healthcare AI.

The authors conclude that while frontier agents show emerging capabilities in specific areas like automated research pipelines, substantial progress is still required to achieve clinically useful performance, particularly in handling multimodal data and complex, multi-step reasoning tasks.