MedCTA: A Benchmark for Clinical Tool Agents

Technical Summary: MedCTA: A Benchmark for Clinical Tool Agents

Problem Statement
Current medical AI agents are expected to move beyond simple recognition tasks to perform clinically grounded decision-making, which requires tool retrieval, evidence acquisition, and integration. Existing benchmarks primarily evaluate isolated perception or single-turn question answering, offering limited insight into failures in planning, tool recruitment, and rollout reliability. In clinical settings, a final answer is insufficient; users require process-level accountability regarding the evidence used and the reasoning path taken. The paper identifies a critical gap: while strong backbone perception models exist, their ability to execute reliable, multi-step agentic behavior in clinical workflows remains untested and likely brittle.

Methodology
The authors introduce MedCTA, a trajectory-aware benchmark and executable evaluation platform for medical tool agents. The methodology involves three core components:

1. Dataset Construction: MedCTA comprises 107 real-world clinical tasks grounded in authentic multimodal inputs (radiology images, pathology slides, reports, etc.). The construction pipeline is semi-automated and human-in-the-loop:

   • Query Construction: Seed queries from existing VQA datasets are transformed into "step-implicit" clinical queries. Large Multimodal Models (LMMs) draft agentic reformulations, which are then rewritten by human annotators to remove procedural hints (tool leakage) and validated by clinicians for medical correctness. The final queries state a clinical goal without revealing the necessary tools or execution order.
   • Tool Chain Construction: Reference trajectories are generated using LMMs and refined through technical annotation (ensuring executability, minimality, and schema compliance) and clinical verification (ensuring medical soundness and workflow alignment).
   • Tools: The benchmark utilizes a library of 5 deployed tools: OCR, ImageDescription, RegionAttributeDescription, GoogleSearch, and Calculator.

2. Evaluation Framework: MedCTA employs a process-aware evaluation suite with three metric groups:

   • Step-by-Step Mode: Measures interaction fidelity, including Instruction Accuracy (InstAcc), Tool Accuracy (ToolAcc), Argument Accuracy (ArgAcc), and Summary Accuracy (SummAcc).
   • Clinical Reasoning Mode: Assesses trajectory-level coherence via Clinical Faithfulness (Facc), Context Integration Score (Cs), and Semantic Completeness (Scomp).
   • Outcome Mode: Measures final task success via Goal Accuracy (Gacc).

3. Experimental Setup: The authors benchmark 18 open- and closed-source multimodal models (including GPT-5.4, Claude, Gemini, Qwen, Llama, etc.) using the OpenCompass evaluation harness and Lagent agent framework. They compare autonomous rollouts against "gold-standard" routing (where the correct tool sequence is provided) to isolate controller failures from reasoning capabilities.

Key Contributions

• MedCTA Benchmark: The first agentic benchmark specifically designed to evaluate tool-augmented LLMs on realistic clinical tasks with executable tool chains and trajectory-level reasoning. It features clinician-validated, step-implicit queries and reference trajectories.
• Process-Aware Metrics: Definition of fine-grained metrics that disentangle controller competence (tool selection, argument validity) from evidence-conditioned reasoning (faithfulness, completeness).
• Diagnostic Analysis: A comprehensive failure analysis demonstrating that current models suffer from significant "controller gaps," where autonomous tool selection fails despite the model possessing the underlying clinical knowledge.

Results

• Low Autonomous Performance: Across 18 models, the best outcome accuracy (Gacc) in the autonomous setting is only 31.54%. No model achieves non-zero strict trajectory success under rigorous rollout diagnostics.
• Controller Gap: When provided with gold-standard tool routing (correct tool sequence), diagnostic answer accuracy improves significantly (e.g., Claude-opus-4-6 rises from 31.32% to 66.40%; GPT-5.4 from 31.54% to 49.50%). This indicates that performance degradation is primarily due to controller failures (protocol instability, premature stopping, incorrect tool recruitment) rather than a lack of clinical reasoning ability.
• Failure Modes: Autonomous rollouts are dominated by API/protocol errors (64.2% of episodes) and under-calling (99.2% relative to reference trajectories). First failures typically occur at protocol handling (58.3%) or tool selection (41.6%), before clinical reasoning can fully engage.
• Perception vs. Agency: Backbone-only VLMs achieve substantially higher zero-shot performance (e.g., GPT-5.4 at 60.74%) when tool interaction is removed, confirming that recognizing medical content is easier than planning multi-step clinical workflows.
• Reasoning Correlation: Clinical reasoning metrics, particularly Semantic Completeness (Scomp) and Clinical Faithfulness (Facc), show stronger correlations with final accuracy than step-by-step execution metrics, highlighting that reasoning quality drives performance, but controller instability prevents models from reaching stable reasoning states.

Significance
The paper positions MedCTA not merely as a leaderboard benchmark but as a diagnostic instrument for auditing and improving clinically grounded agents. It demonstrates that strong backbone perception does not translate into reliable agentic behavior in clinical settings. The findings highlight that the primary bottleneck for medical AI agents is not raw perception or medical knowledge, but the reliability of the controller in managing multi-step tool interactions, protocol adherence, and evidence grounding. MedCTA provides a rigorous testbed for advancing trustworthy medical AI agents by exposing specific failure modes in tool use and reasoning trajectories.