💻 computer science

ABRA: Agent Benchmark for Radiology Applications

The paper introduces ABRA, a novel radiology-agent benchmark featuring 655 programmatically generated tasks within a realistic DICOM environment that reveals current models' strong tool orchestration capabilities but significant limitations in medical image perception and finding localization.

Original authors: Bulat Maksudov, Vladislav Kurenkov, Kathleen M. Curran, Alessandra Mileo

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bulat Maksudov, Vladislav Kurenkov, Kathleen M. Curran, Alessandra Mileo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a new assistant to help a radiologist read X-rays and CT scans. In the past, to test this assistant, you would hand them a stack of printed photos and a list of questions like, "Is there a spot on this photo?"

The paper ABRA (Agent Benchmark for Radiology Applications) argues that this old way of testing is flawed. It's like testing a pilot by asking them to describe a plane on a poster, rather than letting them sit in the cockpit and actually fly the controls.

Here is what the researchers built and discovered, explained simply:

1. The New Test: A "Flight Simulator" for AI

Instead of giving the AI static pictures, ABRA puts the AI inside a live, working medical imaging system.

The Environment: Think of it as a digital radiology room. The AI has to use a mouse and keyboard (via software tools) to open a patient's file, scroll through hundreds of 3D slices of a body, adjust the brightness and contrast (windowing), zoom in, and draw circles around problems.
The Tools: The AI has a toolbox of 21 specific commands. It can't just "see" the whole image at once; it has to actively choose to look at a specific slice or zoom in on a specific area, just like a human doctor does.

2. The Challenge: 655 Different Missions

The researchers created 655 different tasks for the AI to solve, ranging from easy to very hard:

Easy: "Go to slice number 50" or "What is the patient's name?"
Medium: "Find the lung nodule and draw a circle around it."
Hard: "Compare this patient's scan from last year to today and tell me if a new spot has appeared."

They tested 10 different AI models (some from big tech companies, some open-source) on these missions.

3. The Big Discovery: The "Eyes" vs. The "Hands"

The most surprising finding is that the AI models are great at using the tools but terrible at seeing the details.

The Hands (Tool Orchestration): The AI is excellent at following instructions. If you tell it, "Open the file, scroll to slice 50, and draw a circle," it does this perfectly. It knows exactly which buttons to click and in what order.
The Eyes (Visual Perception): When the AI has to actually look at the blurry, gray pixels of a real CT scan to find a tumor, it fails miserably.
- The Proof: The researchers ran a special test. In one version, they gave the AI the answer beforehand (like a cheat sheet saying, "The tumor is here"). In this version, the AI got a 90–100% score because it just had to copy the answer.
- In the real version, where the AI had to find the tumor itself, the score dropped to 0–25%.

The Metaphor: Imagine a very smart student who can perfectly operate a microscope and adjust the focus knobs (the tools). However, when you ask them to look through the lens and identify a specific bacteria, they can't see it at all. They are great at the mechanics of the job but bad at the vision part.

4. What This Means (According to the Paper)

The paper concludes that for AI to be useful in radiology right now, it shouldn't try to be the "doctor" who sees everything. Instead, the best setup is a team:

A specialized "vision" AI that is really good at spotting the spots on the image.
A "workflow" AI (like the ones tested in ABRA) that is really good at using the tools, organizing the data, and writing the report based on what the vision AI found.

Summary

ABRA is a new, tougher test that forces AI to actually use a medical viewer rather than just looking at a picture. It shows that current AI models are like skilled pilots who can fly the plane but have terrible eyesight. They can navigate the controls perfectly, but they struggle to actually see the obstacles in the sky. The paper suggests that until AI gets better at "seeing" the pixels, it needs help from other specialized tools to do real medical work.

Technical Summary: ABRA – Agent Benchmark for Radiology Applications

Problem Statement

Existing benchmarks for medical agents typically deliver imaging data as pre-selected, static samples (e.g., a single image or a concatenated patient record). This approach fails to capture the reality of radiology workflows, where the agent must navigate a complex, interactive environment to locate, inspect, and annotate specific findings. Current medical agent benchmarks often treat imaging as a fixed input rather than a queryable environment, thereby decoupling tool orchestration from visual perception. Furthermore, prior radiology-specific benchmarks, such as RadABench, utilize symbolic simulators where imaging inputs are replaced by placeholder tokens to upper-bound planning capabilities, explicitly excluding visual perception from the evaluation scope.

There is a critical need for a benchmark that places an agent inside a live radiology workstation, requiring it to perform tool-based navigation, windowing, and annotation on actual DICOM data, while simultaneously measuring its ability to orchestrate these tools and perceive visual information.

Methodology

The ABRA Environment

ABRA constructs an executable radiology workstation stack comprising:

OHIF Viewer: An open-source radiology viewer (v3.9.0) extended with a custom interface to expose internal state.
Orthanc PACS: A DICOM server acting as the Picture Archiving and Communication System, hosting the imaging data.
Controller Bridge: A headless browser driver (Puppeteer) and a Python preprocessing sidecar that manage the interaction between the agent and the viewer.

The environment is exposed to agents via a uniform function-calling interface consisting of 21 tools. These tools are partitioned into:

Observation Categories: Metadata queries (DICOM tags), viewer screenshots, DICOM pixel retrieval (via six distinct preprocessing pipelines for different modalities), and oracle predictions (simulated external CAD models).
Action Classes: Navigation/display (slice scrolling, window/level adjustment, zoom, series switching), segmentation (placing circles, bounding boxes, and polygons in pixel coordinates), and reporting (structured BI-RADS reports, longitudinal findings, or free-text answers).

Task Generation and Benchmark Structure

The benchmark contains 655 programmatically generated tasks derived from three public TCIA datasets:

LIDC-IDRI: Thoracic CT with multi-radiologist consensus nodules.
Duke Breast Cancer MRI: Multi-sequence breast MRI with BI-RADS labels.
NLST New-Lesion LongCT: Baseline and follow-up CT pairs with new-lesion annotations.

Tasks are stratified into three difficulty tiers (Easy, Medium, Hard) and eight task types:

Easy: Viewer control, metadata QA, and vision probes (modality recognition).
Medium: Annotation (perceiving pathology and contouring) and Oracle Annotation (acting on simulated detector findings).
Hard: Longitudinal comparison (detecting new lesions across time) and BI-RADS reporting (structured reporting on full MRI studies).

A key design feature is the paired oracle/real variant for perception-heavy tasks. In the "Oracle" variant, the agent is given structured findings from a simulated detector and must only orchestrate tools to report them. In the "Real" variant, the agent must access pixel-level tools to perceive the findings itself. This design isolates visual perception from tool orchestration.

Evaluation Framework

Adopting the framework proposed by Bluethgen et al. [2025], ABRA scores episodes along three dimensions using automatic scorers:

Planning (20% weight): Compares the agent's tool-call trajectory against a programmatic reference trajectory (unordered multiset matching) to assess workflow efficiency.
Execution (30% weight): Evaluates the success of tool calls, parameter quality, turn efficiency, and error recovery.
Outcome (50% weight): Measures task success based on the final deliverable (e.g., Intersection-over-Union for segmentation, exact match for metadata, or field agreement for BI-RADS reports).

The composite score is calculated as $S = 0.20P + 0.30E + 0.50O$ .

Key Results

The authors evaluated ten current models (five closed-weight and five open-weight) on ABRA:

Tool Orchestration vs. Perception: Models demonstrated high competence in tool orchestration. Execution scores were consistently near-ceiling (approx. 0.90–1.00) across all tiers and models, indicating that agents can successfully navigate the viewer, query metadata, and invoke submission actions.
The Perception Bottleneck: A significant performance gap was observed between Oracle and Real variants.
- On Oracle Annotation and Oracle BI-RADS tasks (where perception is bypassed), models achieved Outcome scores ranging from 0.69 to 1.00.
- On Real Annotation tasks (requiring visual perception of DICOM pixels), Outcome scores collapsed to 0.00–0.25 across the roster.
- Similarly, on Hard-tier tasks, Longitudinal New-Lesion detection (requiring visual localization) resulted in near-zero scores, whereas BI-RADS reporting (which allows some prior-based guessing) achieved moderate scores (0.06–0.64).
Model Performance: No single model achieved high performance on the Real variants. While some models (e.g., Gemini 3 Flash) showed slightly better performance on specific nodule detection tasks, the general trend indicates that current frontier models are "tool-competent but perception-limited" when operating on real DICOM data.

Significance and Claims

The paper positions ABRA not as a verdict on the clinical readiness of current AI agents, but as a measurement substrate for the specific gap between tool orchestration and visual perception in radiology.

First Live Viewer Benchmark: ABRA is claimed to be the first medical agent benchmark to place the agent inside a live clinical viewer (OHIF) rather than a static dataset or symbolic simulator.
Isolating the Bottleneck: By comparing Oracle and Real variants, the authors demonstrate that the primary constraint on current agents is visual perception, not the ability to plan or execute tool calls.
Operational Implications: The results suggest that the near-term operating point for clinical assistants involves a specialized perception model supplying findings, while a Large Language Model (LLM) orchestrates the workflow around those findings.
Limitations: The authors acknowledge that the reference trajectories are programmatic rather than radiologist-validated, the dataset scope is limited to three TCIA cohorts (excluding mammography, ultrasound, etc.), and the tasks are tractable proxies rather than full reproductions of complex clinical workflows involving EHR integration or multi-agent collaboration.

In summary, ABRA provides a rigorous, interactive environment to evaluate radiology agents, revealing that while agents can effectively manage the radiology workstation interface, they currently lack the robust visual perception required to interpret raw medical imaging data autonomously.