MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI

MedSPOT introduces a novel workflow-aware sequential grounding benchmark for clinical GUIs that evaluates multimodal models through multi-step, interdependent tasks and a strict error-propagation protocol to address the safety-critical limitations of existing single-step grounding benchmarks.

The Gist

Imagine you are teaching a very smart, but slightly clumsy, robot assistant how to use a complex medical software program. This software is used by doctors to look at X-rays, MRIs, and CT scans. It's not like a simple website where you just click a "Submit" button; it's a dense, crowded control panel with hundreds of tiny buttons, menus, and sliders, all looking very similar.

The paper introduces MedSPOT, a new "test" designed to see if these AI robots can actually navigate this medical software without making dangerous mistakes.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "One-Step" vs. The "Recipe"

The Old Way: Previous tests for AI were like asking a robot, "Can you find the red button?" The robot looks, finds it, and gets a point. Then the test ends. It doesn't matter if the robot pushes the wrong button after that.
The Real World: In a hospital, tasks are like a recipe. To "delete a patient's old scan," you have to:

1. Open the menu.
2. Find the specific file.
3. Click "Delete."
4. Confirm "Yes."

If the robot clicks the wrong file in Step 2, the whole recipe is ruined. The next steps don't even matter because the wrong file is now selected. The old tests didn't care about this chain reaction.

2. The Solution: MedSPOT (The "Medical Obstacle Course")

The authors built MedSPOT, a benchmark (a standardized test) that forces the AI to follow the whole recipe.

• The Setting: They used 10 different real-world medical software programs (like different brands of car dashboards).
• The Tasks: They recorded 216 real-world tasks (like "load an MRI" or "measure a tumor") and broke them down into 597 specific steps.
• The Twist: The test uses a "First Strike" rule. If the AI makes a mistake on Step 1, the test immediately stops and marks the whole task as a failure. It doesn't let the AI "recover" later. This mimics real life: if a doctor clicks the wrong patient's file, the system is now in the wrong state, and the rest of the workflow is broken.

3. The Results: The "Clumsy Robot" Reality Check

The authors tested 16 of the smartest AI models available today (including big names like GPT-4o, Llama, and Qwen). The results were shocking:

• General AI is terrible at this: Even the most famous AI models, which can write poems or solve math problems, scored almost zero on these medical tasks. They got lost immediately.
• Why? These AIs are trained on general internet data. They are like a person who knows how to drive a car but has never seen a Formula 1 race car dashboard. They get confused by the tiny, specific buttons on medical screens.
• The "Specialist" AI: A few models built specifically for computer screens (like GUI-Actor) did better, but even the best one only completed about 43% of the tasks perfectly. This means more than half the time, they still messed up the "recipe."

4. The "Failure Report" (Why did they fail?)

The paper created a "diagnosis" for the AI's mistakes, similar to a mechanic looking at a broken car:

• The "Edge Bias": The AI keeps clicking the very top or bottom edge of the screen, ignoring the actual buttons. It's like a driver who only looks at the horizon and never checks the dashboard.
• The "Toolbar Confusion": The AI clicks the main menu bar at the top instead of the specific tool needed in the middle of the screen. It's like trying to fix a flat tire by pressing the "Radio" button.
• The "Tiny Target" Problem: Medical software has tiny icons. The AI's "eyes" (vision sensors) are too blurry to see them clearly, so it misses the target entirely.

5. Why This Matters

This isn't just about getting a high score on a test. It's about safety.
If an AI is going to help doctors manage patient data, it cannot afford to click the wrong button. A single mistake could delete a critical scan or send a report to the wrong patient.

The Big Takeaway:
Current AI is like a brilliant student who can pass a written exam but fails when asked to actually drive a car in heavy traffic. MedSPOT is the driving test that proves we aren't ready to let AI drive the medical software yet. We need to build AIs that understand not just what to click, but how the whole sequence of clicks works together.

Summary Analogy

Imagine you are teaching a toddler to assemble a complex Lego castle.

• Old Tests: You ask, "Can you find the red brick?" The toddler finds it. You say, "Good job!" and stop.
• MedSPOT: You say, "Build the tower." The toddler picks up the red brick, but puts it on the wrong spot. The tower collapses. MedSPOT says, "Game Over. You failed the whole task because you missed the first step."

The paper tells us that our current "toddlers" (AI models) are still too clumsy to build the castle on their own, especially when the instructions are complex and the stakes are high.

Technical Summary

1. Problem Statement

Despite rapid advancements in Multimodal Large Language Models (MLLMs), their ability to perform reliable visual grounding (mapping natural language instructions to specific UI elements) in clinical software environments remains underexplored.

• Limitations of Existing Benchmarks: Current GUI grounding benchmarks (e.g., ScreenSpot, Mind2Web) focus on isolated, single-step queries in general-purpose environments (web/desktop). They treat grounding as a standalone prediction task, ignoring the sequential, workflow-driven reasoning required in real-world medical interfaces.
• Clinical Complexity: Medical GUIs (e.g., DICOM viewers, treatment planning systems) are densely structured, hierarchically organized, and use domain-specific terminology. In these high-stakes environments, an early grounding error invalidates subsequent steps, potentially leading to severe operational consequences.
• The Gap: There is no existing benchmark that evaluates workflow-aware sequential grounding with strict error propagation analysis in safety-critical medical software.

2. Methodology: The MedSPOT Benchmark

The authors introduce MedSPOT, a benchmark designed to model grounding as a sequence of interdependent spatial decisions within evolving interface states.

A. Dataset Construction

• Scale & Diversity: The dataset comprises 216 task-driven videos across 10 diverse medical software platforms (including Orthanc, 3D Slicer, RadiAnt, ITK-SNAP, etc.).
• Annotations: It contains 597 annotated keyframes. Each task consists of 2–3 interdependent steps (e.g., "Load DICOM" $\rightarrow$ "Select Series" $\rightarrow$ "Apply Filter").
• Content: Covers multiple imaging modalities (CT, MRI, PET, X-Ray, Ultrasound) and interface types (DICOM/PACS viewers, segmentation tools, web-based viewers).
• Annotation Protocol: Human annotators recorded real GUI interactions, extracting keyframes where the interface state changed. Each step includes:
  • GUI screenshot ($I_t$).
  • Natural language instruction ($s_t$).
  • Semantic target description ($y_t$).
  • Ground-truth normalized bounding box ($B_t$).
  • Action type (click).

B. Evaluation Protocol

MedSPOT introduces a strict sequential evaluation protocol that differs fundamentally from existing benchmarks:

1. Early Termination: Evaluation stops immediately upon the first incorrect grounding prediction. A task is only considered complete if all steps are correct.
2. Metrics:
   • Task Completion Accuracy (TCA): Fraction of tasks fully completed.
   • Step Hit Rate (SHR): Ratio of correct steps before the first failure.
   • Step-1 Accuracy (S1A): Accuracy of the initial step only.
   • Weighted Prefix Score (WPS): Emphasizes early correctness using exponential decay.
3. Failure Taxonomy: A structured six-class taxonomy diagnoses specific failure modes:
   • Edge Bias: Predictions collapsing toward image boundaries.
   • Small Target: Errors on tiny UI elements.
   • No Prediction: Model fails to output coordinates.
   • Near Miss: Geometrically close but outside the bounding box.
   • Far Miss: Semantic misunderstanding (wrong element).
   • Toolbar Confusion: Confusing global toolbars with task-specific elements.

3. Key Contributions

1. First Clinical Sequential Benchmark: MedSPOT is the first benchmark to jointly evaluate workflow-aware grounding, early-termination protocols, and structured failure analysis specifically for medical software.
2. Failure-Aware Evaluation: The proposed protocol explicitly measures error propagation, revealing that models often fail to maintain consistency across multi-step workflows even if they succeed in single steps.
3. Structured Failure Taxonomy: Provides a systematic diagnosis of model weaknesses (e.g., spatial precision vs. semantic alignment) in dense medical interfaces.
4. Comprehensive Evaluation: Evaluated 16 state-of-the-art MLLMs (including GPT-4o/5, Qwen, Llama, and specialized agents like GUI-Actor and UI-TARS), establishing a new baseline for medical GUI interaction.

4. Experimental Results

The evaluation of 16 models on MedSPOT revealed significant challenges:

• General-Purpose Model Collapse: Widely used general-purpose MLLMs (e.g., Llama 3.2, Qwen2-VL, DeepSeek-VL2, Gemma 3) achieved 0% Task Completion Accuracy (TCA). Even frontier models like GPT-5 reached only 2.8% TCA, despite decent single-step performance.
• Sequential Fragility: There is a massive drop-off from Step-1 Accuracy (S1A) to TCA. For example, GUI-Actor achieved 65.0% S1A but only 43.5% TCA. This indicates that while models can identify the first element, they fail to maintain spatial consistency as the interface state evolves.
• Specialized Models Perform Best: Models explicitly designed for GUI interaction performed significantly better. GUI-Actor was the top performer (43.5% TCA), followed by Qwen3-VL (35.0%) and UI-TARS (30.8%).
• Failure Patterns:
  • General models suffered heavily from No Prediction (LLaMA) and Far Miss (DeepSeek).
  • Closed-source models (GPT-5) struggled with Toolbar Confusion and Edge Bias.
  • Small Target errors persisted across all models (12–18%), highlighting the difficulty of localizing compact toolbar icons in DICOM viewers.
• Software Difficulty: Performance varied significantly by software. ITK-SNAP (cleaner interface) was the easiest (avg. TCA ~30%), while RadiAnt and Orthanc (dense toolbars) were the hardest (avg. TCA <10%).

5. Significance and Impact

• Safety-Critical Benchmarking: MedSPOT establishes a realistic standard for assessing MLLMs in healthcare, where "good enough" single-step accuracy is insufficient; workflow reliability is paramount.
• Revealing Model Limitations: The results demonstrate that current parameter scale and general multimodal pretraining do not translate to reliable spatial grounding in structured, high-stakes clinical environments.
• Future Directions: The benchmark highlights the need for models trained with explicit spatial supervision and architectures capable of handling state-dependent, multi-step reasoning. It serves as a critical tool for developing safe medical automation agents.
• Open Source: The dataset, code, and evaluation tools are publicly available to foster further research in medical AI and GUI grounding.

In conclusion, MedSPOT shifts the paradigm from isolated visual grounding to workflow-aware sequential reasoning, exposing a substantial performance gap in current AI systems and providing a rigorous framework for future development in clinical software automation.