💻 computer science

CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcare

This paper introduces CarePilot, a multi-agent framework utilizing an actor-critic paradigm with dual-memory mechanisms to overcome the limitations of existing vision-language models in automating complex, long-horizon computer tasks within specialized healthcare systems, achieving state-of-the-art performance on the newly proposed CareFlow benchmark.

Original authors: Akash Ghosh, Tajamul Ashraf, Rishu Kumar Singh, Numan Saeed, Sriparna Saha, Xiuying Chen, Salman Khan

Published 2026-03-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Akash Ghosh, Tajamul Ashraf, Rishu Kumar Singh, Numan Saeed, Sriparna Saha, Xiuying Chen, Salman Khan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly clumsy, robot assistant how to perform a complex surgery on a patient. You can't just say, "Fix the heart." You have to guide it through hundreds of tiny steps: "Pick up the scalpel," "Cut here," "Suture that," "Check the monitor," "Save the file."

If the robot gets confused at step 45 and clicks the wrong button, the whole surgery fails, and the patient (or in this case, the medical data) is in trouble.

This is the exact problem the paper "CarePilot" tries to solve, but instead of a physical robot, it's teaching a computer program how to navigate medical software (like X-ray viewers, patient record systems, and lab tools).

Here is the story of how they did it, broken down into simple concepts.

1. The Problem: The "Short-Term Memory" Robot

Current AI models (like the ones powering chatbots) are great at answering questions or writing emails. But when you ask them to click through a complex medical software program to do a specific task (like "Find a lung nodule, measure it, and save the report"), they often fail.

Why?

They forget: They have short attention spans. By step 10 of a 20-step task, they forget what they did in step 1.
They get lost: Medical software is confusing. It has tiny buttons, weird menus, and changes every time a hospital updates it.
They hallucinate: They might guess, "I think I should click here," even if the button isn't there.

The authors realized that to fix this, they needed two things: a new test to see how bad the robots really are, and a new brain to help them think better.

2. The New Test: "CareFlow" (The Medical Driving Test)

Before they could build a better robot, they needed a better driving test. Existing tests were like asking a robot to "turn left" or "park the car." That's too easy.

They created CareFlow, a massive library of real-world medical scenarios.

The Analogy: Imagine a driving simulator that doesn't just have a straight road. It has a storm, a detour, a broken traffic light, and a passenger screaming instructions.
The Content: They recorded 1,100 real tasks done by doctors and radiologists. These tasks involve opening X-rays, zooming in on tumors, measuring them, and updating patient records.
The Result: When they ran existing AI models on this test, they failed miserably. They got lost in the middle of the task, clicked the wrong buttons, and couldn't finish the job.

3. The Solution: "CarePilot" (The Pilot and the Co-Pilot)

To fix the failing robots, the authors built CarePilot. They didn't just make one big AI; they made a team of two AIs working together, inspired by how a pilot and a co-pilot fly a plane.

The Actor (The Pilot)

This is the AI that actually does the clicking and typing.

Superpower: It has Two Memories.
- Short-Term Memory: "I just clicked the 'Zoom' button. Now I need to zoom in."
- Long-Term Memory: "I am on step 5 of 15. My goal is to measure a tumor. I am currently in the 'Lung' module."
The Toolkit: Before it clicks, it uses special tools (like a magnifying glass or a text reader) to make sure it knows exactly where the buttons are. It doesn't guess; it looks.

The Critic (The Co-Pilot)

This is the AI that watches the Pilot.

Role: It doesn't touch the mouse. It just watches and says, "Wait! You clicked 'Zoom' when you should have clicked 'Measure'!"
The Feedback Loop: If the Pilot makes a mistake, the Critic stops the process, explains why it was wrong, and helps the Pilot try again.
The Magic: During training, the Critic teaches the Pilot. Once the Pilot learns enough from the Critic, the Critic steps back. The Pilot becomes so smart it can fly the plane alone without needing the Co-Pilot to correct it every second.

4. How It Works in Real Life (The "Surgery" Analogy)

Let's say the task is: "Open the patient's CT scan, find the tumor in the liver, measure it, and write a note."

The Pilot (Actor) looks at the screen. It uses its Short-Term Memory to remember it just opened the file. It uses its Long-Term Memory to remember the goal is "Liver Tumor."
It uses a Tool (like a magnifying glass) to find the "Liver" button.
It clicks "Liver."
The Critic checks: "Good job. But wait, you didn't zoom in yet. You can't measure a tiny tumor from far away."
The Critic sends a note to the Pilot's memory: "Next time, Zoom first."
The Pilot tries again, zooms in, measures the tumor, and types the note.
Because the Pilot learned from the Critic, it gets faster and more accurate every time.

5. The Results: A New World Record

When they tested CarePilot against the best AI models in the world (including the famous GPT-4 and GPT-5):

The Old Models: Got about 25% of the tasks right. They were confused and kept making the same mistakes.
CarePilot: Got nearly 50% of the tasks right (and up to 88% on simple steps).
The Gap: CarePilot was significantly better, proving that giving an AI a "memory" and a "teacher" (the Critic) makes it much better at complex, long jobs.

Summary

Think of CarePilot as a medical intern who has a photographic memory (Long-Term Memory), a magnifying glass (Tools), and a strict mentor (The Critic) standing over their shoulder.

Without CarePilot: The intern is smart but forgetful and gets lost in the hospital maze.
With CarePilot: The intern remembers where they are, uses the right tools to see clearly, and listens to the mentor until they become a master surgeon of software.

This is a huge step forward because it means computers might soon be able to help doctors with boring, repetitive computer tasks, freeing up human doctors to focus on the patients.

1. Problem Statement

The paper addresses the critical gap in long-horizon computer task automation within the healthcare domain. While recent multimodal agents (powered by Vision-Language Models or VLMs) have shown success in short-horizon tasks on general interfaces (mobile, desktop, web), they struggle significantly in complex, domain-specific clinical environments.

Key Challenges Identified:

Long-Horizon Dependencies: Clinical workflows often require 8–24 interdependent steps (e.g., opening a study, configuring views, annotating, exporting, updating records). Current models fail to maintain coherence over such long sequences.
Domain Heterogeneity & Constraints: Healthcare software (DICOM viewers, EHRs, LIS) is highly heterogeneous, frequently updated, and constrained by strict privacy and data integrity policies. Agents that overfit to surface-level layouts fail to generalize.
Lack of Benchmarks: There was no standardized, high-quality benchmark for evaluating multimodal agents on real-world clinical software interactions.
Reasoning & Grounding: Existing models struggle with "tool grounding" (identifying specific UI elements in complex medical interfaces) and lack robust memory mechanisms to track state across dozens of steps.

2. Methodology

The authors propose a two-pronged solution: a new benchmark (CareFlow) and a novel multi-agent framework (CarePilot).

A. CareFlow: The Benchmark

CareFlow is a high-quality, human-annotated benchmark designed to evaluate long-horizon automation in healthcare.

Scope: Covers four major clinical systems:
1. DICOM Viewers/Infrastructure: (Orthanc, Weasis)
2. Medical Image Computing/Annotation: (3D Slicer)
3. Electronic Medical Records (EMR/EHR): (OpenEMR)
4. Laboratory Information Systems (LIS): (OpenHospital)
Data Composition: 1,100 tasks, each consisting of 8–24 consecutive steps.
Annotation Pipeline: Developed in collaboration with medical experts to ensure realism. Tasks involve natural language goals paired with GUI screenshots. Actions are labeled as interface-invariant semantic actions (e.g., CLICK, SCROLL, ZOOM, TEXT, SEGMENT, COMPLETE).
Metrics: Evaluated using Step-Wise Accuracy (SWA) and Task Accuracy (TA) (exact match of the full sequence).

B. CarePilot: The Framework

CarePilot is a multi-agent framework based on the Actor-Critic paradigm, enhanced with dual-memory mechanisms and tool grounding.

Tool Grounding:
- To handle complex medical interfaces, the system integrates lightweight perception tools:
  - UI Object Detection: Identifies widgets via open-vocabulary queries.
  - Zoom/Crop: Inspects small controls.
  - OCR: Extracts text (patient IDs, labels) to disambiguate visual elements.
  - Template Matching: Recognizes icons regardless of theme or scale.
- These tools provide a grounded feature vector ( $\phi_t$ ) to the agent.
Dual-Memory Mechanism:
- Short-Term Memory (STM): Summarizes the immediate context (previous screenshot, action, and critic feedback).
- Long-Term Memory (LTM): A compact trajectory embedding updated with tool-grounding features, maintaining historical context and semantic consistency across the workflow.
Actor-Critic Architecture:
- Actor Agent: A multimodal LLM (e.g., Qwen-VL) that observes the current state, tools, and memories to predict the next semantic action.
- Critic Agent: Evaluates the Actor's proposed action.
  - Hierarchical Reflection: If an action is incorrect, the Critic performs three levels of reflection:
    1. Action Reflector: Checks for local grounding errors.
    2. Trajectory Reflector: Diagnoses stalled progress or violated preconditions.
    3. Global Reflector: Evaluates overall goal consistency.
  - The Critic updates both STM and LTM with feedback and corrections.
Training Strategy (Reasoning Distillation):
- During training, the Actor learns from trajectories where the Critic has verified and corrected actions.
- The Critic's feedback is distilled into the Actor's policy.
- Inference: Only the distilled Actor is used. It approximates the Critic's reasoning internally, eliminating the runtime overhead of a separate Critic agent while preserving performance.

3. Key Contributions

Problem Formulation: Defined the task of long-horizon computer automation specifically for healthcare software, emphasizing the need for domain knowledge and multi-step reasoning.
CareFlow Benchmark: Introduced the first expert-annotated benchmark for healthcare GUI agents, covering 1,100 tasks across four distinct clinical domains with rigorous quality assurance.
CarePilot Framework: Proposed a novel multi-agent architecture integrating tool grounding with dual-memory (STM/LTM) and hierarchical reflection to solve long-horizon dependencies.
State-of-the-Art Performance: Demonstrated that CarePilot significantly outperforms existing baselines, proving that memory and grounding are essential for clinical automation.

4. Experimental Results

The authors evaluated CarePilot against strong open-source (Qwen, Llama, Mistral) and closed-source (GPT-4o, GPT-5, Gemini) baselines.

Performance on CareFlow:
- Task Accuracy (TA): CarePilot (Qwen 3 VL-8B) achieved 51.45%, outperforming the strongest baseline (GPT-5 at 36.19%) by ~15.26%.
- Step-Wise Accuracy (SWA): CarePilot achieved 90.18%, surpassing GPT-5 (85.22%) by ~5 points.
- CarePilot consistently outperformed all baselines across all four software domains (Weasis, 3D Slicer, Orthanc, OpenEMR).
Out-of-Distribution (OOD) Performance:
- Tested on OpenHospital (not seen during training). CarePilot achieved 38.18% TA, significantly outperforming the best baseline (GPT-5 at 34.80%), demonstrating robust generalization.
Ablation Studies:
- Critic Agent: Removing the Critic caused TA to drop from 48.90% to 3.75% (without tools) or 12.5% (with tools), highlighting the Critic's role in hierarchical reflection.
- Tool Grounding: Removing tool grounding dropped TA to 9.37%, proving its necessity for accurate action prediction.
- Memory: Long-Term Memory (LTM) was found more critical than Short-Term Memory (STM) for long-horizon tasks.
Qualitative Analysis:
- Baselines frequently failed due to action-mode confusion (e.g., clicking instead of zooming, or scrolling instead of segmenting) and premature text emission.
- CarePilot maintained state awareness, correctly sequencing tool selection and execution.

5. Significance and Impact

Clinical Automation: CarePilot provides a pathway toward safe, interpretable, and generalizable automation in clinical environments, potentially reducing the administrative burden on medical professionals.
Methodological Advancement: The paper demonstrates that for complex, long-horizon tasks, simple end-to-end VLMs are insufficient. A combination of external tool grounding, hierarchical memory, and iterative self-reflection (distilled into a single agent) is required.
Benchmarking: CareFlow fills a critical void in AI research by providing a realistic, domain-specific testbed that moves beyond generic web/mobile interfaces to specialized medical software.
Future Directions: The work suggests future needs for pixel-level grounding, support for even longer workflows, and expansion to more diverse clinical platforms.

In summary, CarePilot represents a significant leap forward in multimodal agent research, specifically tailored to the high-stakes, complex, and long-horizon nature of healthcare software automation.