🤖 AI

SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Robotic Surgical Video Analysis

This paper introduces SurgRAW, a multi-agent workflow leveraging Chain-of-Thought reasoning and a new benchmark (SurgCoTBench) to achieve superior, clinically aligned zero-shot understanding of robotic surgical scenes by overcoming the limitations of isolated models and general vision-language models through hierarchical collaboration and retrieval-augmented generation.

Original authors: Chang Han Low, Ziyue Wang, Tianyi Zhang, Zhu Zhuo, Zhitao Zeng, Evangelos B. Mazomenos, Yueming Jin

Published 2026-09-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chang Han Low, Ziyue Wang, Tianyi Zhang, Zhu Zhuo, Zhitao Zeng, Evangelos B. Mazomenos, Yueming Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In modern operating rooms, robotic systems have become essential tools, allowing surgeons to perform delicate procedures with a level of precision and steadiness that human hands alone cannot always achieve. These machines, often guided by high-definition cameras, transform the surgeon's movements into tiny, controlled actions inside the patient's body. However, for these systems to truly assist in the future, they must do more than just move instruments; they must understand what is happening on the screen. This requires a form of artificial intelligence capable of looking at a surgical video and grasping the complex story unfolding: identifying the tools being used, recognizing the specific actions the surgeon is taking, and predicting what will happen next. The challenge lies in the fact that surgical scenes are visually chaotic, with instruments and tissues often overlapping or moving rapidly, making it difficult for computers to interpret the scene without getting confused or inventing details that are not there.

Researchers have long tried to teach computers to understand these scenes, but previous attempts often relied on separate programs designed for single tasks, such as one program to find tools and another to name the actions. This approach created a fragmented view of surgery, where the computer could not connect the dots between what it saw and why it mattered. More recently, powerful language models have been adapted to look at images, offering a way to reason through visual problems. Yet, when applied to surgery, these general models often struggle with the specialized language and logic of the operating room, frequently producing confident but incorrect answers or failing to understand the step-by-step flow of a procedure. To solve this, a team of researchers has developed a new system that mimics the way a surgical team collaborates, using a structured, step-by-step reasoning process to analyze robotic surgery videos with unprecedented accuracy.

The team introduced a new framework called SurgRAW, which stands for a reasoning workflow designed specifically for surgical intelligence. Instead of asking a single artificial intelligence to look at a video frame and guess the answer, this system breaks the problem down into a coordinated effort among several specialized "agents." Imagine a panel of experts sitting around a table, each with a specific role: one focuses on identifying the instruments, another on the actions being performed, and a third on the patient's context. In this digital panel, these agents do not work in isolation; they discuss the evidence, check each other's logic, and refine their conclusions before reaching a final decision. This approach is built on a new dataset the researchers created, containing thousands of question-and-answer pairs derived from real surgical videos, covering five key areas of reasoning: recognizing instruments, identifying actions, predicting the next step, understanding patient details, and assessing the surgical outcome.

To ensure the system thinks like a surgeon, the researchers designed specific instructions that guide the artificial intelligence through a logical chain of thought. Rather than letting the model jump to a conclusion, the system forces it to first analyze the question, then extract visual details from the image, cross-reference those details with known surgical rules, eliminate impossible options, and finally verify the answer against the overall scene. For tasks that require knowledge beyond what is visible in the video, such as predicting the next step in a complex procedure, the system also consults a digital library of medical guidelines to ground its reasoning in established facts. This combination of structured reasoning, collaborative debate among agents, and access to verified medical knowledge allows the system to avoid the common pitfalls of artificial intelligence, such as hallucinating details that do not exist or misinterpreting the relationship between tools and tissue.

The results of testing this system were striking. When compared to existing artificial intelligence models, including those that had been specifically trained on large amounts of medical data, the new system performed significantly better. In tests measuring accuracy across various surgical tasks, the system outperformed a standard supervised model by 14.61 percent, a substantial margin in this field. It also demonstrated remarkable consistency, producing reliable results with very little variation between different runs, whereas other models often fluctuated wildly in their performance. The system was particularly effective at tasks requiring deep understanding, such as predicting the next step in a surgery or inferring patient details from visual cues, areas where previous models had struggled the most. By successfully integrating these different reasoning streams, the researchers showed that a unified, collaborative approach can provide a much clearer and more accurate understanding of robotic surgery than isolated methods.

This work represents a significant step forward in making artificial intelligence a trustworthy partner in the operating room. By moving away from simple pattern recognition toward a system that reasons, debates, and verifies its own conclusions, the researchers have created a tool that can explain its thinking in a way that aligns with clinical reality. The system does not just identify a tool; it understands what that tool is doing and why. It does not just see a frame; it grasps the narrative of the procedure. While the current system operates on individual frames sampled from continuous video, the underlying logic is designed to support the complex, flowing nature of surgery. The researchers plan to expand this work by including more types of procedures and exploring how the system can learn from real-time feedback from surgeons, aiming to eventually provide cognitive assistance that enhances safety and outcomes in the operating room.

Technical Summary: SurgRAW

Problem Statement

Robotic-assisted surgery (RAS) requires intelligent systems capable of accurate scene understanding to provide cognitive assistance to surgeons. Existing AI approaches for surgical video analysis suffer from three primary limitations:

Fragmentation: Most methods rely on isolated, task-specific models (e.g., separate networks for instrument detection or phase classification), leading to pipelines with limited interpretability and no unified understanding of the surgical scene.
Domain Gaps and Hallucinations: While Vision-Language Models (VLMs) offer zero-shot reasoning, they are primarily trained on natural scenes. When applied to RAS, they exhibit significant domain gaps, struggle with hallucinations, and lack the specific domain knowledge required for safe, reliable inference.
Lack of Unified Reasoning: Current VLMs often treat surgical tasks independently, failing to model the hierarchical dependencies and workflow interdependencies inherent in surgical procedures. Furthermore, existing datasets typically provide only one-to-three task labels per scene, insufficient for training holistic reasoning frameworks.

Methodology

The authors propose SurgRAW, a clinically aligned, Chain-of-Thought (CoT) driven agentic workflow designed for zero-shot multi-task reasoning in RAS. The system operates on a hierarchical multi-agent framework without requiring fine-tuning of the underlying VLMs.

1. SurgCoTBench Dataset

To address the lack of unified data, the authors introduced SurgCoTBench, the first reasoning-focused benchmark for RAS.

Composition: Derived from 12 patient videos (prostatectomy and lobectomy), containing 2,277 curated frames and 14,256 QA pairs.
Taxonomy: Covers five core reasoning tasks: Instrument Recognition, Action Recognition, Action Prediction, Patient Detail extraction, and Surgical Outcome assessment.
Annotation: Questions are generated using GPT-4o, leveraging surgeon narrations (transcribed via WhisperX) for semantic grounding. The dataset includes frame-level annotations and is split by patient for supervised baselines.

2. Hierarchical Multi-Agent Workflow

SurgRAW employs a central Orchestrator that classifies incoming queries into two reasoning streams:

Visual-Semantic (VS) Stream: Focuses on perceptual analysis (e.g., identifying instruments, recognizing actions).
Cognitive-Inference (CI) Stream: Focuses on procedural logic, patient context, and outcome prediction.

The orchestrator routes tasks to specialized agents (Instrument Specialist, Action Interpreter, Action Predictor, Patient Advocate, Outcome Analyst).

3. Key Mechanisms

Panel Discussion (for VS Tasks): To handle visual ambiguity, VS agents engage in a "panel discussion" moderated by an Action Evaluator. This agent:
- Checks consistency against a Surgical Knowledge Graph (KG) encoding valid instrument-action relationships.
- Applies Inter-Chain Coherence and Inter-Agents Collaborative Synergy rubrics to evaluate logical alignment and collaborative quality.
- Iteratively refines responses until consistency thresholds are met or a maximum iteration limit is reached.
Retrieval-Augmented Generation (RAG) (for CI Tasks): To bridge the domain gap, the CI stream integrates a RAG module. It queries a repository of medical resources (e.g., MedlinePlus) to retrieve surgical guidelines and procedural norms, which are then injected into the agent's reasoning process to ground predictions in clinical reality.
Task-Specific CoT Prompts: Instead of relying on generic auto-generated CoT, the authors designed structured, domain-specific CoT prompts.
- VS CoTs: Follow a five-stage process (Question Analysis $\rightarrow$ Feature Extraction $\rightarrow$ Cross-Referencing $\rightarrow$ Option Elimination $\rightarrow$ Verification).
- CI CoTs: Follow a six-stage process starting with Question Decomposition and incorporating retrieved domain insights before final verification.

Key Contributions

SurgCoTBench: The first benchmark for RAS focused on multi-task reasoning, unifying five major vision tasks with frame-level annotations.
SurgRAW Framework: A novel agentic framework capable of zero-shot reasoning across the surgical workflow, utilizing a hierarchical structure to separate visual perception from cognitive inference.
Synergistic Modules: The introduction of a Panel Discussion mechanism for cross-verification in visual tasks and a RAG module for grounding cognitive tasks in external medical knowledge.
Structured CoT Design: Task-specific CoT prompts tailored to RAS elements (instruments, tissue, actions) to enhance transparency and reduce hallucinations.

Experimental Results

Experiments were conducted in a zero-shot setting, comparing SurgRAW against general VLMs (e.g., GPT-4o, Qwen3VL), domain-adapted medical VLMs (e.g., LLaVA-Med), CoT-based prompting methods, and other agentic frameworks. A supervised baseline (Surgical-VQA) was also included.

Performance: SurgRAW significantly outperformed all baselines.
- It surpassed the supervised Surgical-VQA model by 14.61% in overall accuracy.
- Compared to standard zero-shot VLMs, SurgRAW achieved an average overall improvement of 29.32%.
- Notable gains were observed in Cognitive-Inference tasks, such as an 86.77% boost in Patient Detail Extraction compared to the GPT-4o baseline.
Reliability: Unlike the supervised baseline, which exhibited high standard deviation ( $\pm5.46\%$ ), SurgRAW maintained consistent performance ( $\pm1.51\%$ ) across runs.
Ablation Studies:
- CoT was identified as the most significant contributor, improving CI task performance by ~38.86%.
- RAG provided complementary gains for CI tasks, with the RAG-CoT combination achieving the highest accuracy (70.39%).
- Panel Discussion was crucial for VS tasks, particularly Action Recognition, showing synergistic effects when combined with CoT.

Significance and Claims

The paper claims that SurgRAW establishes a new benchmark for robotic surgical video understanding by demonstrating that zero-shot, multi-agent reasoning can outperform both supervised models and standard VLMs in complex surgical scenarios.

The authors emphasize that their approach addresses the critical need for interpretability and clinical alignment in surgical AI. By decomposing tasks into specialized agents and enforcing consistency through panel discussions and retrieved knowledge, SurgRAW mitigates the hallucinations and domain gaps that currently hinder the deployment of VLMs in safety-critical surgical settings. The work suggests that a unified, reasoning-driven framework is a viable path toward providing meaningful cognitive assistance to surgeons, moving beyond isolated task detection toward holistic scene understanding.