💻 computer science

SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark

The paper introduces SurgCoT, a comprehensive benchmark designed to evaluate and advance the fine-grained spatiotemporal chain-of-thought reasoning capabilities of Multi-modal Large Language Models across diverse surgical procedures by defining five core reasoning dimensions and a structured annotation protocol.

Original authors: Gui Wang, YongSong Zhou, Kaijun Deng, Wooi Ping Cheah, Rong Qu, Jianfeng Ren, Linlin Shen

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gui Wang, YongSong Zhou, Kaijun Deng, Wooi Ping Cheah, Rong Qu, Jianfeng Ren, Linlin Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to be a surgeon. You show it thousands of hours of surgery videos. The robot can easily tell you, "That's a scalpel," or "That's a kidney." But can it understand why the surgeon is doing what they are doing? Can it predict what happens next? Can it spot a tiny mistake before it becomes a disaster?

Right now, most AI models are like tourists looking at a surgery video: they see the sights but don't understand the plot. They miss the subtle clues, the timing, and the cause-and-effect relationships that a real surgeon knows instinctively.

This paper introduces SurgCoT, a new "training ground" and "exam" designed to fix this. Here is the breakdown in simple terms:

1. The Problem: The "Tourist" vs. The "Detective"

Current AI models are great at answering simple questions like "What tool is this?" (Tourist mode). But surgery is complex. It requires spatiotemporal reasoning—a fancy way of saying: Understanding how things move in space and time to figure out what is happening.

A real surgeon acts like a detective. They don't just see a cut; they see why the cut was made, when it happened, and what it implies for the next step. They track the story of the surgery, not just the pictures.

2. The Solution: SurgCoT (The "Detective Academy")

The authors built a massive library of 2,841 surgery videos covering 35 different types of operations (from eye surgery to heart surgery). But they didn't just dump the videos on the AI. They created a special exam format called Chain-of-Thought (CoT).

Think of this like teaching a student to solve a math problem:

Old Way: Show the problem, ask for the answer. (Did the student guess right, or did they actually learn?)
SurgCoT Way: Force the student to show their work step-by-step.

3. The Secret Sauce: The "Three-Stage Detective Ladder"

To test if the AI is thinking like a surgeon, SurgCoT breaks every question down into three levels, like climbing a ladder:

Step 1 (The Big Picture): "Is there a problem?" (e.g., "Is there bleeding?")
- Analogy: A detective looking at a crime scene and asking, "Did a crime happen?"
Step 2 (The Timeline & Location): "When did it start and where exactly?" (e.g., "The bleeding started at 4:05 PM near the artery.")
- Analogy: The detective narrowing it down: "It happened right after the suspect dropped the knife, near the window."
Step 3 (The Micro-Detail): "Show me the exact frame and pixel." (e.g., "Here is the exact moment the vessel tore.")
- Analogy: The detective zooming in on the fingerprint on the knife handle.

4. The "Five-Tuple" Cheat Sheet

To help the AI learn, the researchers didn't just ask questions. They provided a special five-part hint system for every question:

Question: What are we asking?
Options: Possible answers (to rule out wrong guesses).
Knowledge: The "Textbook" info (e.g., "Usually, arteries bleed bright red").
Clue: The "Video Evidence" (e.g., "Look at the red spot at 4:05 PM").
Answer: The final verdict.

This forces the AI to combine what it knows (Knowledge) with what it sees (Clue) to reach a conclusion. It's like giving a student a textbook and a magnifying glass to solve a mystery.

5. What They Found (The Results)

They tested 10 of the smartest AI models in the world (including big commercial ones like GPT-5 and open-source ones).

The Good News: The "Big Brain" commercial models are currently the best detectives. They are getting better as you give them more hints (Knowledge and Clues).
The Bad News: Even the best models are still struggling. They often get the final answer right by guessing, but they fail the "middle steps."
- Analogy: Imagine a student who gets the math answer "42" right, but their work shows they added 5 + 5 and got 42. They got lucky, but they don't actually understand the logic.
The Gap: There is a huge gap between what AI can do today and what a real human surgeon needs for safety. AI is currently "hallucinating" the logic even when the answer is correct.

6. Why This Matters

SurgCoT isn't just a test; it's a blueprint for the future.

It proves that AI needs to be taught to think in steps, not just guess.
It shows that giving AI "clues" (like pointing out the exact time and place in the video) helps them reason much better.
It sets a new standard so we can build AI that doesn't just watch surgery, but actually understands it, helping to keep patients safe in the future.

In a nutshell: The authors built a "Detective Academy" for AI using surgery videos. They taught the AI to solve mysteries by breaking them down into small, logical steps. While the AI is still a bit clumsy and needs more training, this new system shows us exactly how to teach it to think like a human surgeon.

1. Problem Statement

While Multi-modal Large Language Models (MLLMs) have shown promise in surgical video analysis (e.g., phase recognition, tool detection), current capabilities in fine-grained spatiotemporal reasoning remain largely unexplored. Existing benchmarks suffer from two main limitations:

Task Narrowness: They focus on generic questions or narrow procedural domains (e.g., only endoscopy or only laparoscopy).
Reasoning Depth: They rely on frame-level or clip-level Visual Question Answering (VQA), treating videos as discrete segments. This ignores cross-temporal dependencies and fails to evaluate the causal, progressive reasoning required for clinical decision-making (e.g., tracking an anomaly's onset or understanding the causal order of micro-actions).

There is a critical gap between current MLLM capabilities and the expert-level, hierarchical reasoning needed in real-world surgical workflows.

2. Methodology: The SurgCoT Framework

The authors introduce SurgCoT, a unified benchmark designed to evaluate and scaffold Chain-of-Thought (CoT) reasoning in surgical videos.

A. Dataset Construction

Scale & Diversity: The dataset comprises 2,841 surgical videos covering 7 specialties (Colorectal, Urological, Upper GI, Ocular, Gynecologic, General Surgery, Hepatobiliary-Pancreatic) and 35 diverse procedures.
Annotation Volume: It includes 19,345 main questions and 59,177 sub-questions.
Data Processing: Videos are processed into standardized clips using hierarchical segmentation (visual scenes, tool/tissue transitions) and ASR alignment. An end-to-end evidence mining paradigm extracts spatiotemporal supervision units (tool masks, tissue bounding boxes, action tracks) from unannotated clips.

B. Three-Stage Progressive Reasoning Framework

SurgCoT decomposes complex clinical reasoning into a hierarchical, three-stage process ( $Q1 \to Q2 \to Q3$ ):

Stage 1 (Video-Level Comprehension): Identifies high-level clinical events (e.g., "Is active bleeding present?") to establish a global hypothesis space.
Stage 2 (Clip-Level Localization): Operates under the validated output of Stage 1 to determine when and where an event emerges within a bounded temporal window.
Stage 3 (Frame/Patch-Level Localization): Requires precise localization of exact onset frames and anatomical sites (pixel/bbox level) based on the constraints of Stage 2.

C. Five-Tuple Annotation Protocol

To enable verifiable CoT reasoning, each stage is annotated with five fields:

Question: A clinically meaningful query.
Option: Mutually exclusive candidates to disambiguate similar phenomena.
Knowledge: Domain priors (anatomy, device behavior) providing the "why."
Clue: Video-grounded evidence (temporal windows, spatial ROIs) providing the "where/when."
Answer: The adjudicated target.

Mechanism: The answer from one stage serves as the conditioning context for the next, enforcing a strict logical dependency and narrowing the spatiotemporal scope progressively.

D. Five Core Reasoning Dimensions

The benchmark evaluates MLLMs across five specific cognitive tasks:

Causal Action Ordering (CAO): Determining the causal sequence of micro-actions.
Cue–Action Alignment (CAA): Aligning pre-action cues with execution to pinpoint intent.
Affordance Mapping (AM): Grounding tool-tissue interactions.
Micro-Transition Localization (MTL): Identifying frame-level boundaries between micro-phases.
Anomaly Onset Tracking (AOT): Detecting and localizing the onset of surgical anomalies.

3. Key Contributions

First Unified Surgical Reasoning Benchmark: SurgCoT is the first benchmark to offer cross-specialty coverage (7 specialties, 35 procedures) with a five-dimensional evaluation framework aligned with surgical workflows.
Novel Annotation Protocol: Introduction of a three-stage progressive reasoning framework combined with a five-tuple annotation protocol (Question→Option→Knowledge→Clue→Answer) that integrates hierarchical video-to-frame localization with synergistic knowledge/clue annotations.
Comprehensive Evaluation: Systematic evaluation of 12 state-of-the-art MLLMs (Commercial, Open-Source, and Medical-Specialized) revealing significant gaps in current models' ability to perform progressive spatiotemporal reasoning.

4. Experimental Results

The authors evaluated 10 leading MLLMs (including GPT-5, Claude-Sonnet-4.5, Gemini-2.5-Pro, Qwen, InternVL, and medical-specialized models like LLaVA-Med) under three settings:

Baseline (BL): Video + Question only.
Knowledge-Enhanced (KE): Adding clinical background knowledge.
Full-Context (FC): Adding Knowledge + Spatiotemporal Clues.

Key Findings:

Commercial Models Lead: Commercial models (GPT-5, Claude, Gemini) consistently outperformed open-source and medical-specialized models across all dimensions.
Significant Reasoning Gaps: Even top-tier models showed a "performance dissociation." For example, GPT-5 achieved ~76% accuracy on main questions but dropped to ~47% on sub-questions (Stage 3), indicating a failure to sustain coherent multi-step reasoning chains.
Effectiveness of CoT Scaffolding: The five-tuple protocol significantly boosted performance.
- Adding Knowledge (KE) improved medical-specialized models by ~7% on average.
- Adding Clues (FC) provided further gains (e.g., +13.44% for Claude-Sonnet-4.5), proving that explicit spatiotemporal grounding is critical for fine-grained reasoning.
Progressive Improvement: All models showed stable performance progression ( $BL \to KE \to FC$ ), validating that structured reasoning scaffolds can enhance MLLM capabilities in surgical contexts.

5. Significance

Clinical Relevance: SurgCoT moves beyond simple recognition tasks to evaluate the reasoning capabilities required for clinical safety, such as tracking anomalies and understanding causal workflows.
Reproducible Testbed: It provides a standardized, reproducible environment to measure the gap between current AI capabilities and clinical reasoning demands.
Path Forward: The results demonstrate that while current MLLMs struggle with independent progressive reasoning, they can achieve significant improvements when guided by structured CoT frameworks with explicit knowledge and spatiotemporal evidence. This suggests a viable path for developing AI assistants that can support surgeons with evidence-based, traceable decision-making.

Code and Data: Available at https://github.com/CVI-SZU/SurgCoT.