🤖 machine learning

Sum-of-Checks: Structured Reasoning for Surgical Safety with Large Vision-Language Models

The paper introduces "Sum-of-Checks," a framework that improves the accuracy and transparency of large vision-language models in assessing the Critical View of Safety during laparoscopic surgery by decomposing complex clinical criteria into structured, expert-defined reasoning checks.

Original authors: Weiqiu You, Cassandra Goldberg, Amin Madani, Daniel A. Hashimoto, Eric Wong

Published 2026-04-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Weiqiu You, Cassandra Goldberg, Amin Madani, Daniel A. Hashimoto, Eric Wong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Black Box" Surgeon Assistant

Imagine you are teaching a trainee surgeon how to perform a delicate operation. You tell them, "Look at the screen and tell me if it's safe to proceed."

The trainee looks at the screen and says, "Yes, it's safe!"

But then you ask, "Why? What exactly did you see?" and the trainee just shrugs and says, "I don't know, it just looked right to me."

In the world of Artificial Intelligence, this is a huge problem. We have "Large Vision-Language Models" (LVLMs)—super-smart AI that can "see" surgical videos and "talk" about them. However, when these AIs make a life-or-death decision, they often act like that shrug-happy trainee. They give an answer, but they can't explain their logic, and if they are wrong, we have no way of knowing where their thinking went off the rails. In surgery, a "black box" decision is a dangerous decision.

The Solution: "Sum-of-Checks" (The Surgical Checklist)

The researchers created a new framework called Sum-of-Checks.

Instead of asking the AI, "Is this surgery safe?" (which is too big and vague a question), they force the AI to act like a meticulous pilot going through a pre-flight checklist.

The Analogy: The Master Chef’s Tasting
Imagine a Master Chef is judging a soup.

The Old Way (Direct Prompting): The judge takes one sip and says, "This soup is a 7/10." (You don't know if it's a 7 because of the salt, the heat, or the texture).
The Sum-of-Checks Way: The judge must answer a series of specific questions first:
1. Is the salt level correct? (Yes/No)
2. Is the temperature right? (Yes/No)
3. Is the vegetable texture tender? (Yes/No)
4. Is the color appetizing? (Yes/No)

Only after answering these specific "checks" does the judge give a final score. If the score is low, the chef can look at the notes and see, "Ah, the salt was fine and the temperature was great, but the vegetables were too crunchy." Now, the chef knows exactly how to fix it.

How It Works in the Operating Room

The paper focuses on a specific procedure called a cholecystectomy (gallbladder removal). To do this safely, surgeons look for something called the "Critical View of Safety" (CVS). This is a specific visual pattern that proves they aren't about to accidentally cut a vital bile duct.

The Sum-of-Checks framework breaks the CVS down into tiny, bite-sized "reasoning checks" designed by real surgeons:

Visibility Check: "Can I actually see the gallbladder clearly?"
Obstruction Check: "Is a surgical tool blocking my view?"
Anatomy Check: "Are there exactly two tubes entering the gallbladder?"

The AI evaluates each tiny check, provides a written reason ("I see the gallbladder, but a tool is in the way"), and then calculates a final safety score based on how many checks passed.

The Results: Smarter and More Honest

The researchers tested this against the best AI models available (like GPT-4 and Claude). Here is what they found:

It’s Much More Accurate: By breaking the problem down, the AI's accuracy jumped significantly (by about 12–14%). It stopped "guessing" and started "verifying."
It’s Transparent (Auditable): If the AI says a frame is "unsafe," a human doctor can look at the checklist and see exactly which check failed. It turns a "black box" into a "glass box."
It Reveals AI Weaknesses: The study found something fascinating: the AI is great at "observational" tasks (like noticing if a tool is in the way), but it still struggles with "anatomical" tasks (like identifying specific tiny tubes). This tells scientists exactly where they need to improve the AI next.

The Bottom Line

Sum-of-Checks moves AI from being a "gut-feeling" observer to a "checklist-driven" expert. It ensures that when AI assists in the operating room, it isn't just giving an opinion—it's providing a documented, step-by-step proof of safety.

Technical Summary: Sum-of-Checks: Structured Reasoning for Surgical Safety with Large Vision-Language Models

Problem Statement

In laparoscopic cholecystectomy (gallbladder removal), surgeons must achieve the Critical View of Safety (CVS) to prevent bile duct injuries, which are high-morbidity complications. Assessing CVS requires complex reasoning over anatomical visibility, spatial relationships, and the exclusion of ambiguity.

While Large Vision-Language Models (LVLMs) offer potential for automating this assessment, they face two major hurdles in surgical settings:

Unreliability: They struggle with interdependent visual evidence and often fail on decision-critical anatomical tasks.
Lack of Auditability: Standard prompting methods (like Chain-of-Thought) produce "black-box" reasoning that is difficult for clinicians to verify or trust in safety-critical environments.

Methodology: The "Sum-of-Checks" Framework

The authors propose a structured framework that decomposes high-level surgical safety criteria into granular, expert-defined verification steps. The process follows three main stages:

Decomposition into Reasoning Checks: Instead of asking the model "Is CVS achieved?", the framework breaks each of the three CVS criteria into a set of $m$ natural-language "reasoning checks" ( $R_j$ ). These checks are designed by surgeons to cover:
- Observational reliability: (e.g., Is the camera exposure sufficient? Is an instrument obstructing the view?)
- Anatomical evidence: (e.g., Is the gallbladder neck visible? Are exactly two tubular structures entering the gallbladder?)
- Spatial configuration: (e.g., Is the hepatocystic triangle cleared of fat?)
Evidence Elicitation (LVLM Inference): For a given surgical frame, the LVLM evaluates each individual check. For every check, the model must produce:
- A justification (textual rationale).
- A binary judgment (1 for affirmative, 0 for negative/ambiguous).
Weighted Aggregation: The final criterion-level score ( $\hat{y}$ ) is calculated using a deterministic, weighted sum of the binary check outcomes:
$\hat{y} = \sum_{j=1}^{m} w_j \cdot r_j$
The weights ( $w_j$ ) are fixed and assigned by experts to reflect the clinical importance of specific anatomical landmarks or conditions.

Key Contributions

Structured Reasoning Framework: Introduced a method to operationalize expert surgical logic into intermediate decision units, separating evidence gathering from final decision-making.
Improved Interpretability: By providing specific rationales for each sub-check, the system allows clinicians to pinpoint exactly why a safety criterion was or was not met (e.g., "The score is low because the cystic artery is obscured by an instrument").
Empirical Validation of Structured Prompting: Demonstrated that explicit structure outperforms free-form reasoning (CoT) and model-generated sub-questions (SubQ).

Results

The framework was evaluated on the Endoscapes2023 benchmark using three frontier models (GPT-4.1-mini, Claude Haiku 4.5, and Claude Opus 4.5).

Performance Gains: Sum-of-Checks improved the average frame-level mean Average Precision (mAP) by 12–14% compared to the best baseline methods.
Robustness: Ablation studies showed that both few-shot examples (providing canonical clinical instances) and weighted aggregation (rather than letting an LLM decide the final score) are essential for high accuracy.
Error Analysis: The study revealed a performance gap: LVLMs are highly reliable at observational checks (visibility, obstruction) but remain inconsistent on anatomical/spatial reasoning (identifying specific structures), highlighting a remaining frontier for surgical AI.

Significance

This work shifts the paradigm of surgical AI from "end-to-end prediction" to "structured verification." By aligning model reasoning with the mental models used by human surgeons, Sum-of-Checks provides a blueprint for building auditable, transparent, and reliable AI assistants capable of supporting high-stakes intraoperative decision-making.