SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA?

Technical Summary: SurgCheck

Problem Statement
Despite the promising performance of Vision-Language Models (VLMs) in Surgical Visual Question Answering (VQA), there is a critical uncertainty regarding whether these models possess genuine visual reasoning capabilities or merely exploit "linguistic shortcuts." Existing surgical VQA datasets often contain questions where the phrasing implicitly constrains the answer space (e.g., explicitly naming an instrument like "bipolar" in the question). In such cases, models may infer answers based on statistical correlations between question text and answers rather than analyzing the visual evidence. In safety-critical surgical environments, it is unclear if reported benchmark accuracy reflects true visual understanding or reliance on these textual biases.

Methodology
To address this evaluation gap, the authors introduce SurgCheck, a diagnostic benchmark designed to quantify linguistic shortcut reliance through a bias-controlled, paired-question design.

• Dataset Construction: SurgCheck is built upon the Endoscapes-SG201 dataset, utilizing its frame-level scene-graph annotations (instruments, anatomical structures, and relationships). It comprises 1,933 frames and 29,635 QA pairs across eight task categories: Instrument Count, Anatomy, Instrument, Action, Target, Triplet, Anatomy-hop, and Critical View of Safety (CVS).
• Paired-Question Design: For each surgical frame, two semantically equivalent questions are generated:
  1. Original Question: Follows conventional surgical phrasing with explicit entity names (e.g., "What action is the bipolar performing?").
  2. Less-Biased Question: Removes explicit entity names while preserving the identical visual content and ground-truth answer.
• Grounding Cues: To ensure the less-biased questions remain well-defined without entity names, four grounding cues are incorporated to reference the target object:
  1. Bounding Box: Visual localization via a red box.
  2. Arrow: Visual localization via a red arrow.
  3. Spatial Position: Textual description of location (e.g., "bottom-right of the image").
  4. Periphrasis: Contextual description (e.g., "tool used by the operator's right hand").
• Evaluation Protocol: The study evaluates five VLMs (three general-purpose: Qwen2.5-VL, LLaVA-OneVision, PaliGemma2; two surgical-specific: PitVQA-Net, EndoChat) under both zero-shot and fine-tuned settings.
  • For open-ended zero-shot responses, an LLM-as-a-Judge protocol (using GPT-5-mini) is employed to semantically assess answers against a rubric, assigning scores from 1 to 5.
  • For fine-tuned models producing categorical outputs, standard exact-match accuracy and F1-scores are used.
• Ablation Study: A text-only ablation is conducted where visual input is removed during inference to isolate the contribution of linguistic cues versus visual evidence.

Key Contributions

1. Bias-Controlled Paired-Question Benchmark: SurgCheck introduces a novel framework where original and less-biased questions share identical visual inputs and answers, differing only in the presence of linguistic shortcuts. This allows for a direct diagnostic signal of shortcut reliance.
2. Quantitative Analysis of Performance Gaps: The authors provide a systematic evaluation of five VLMs, measuring the performance degradation between original and less-biased questions to quantify the extent of shortcut reliance.
3. Linguistic Shortcut Diagnosis via Text-Only Ablation: By removing visual input, the study demonstrates that high performance on original questions for certain tasks is driven primarily by text rather than visual reasoning.

Results

• Zero-Shot Performance: All models exhibited performance degradation on less-biased questions compared to original questions, indicating reliance on linguistic shortcuts even before domain-specific fine-tuning. For instance, Qwen2.5-VL-7B saw an accuracy drop from 28.06% to 24.62%.
• Fine-Tuned Performance: While fine-tuning improved overall accuracy, the performance gap persisted. Models consistently performed worse on less-biased questions despite identical visual inputs. LLaVA-OV-1.5-8B showed a significant drop from 71.19% to 63.19% accuracy.
• Task-Specific Sensitivity:
  • Action and Target Tasks: These categories showed the largest F1-score drops when linguistic cues were removed. Text-only ablation revealed that performance on these tasks remained high even without images, confirming that predictions were driven by linguistic shortcuts.
  • Anatomy-hop and Triplet Tasks: These tasks, which require spatial reasoning, showed minimal degradation on less-biased questions and performed significantly better with images than with text alone, indicating a reliance on genuine visual evidence.
• Grounding Cue Analysis: Explicit visual cues (bounding boxes, arrows) did not fully mitigate the performance drop; models still struggled to associate localized visual evidence with actions/targets without entity names. Periphrasis cues showed the smallest degradation, suggesting that contextual descriptions encourage better visual integration than simple spatial coordinates.

Significance
The paper concludes that strong benchmark performance in surgical VQA does not necessarily imply faithful visual understanding. SurgCheck provides a controlled diagnostic framework that exposes failure modes masked by linguistic bias in existing benchmarks. The findings underscore the necessity for bias-aware evaluation in surgical VQA to ensure that models rely on visual evidence rather than exploiting question priors, which is critical for the safety and reliability of intraoperative decision support systems.