Technical Summary: SurgiQ

Problem Statement

Reliable evaluation of Large Language Models (LLMs) in the domain of surgery remains underdeveloped. While existing medical benchmarks (e.g., MedQA-USMLE, MedMCQA) effectively test broad clinical and biomedical knowledge, they often fail to capture the specific complexities of surgical reasoning. Surgery requires procedural sequencing, operative anatomical knowledge, intraoperative judgment, perioperative management, and the ability to select the optimal option among clinically plausible alternatives. Furthermore, recent surgical benchmarks have focused primarily on multimodal perception (visual understanding of surgical scenes, instruments, and workflows) rather than source-grounded textual assessment of surgical knowledge and operative decision-making across specialties. There is a lack of a large-scale, text-only benchmark capable of evaluating surgical reasoning and calibration in open-weight LLMs.

Methodology

Dataset Construction (SurgiQ)

The authors present SurgiQ, a text-only, source-grounded benchmark comprising 13,055 four-option multiple-choice questions (MCQs).

• Sources: The dataset is constructed from 27 surgical textbooks, 635 open-access research papers, and surgical examination materials.
• Domains: The questions span six surgical specialties: General Surgery (53.2%), Neurosurgery (16.9%), Robotic Surgery (12.6%), Orthopedic Surgery (9.1%), Critical Care/Emergency (4.5%), and Laparoscopic Surgery (3.8%).
• Question Formats: The dataset includes four distinct formats to test different reasoning capabilities:
  1. Case-based: Clinical scenarios requiring a decision.
  2. Reasoning: Questions requiring multi-step logical deduction.
  3. Best-option: Selecting the optimal choice among partially correct alternatives.
  4. Negative: Identifying statements inconsistent with current evidence (handling negation).

Generation Pipeline

The dataset was generated using a multi-stage pipeline built on Gemini 2.5 Flash:

1. Extraction: Raw textual content is extracted from source documents, organized into structured segments (chapters/pages), and cleaned of non-textual elements.
2. Fact Extraction & Generation: High-yield facts are extracted from text segments, which are then used to generate clinically grounded MCQs. Prompts are designed to encourage multi-step reasoning and avoid trivial recall.
3. Verification & Filtering: A unified pipeline performs answer verification, quality filtering, rule-based checks, and deduplication.
   • Initial generation produced 17,068 candidates.
   • 4,013 candidates (23.5%) were removed due to issues such as unsupported grounding, ambiguity, malformed options, or answer-label inconsistencies.
   • The final set contains 13,055 questions.
4. Quality Control: A physician audit of 300 randomly sampled questions confirmed 92% answer correctness and 90% clarity. Inter-annotator agreement was substantial (Fleiss' κ = 0.84 for correctness).

Evaluation Protocol

The authors evaluated 35 open-weight LLMs under a unified, deterministic zero-shot and few-shot framework.

• Models: The evaluation covered general-purpose, biomedical, and reasoning-oriented families (e.g., Qwen2.5, LLaMA-based, Mistral, DeepSeek-R1-Distill, MedGemma).
• Scoring: Predictions were computed using log-likelihood over the answer options (A, B, C, D) rather than autoregressive sampling to ensure determinism. The option with the highest log-probability was selected.
• Metrics: Primary metric is accuracy. Secondary analyses include calibration (Expected Calibration Error, Brier score), domain performance, and error pattern analysis.

Key Contributions

1. Dataset: SurgiQ provides 13,055 source-grounded surgical MCQs with metadata, covering six domains and four question types, released as a versioned public resource.
2. Validation: Benchmark quality is established through physician audits, human surgeon baselines, near-duplicate analysis, and robustness tests, confirming the dataset cannot be solved by surface-level heuristics.
3. Evaluation: A comprehensive assessment of 35 open-weight LLMs using a unified log-likelihood protocol, accompanied by calibration, domain, and error analyses.
4. Findings: The study reveals that general-purpose models often outperform specialized biomedical models in surgical reasoning, and that even high-performing models struggle with calibration on clinically plausible distractors.

Results

Model Performance

• Top Performers: Qwen2.5-72B achieved the highest overall accuracy at 68.1%, followed by HuatuoGPT-o1-70B (65.45%) and Meerkat-70B (65.19%).
• General vs. Specialized: General-purpose models (specifically Qwen2.5) outperformed most biomedical-specialized models. This suggests that current medical specialization does not yet provide sufficiently broad surgical coverage and that broad reasoning ability is critical.
• Model Scale: Larger models generally outperformed smaller ones. Smaller models (2B–9B parameters) often performed near the 25% random baseline, highlighting the difficulty of the task.
• Human Baseline: On a manually audited 300-question subset, human surgeons achieved 89.1% accuracy, significantly outperforming the best LLM (51.3% on the same subset), indicating the benchmark is far from saturated.

Question Type and Domain Analysis

• Question Types: "Best-option" questions were consistently the most challenging due to the presence of multiple partially correct alternatives. "Negative" questions showed high variance across models, indicating difficulties in handling logical negation.
• Domains: Orthopedic Surgery was the lowest-scoring domain (49.23% mean accuracy across models), significantly lower than other domains. Laparoscopic Surgery also scored lower than Neurosurgery.

Calibration and Error Analysis

• Calibration: Even strong models exhibited calibration gaps. Qwen2.5-72B was mildly underconfident, while Meerkat-70B was substantially overconfident (ECE = 0.120).
• Failure Patterns: Qualitative analysis of errors revealed that failures were rarely due to simple factual recall. Instead, the primary error modes were:
  1. Confusion among clinically plausible "best-option" distractors (39%).
  2. Negation/exception failures (34%).
  3. Superficial cue anchoring (21%).
• Few-Shot Learning: Adding few-shot examples provided inconsistent gains; for some models (e.g., Qwen2.5-72B), a single example improved performance by aligning the format, but additional examples did not yield consistent improvements.

Significance and Claims

The paper positions SurgiQ as a complementary evaluation axis to existing multimodal surgical benchmarks. Its primary significance lies in:

• Identifying Gaps: Demonstrating that current medical specialization in LLMs is insufficient for surgical reasoning and that broad general-purpose reasoning is currently more effective.
• Reliability Concerns: Highlighting that high accuracy does not equate to reliability, as strong models frequently make "confident mistakes" on clinically plausible distractors.
• Benchmarking Standard: Providing a controlled, text-based environment to measure the prerequisite capability of discriminating among surgical decisions, distinct from deployment readiness.

The authors explicitly state that SurgiQ is not a clinical decision-support system and performance on these MCQs does not establish clinical competence or safety for real-world surgical settings. The benchmark is intended to drive the development of surgical LLMs with broader coverage and better reliability through rigorous, source-grounded evaluation.