Technical Summary: SPARC-Rad

Problem Statement

While Vision-Language Models (VLMs) are increasingly evaluated for medical imaging tasks such as disease classification, report generation, and broad visual question answering (VQA), existing benchmarks often fail to isolate spatial perception and anatomical reasoning. Radiology practice fundamentally relies on interpreting anatomy as a spatial system—determining laterality, localizing structures, reasoning across imaging planes, and describing relationships between organs and devices. Current VQA resources (e.g., VQA-RAD, SLAKE) do not specifically target these spatial constructs. Consequently, a model may achieve high performance on general medical VQA or report generation while still committing clinically critical spatial errors, such as reversing laterality, mislocalizing medical devices, or confusing adjacent anatomical regions. There is a lack of a standardized framework to assess whether VLMs can reason about radiologic anatomy as a spatial system rather than relying solely on memorized patterns or general medical knowledge.

Methodology

The authors developed SPARC-Rad (Spatial Perception and Anatomical Reasoning in Clinical Radiology), a manually curated multimodal benchmark dataset and evaluation pipeline.

Dataset Construction

• Source: The dataset comprises 300 image-question pairs derived from healthy-control imaging studies in The Cancer Imaging Archive (TCIA). Healthy controls were selected to minimize the confounding influence of pathology and focus on normal anatomical understanding and spatial orientation.
• Modality Distribution: The dataset spans three modalities to test different spatial reasoning demands:
  • Radiography (114 pairs, 38.0%)
  • CT (98 pairs, 32.7%)
  • MRI (88 pairs, 29.3%)
• Anatomical Coverage: Cases cover five major categories: Abdomen, Chest, Breast, Neuro, and Musculoskeletal.
• Question Design: Radiology trainees manually designed questions requiring image-grounded reasoning. These target specific reasoning types: anatomical identification, localization, laterality, regional recognition, device identification, counting, and inter-structure spatial relationships.
• Ground Truth: Each pair includes a reference answer and a list of accepted synonyms to handle radiological abbreviations and terminology variations.

Evaluation Pipeline

SPARC-Rad is not just a dataset but a reproducible evaluation framework:

1. Standardized Prompting: Models receive a concise prompt instructing them to answer based only on the provided image.
2. Output Collection & Normalization: Raw model outputs are stored and normalized (e.g., lowercasing, punctuation removal, synonym mapping) to reduce false errors caused by wording differences while preserving clinically meaningful distinctions (e.g., "left" vs. "right").
3. Grading Strategy:
   • Binary Correctness: The primary metric is whether the response matches the reference answer or an accepted equivalent.
   • LLM-as-Judge: Due to the complexity of radiological synonyms, a separate judging model evaluates if the response is anatomically and spatially equivalent to the ground truth.
   • Human Review: Ambiguous cases, partial answers, and specific spatial errors (laterality, device localization) undergo human review to resolve disagreements and ensure clinical validity.
4. Statistical Analysis: The framework supports overall accuracy calculation with 95% confidence intervals, subgroup analysis (by modality, anatomy, and reasoning type), and pairwise model comparisons using Chi-square or McNemar tests for matched sets.

Key Contributions

1. Specialized Benchmark: SPARC-Rad is the first benchmark explicitly designed to isolate and evaluate spatial perception and anatomical reasoning in radiology VLMs, distinct from disease diagnosis or report generation.
2. Multimodal and Multidimensional Design: By including CT, MRI, and radiography across five anatomical regions, the benchmark tests a model's ability to generalize across cross-sectional and projection-based imaging.
3. Robust Evaluation Pipeline: The paper introduces a comprehensive workflow that combines answer normalization, LLM-as-judge grading, and human review to handle the nuances of medical terminology and spatial correctness.
4. Granular Error Analysis: The framework enables the identification of specific failure modes (e.g., laterality errors, wrong structure identification, hallucinated findings) rather than relying on a single aggregate score.

Results and Outputs

The paper presents the finalized structure of the SPARC-Rad dataset and the evaluation framework rather than the performance scores of specific models. The key outputs defined by the benchmark include:

• Overall Accuracy: The proportion of the 300 pairs answered correctly.
• Stratified Accuracy: Performance breakdowns by modality (CT, MRI, XR), anatomical category, and reasoning type.
• Dispersion Metrics: Measures of consistency across subgroups to identify models that may have high average accuracy but poor reliability in specific domains.
• Error Taxonomy: A structured classification of failure modes, including laterality errors, wrong structure identification, relational errors, and modality misunderstandings.

Significance and Claims

The authors position SPARC-Rad as a foundational tool for the safe and effective development of radiology VLMs.

• Foundational Assessment: The benchmark assesses "foundational anatomical reasoning" rather than comprehensive diagnostic competence. The authors argue that reliable spatial understanding is a prerequisite for clinical deployment, and failures in laterality or localization can occur even in the absence of disease.
• Beyond Leaderboards: SPARC-Rad aims to move the field beyond single leaderboard scores by providing a multidimensional performance profile. This allows developers to identify specific weaknesses (e.g., poor performance on musculoskeletal laterality) that might be hidden in aggregate metrics.
• Pre-deployment Screening: The benchmark is intended to serve as a component of a broader evaluation strategy to identify foundational weaknesses before a model is considered for educational or clinical use.
• Limitations and Modesty: The authors explicitly acknowledge limitations, including the current focus on healthy controls (excluding complex pathology), the absence of modalities like ultrasound, and the potential for data contamination from public TCIA collections. They emphasize that SPARC-Rad is not a substitute for clinical validation, prospective testing, or safety-critical workflow studies. Future work is proposed to expand the dataset to include pathological cases, postoperative anatomy, and additional modalities.