ScintiGround-38K: A Grounded Visual Question Answering Benchmark for Paired Anterior–Posterior Bone Scintigraphy

Technical Summary: ScintiGround-38K

Problem Statement
Whole-body bone scintigraphy (WBBS) is a standard, low-cost screening modality for skeletal metastases, typically acquired as paired anterior and posterior views. While deep learning has advanced lesion detection and classification in this domain, existing benchmarks often reduce clinical findings to free-text labels, severing the link between answers and the specific image evidence required to support them. Furthermore, standard Visual Question Answering (VQA) evaluation relies heavily on raw answer accuracy, which can be misleading in medical contexts where models may exploit question-conditioned priors or class frequency rather than visual content. There is a lack of a benchmark that preserves the original image geometry, maintains traceable source annotations (bounding boxes and labels), and simultaneously supports answer-only, localization-only, and grounded VQA tasks for paired WBBS data.

Methodology
The authors present ScintiGround-38K, a frozen, provenance-preserving dataset derived from two source collections: BS-80K (providing image references, region/lesion labels, and bounding boxes) and LIBS-160K (providing controlled, rule-based question templates).

• Dataset Construction: The benchmark contains 38,469 instances derived from 54,768 source records and 23,967 anterior–posterior image pairs from 2,815 patients. It defines three distinct task types:
  1. VQA: Requires a yes/no answer regarding a specified region or the whole body, without a predicted box.
  2. Image Localization: Requires one or two bounding boxes for a named bone region or hotspot, without answer scoring.
  3. Grounded VQA: Requires both a correct answer and the corresponding target box(es).
• Provenance and Integrity: Every instance retains its source identifier, view, native-image box coordinates, and patient-level split assignment. Metadata-only rows are excluded. The release distributes derived annotations only; raw upstream images are not redistributed.
• Data Splits: Patients are partitioned at the subject level (80% training, 10% validation, 10% test) to prevent data leakage and shortcut learning based on patient identity.
• Baseline Protocols: Two vision-language models, Qwen3-VL-2B-Instruct and PaliGemma-3B-pt-224, were adapted using Low-Rank Adaptation (LoRA) on the same multitask training set. Qwen3-VL received native anterior and posterior images, while PaliGemma received a composed anterior–posterior canvas.
• Evaluation: Metrics include raw accuracy, balanced accuracy, macro-F1, and class-specific precision/recall/F1. For localization, mean Intersection over Union (IoU), box accuracy (IoU ≥ 0.50), and strict localization accuracy are reported. Grounded VQA performance is measured via strict grounded accuracy, which credits a prediction only if both the answer and the box are correct.

Key Contributions

1. Task Definition: The paper defines paired anterior–posterior VQA, image localization, and grounded VQA tasks that operate across bone-region, lesion, and whole-body levels using original-image locations.
2. Provenance Preservation: A rigorous release boundary is established where every instance is traceable to source annotations via recorded links and hashes, ensuring no clinical evidence is generated de novo.
3. Methodological Baselines: Parameter-efficient training protocols are specified for two distinct model families, explicitly reporting differences in input formats and selection procedures.
4. Metric Separation: The study reports class-balanced and localization metrics that decouple raw answer accuracy from minority-class performance and evidence-level behavior.

Results
Evaluation on the held-out test set reveals a significant dissociation between answer correctness and evidence grounding:

• Answer Accuracy: Both models achieved high raw answer accuracy (>0.90) on VQA and Grounded VQA tasks.
• Balanced Performance: Despite high raw accuracy, balanced accuracy remained near 0.53 (0.528 for Qwen3-VL and 0.535 for PaliGemma on VQA), indicating performance close to random guessing for the minority class.
• Minority Class Performance: The F1 score for the "Abnormal" class remained below 0.15 for all configurations.
• Localization: Models demonstrated competence in spatial layout, achieving mean IoU around 0.64 and box accuracy between 0.78 and 0.86.
• Grounded Accuracy: Strict grounded accuracy (requiring both correct answer and box) was 0.765 for Qwen3-VL and 0.701 for PaliGemma. This gap between box accuracy and strict grounded accuracy highlights that models often localize correctly but fail to generate the correct classification for the minority class.

Significance and Claims
The paper claims that ScintiGround-38K serves as a methodological resource to demonstrate that high raw answer accuracy can conceal weak minority-class and evidence-level behavior in medical VQA. The authors assert that answer correctness and evidence localization are separable capabilities in this modality; a benchmark measuring only the former would incorrectly characterize these systems as robust.

The work emphasizes that the dataset is not intended for clinical diagnosis, treatment selection, or patient-care deployment, as it has not undergone clinical validation and relies on specialist image review labels rather than pathological ground truth. The primary contribution is the provision of a tool to measure the alignment between answers and image evidence, highlighting the necessity of balanced metrics and strict grounding evaluation in medical AI research. Future directions identified by the authors include establishing non-image control baselines, addressing class imbalance through stratified reporting, and conducting external validation on independent cohorts.