Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence

Technical Summary: Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence

Problem Statement
While Large Language Models (LLMs) have demonstrated proficiency in Polish medical text-only examinations (e.g., PES, LEK, LDEK), a significant gap remains in evaluating their ability to process multimodal medical data. Visual information is a critical component of real-world clinical tasks and specific Polish Board Certification Examination (PES) items, yet existing benchmarks for Polish medical AI are limited to text-only inputs. Furthermore, non-English medical Visual Question Answering (VQA) remains underexplored, and performance in English cannot be assumed to transfer to Polish due to domain-specific terminology and local examination formats. A key concern in VQA research is whether models genuinely utilize visual evidence or rely on linguistic shortcuts and answer-choice artifacts.

Methodology
The authors introduce a new benchmark derived from the Polish Board Certification Examination (PES), designed for physicians and dentists pursuing specialist certification.

• Dataset Construction: The team collected 363 examination sheets (2023–2026) from the Medical Examination Center (CEM). They curated a VQA subset of 286 image-containing questions and a QA control subset of 480 text-only questions from the same specialties to ensure comparability.
• Question Categorization: VQA questions were categorized by image importance:
  • Category 0: Image non-essential (text sufficient).
  • Category 1: Image and text complementary.
  • Category 2: Image dominant (answer relies primarily on visual interpretation).
    Visual domains were also classified (e.g., Radiological images, Waveforms, Plots, Graphics).
• Models Evaluated: Nine models were tested, including Polish-oriented open-weight VLMs (LLaVA-Bielik, LLaVA-PLLuM), general-purpose open-weight models (Qwen3.5/3.6, Gemma-4), and commercial models (GPT-5.4-nano, GPT-5.6-sol).
• Input Configurations: To assess visual grounding and reliance on shortcuts, models were evaluated under four controlled input settings:
  • C: Answer choices only.
  • C+Q: Answer choices + Question text.
  • C+I: Answer choices + Image.
  • C+Q+I: Full input (Choices + Text + Image).
• Evaluation Metrics: Accuracy was measured against official answer keys. A subset of questions included human examinee responses for baseline comparison. Data contamination was assessed using the Data Contamination Quiz (DCQ) framework.

Key Contributions

1. Benchmark Creation: The first Polish-language medical VQA benchmark based on specialist-level certification exams, accompanied by a text-only QA control set.
2. Ablation Analysis: A systematic evaluation of model performance under incomplete input configurations to isolate the contribution of answer choices, textual context, and visual evidence.
3. Comparative Analysis: A direct comparison of model performance on image-containing VQA questions versus text-only QA questions within the same medical domain, alongside a comparison with human examinee performance.

Results

• Overall Performance: The task remains challenging. The best model achieves 79.0% accuracy on the full VQA set (specifically GPT-5.6-sol with medium reasoning effort on the full set is 77.78%, while the abstract notes 79.0% as the best model achievement on the full set; however, Table 3 shows GPT-5.6-sol (Y) at 77.78% on VQA). Only GPT-5.6-sol surpasses the approximate human reference (70.14%) on the subset with available human responses; all other evaluated models perform worse than humans.
• Visual Grounding: Models consistently derived more useful information from the question text than from the image. In ablation studies, removing the image (C+Q) resulted in higher accuracy than removing the question text (C+I). Models performed significantly worse on "image-dominant" questions (Category 2) compared to text-sufficient or complementary questions, even when the full multimodal input was provided.
• Answer Choice Reliance: Across both QA and VQA tasks, models achieved above-chance accuracy (random guessing is 20%) using only the answer choices (Configuration C). This suggests that models exploit artifacts, priors, or intrinsic errors within the options themselves, rather than relying solely on the clinical scenario.
• Model Hierarchy: Larger models generally outperformed smaller ones. Among open-weight models, Qwen3.5-397B-A17B and Gemma-4-31B-it achieved passing scores (>60%), but commercial models (GPT-5.6) significantly outperformed open-source counterparts.

Significance and Claims
The paper claims that current Vision-Language Models (VLMs) underutilize visual evidence in the context of Polish medical examinations. The findings indicate that:

• High benchmark accuracy does not necessarily reflect robust multimodal competence, as models can achieve non-trivial performance even when key components (images or questions) are missing.
• Models rely heavily on textual cues and answer-choice patterns rather than genuine visual reasoning, particularly in image-dominant scenarios.
• The dataset serves as a rigorous test bed for specialist-level medical knowledge, highlighting a gap between current AI capabilities and human clinical performance in multimodal settings.

The authors emphasize that while these models show promise in medical education and decision support, their limitations in visual grounding and reliance on shortcuts necessitate caution. They should not be viewed as equivalent to clinicians, especially given that clinical practice requires dynamic information gathering and decision-making under uncertainty that static examination items cannot fully capture.