CXR-ContraBench: Benchmarking Negated-Option Attraction in Medical VLMs

Technical Summary: CXR-ContraBench

Problem Definition: Negated-Option Attraction
This paper identifies a specific, clinically critical failure mode in Medical Vision-Language Models (MVLMs) termed negated-option attraction. While MVLMs often demonstrate high aggregate accuracy on radiology benchmarks, they exhibit a systematic "polarity reversal" when presented with multiple-choice questions containing both a finding (e.g., "Consolidation") and its explicitly negated counterpart (e.g., "No consolidation").

The core failure occurs when the model selects the negated option despite visual evidence supporting the positive finding. This is not merely a classification error but a presence-side semantic reversal, where the model emits a clinical statement directly contradicting the image (e.g., stating "No consolidation" when consolidation is visible). The authors argue that standard aggregate accuracy metrics mask this failure, as the model may still identify the correct concept in other contexts but fails specifically when the answer space includes a negated surface form. Furthermore, the paper notes that Chain-of-Thought (CoT) prompting, often assumed to improve reasoning reliability, fails to consistently resolve this issue and can sometimes amplify contradictions, particularly in absence-side diagnostic tasks.

Methodology: CXR-ContraBench and QCCV-Neg
To diagnose and quantify this failure, the authors introduce CXR-ContraBench (Chest X-Ray Contradiction Benchmark). Unlike general medical VQA benchmarks, CXR-ContraBench is a diagnostic instrument designed with controlled answer spaces to isolate polarity confusion.

• Benchmark Construction: The benchmark spans three protocol families:

  1. Internal ReXVQA Slices: Large-scale in-distribution data (up to 30k samples) to test sparsity and scale.
  2. OpenI Retrospective Audits: Derived from OpenI reports and CheXStruct annotations, including a "presence" subset (where the finding is visible but a negated option exists) and an "absence" subset.
  3. External Direct Protocols: Dedicated tests using CheXpert validation and training splits. These include a Direct Presence Probe (asking "Which finding is present?") and Report-Style Absence Protocols (asking "Which finding is absent?").
  • Design: Each question pairs a target concept with its negated form and at least one positive distractor. This structure makes "polarity-confused" predictions (selecting the negated option when the positive is the gold standard) directly countable.

• QCCV-Neg (Question-Conditioned Consistency Verifier for Negation): The authors propose a deterministic, post-hoc verifier that operates without retraining. It triggers only when four strict conditions are met:

  1. The answer set contains exactly one explicitly negated option.
  2. The baseline model predicts that negated option.
  3. The question polarity (presence vs. absence) is safely identifiable.
  4. A unique positive counterpart for the concept exists in the remaining options.
     If triggered, the verifier deterministically remaps the prediction to the positive counterpart. This approach aims to audit and repair the specific subset of polarity-confused errors without altering the model's general reasoning.

Key Results
The evaluation covers multiple backbones, including MedGemma (4B and 27B), Qwen2.5-VL, LLaVA variants, CheXagent, and GPT-4o.

• Systematic Failure: On the direct CheXpert presence probe, strong models exhibit severe failure rates. MedGemma-4B and Qwen2.5-VL achieve only 31.49% and 30.21% accuracy, respectively, with over 153 dangerous "No X" reversals each. Even MedGemma-27B drops to 51.91% accuracy with 105 reversals.
• Scale Confirmation: The failure is not an artifact of small validation sets. On a matched CheXpert training-split protocol with 135,754 records, MedGemma and Qwen2.5-VL still emit negated-option selections on over 62% of presence questions.
• Ineffectiveness of CoT: Chain-of-Thought prompting reduces the frequency of reversals slightly for some models on presence probes but does not eliminate them. Crucially, CoT amplifies errors on absence-side protocols; for example, LLaVA-Med's contradictions on CheXpert rose from 42 to 446 with CoT.
• Deterministic Repair: The QCCV-Neg verifier successfully repairs the measured polarity-confused subset without retraining.
  • For MedGemma-4B on the direct presence probe, accuracy jumps from 31.49% to 96.60% (repairing all 153 reversals).
  • For Qwen2.5-VL, accuracy rises from 30.21% to 95.32%.
  • On the large-scale training split, QCCV-Neg removes all measured contradictions, raising accuracy to ~95.6% for both models.
• Robustness: The failure persists across non-canonical negation paraphrases (e.g., "Absence of X," "No evidence of X"). While the original literal-surface verifier loses coverage on these variants, a paraphrase-aware extension of the verifier restores coverage and eliminates contradictions without worsening performance.

Significance and Claims
The paper claims that CXR-ContraBench exposes a "silent error" where aggregate accuracy hides clinically meaningful inference-time polarity failures. The primary contribution is the isolation of negated-option attraction as a distinct, systematic failure mode in generative MVLMs that is not reliably fixed by scaling or CoT prompting.

The authors emphasize that their findings are modest in scope but high in clinical relevance:

1. Measurability: They demonstrate that this specific failure is countable and auditable through controlled answer spaces.
2. Repairability: They show that a significant portion of these errors can be deterministically repaired at inference time without model retraining, suggesting the error lies in answer-space polarity handling rather than a fundamental lack of visual understanding.
3. Limitations: The authors explicitly state that QCCV-Neg is a diagnostic mitigation, not a deployment guarantee. It targets a specific answer-space conflict and does not imply the model has acquired robust, transferable clinical negation reasoning. The paper concludes that future work should investigate the cross-modal representation geometry of polarity, but currently, the benchmark serves to make these specific, dangerous errors visible.