Technical Summary: DoubleTake – Contrastive Reasoning for Faithful Decision-Making in Medical Imaging

1. Problem Statement

Accurate decision-making in medical imaging often fails not because a model misses a finding, but because it confuses two visually similar conditions. While modern vision-language models excel at coarse recognition, they struggle in "confusion-heavy" regimes where images share global structure, modality, and anatomical context but differ only in subtle, localized cues.

Existing retrieval-augmented approaches typically rely on nearest-neighbor search in embedding spaces to retrieve relevant examples. The paper argues this approach is fundamentally misaligned with discriminative decision-making:

• Redundancy: Nearest-neighbor retrieval optimizes for similarity, often returning near-duplicate images from the same document or depicting the exact same condition.
• Hypothesis Reinforcement: Instead of clarifying decision boundaries, redundant evidence reinforces a single dominant hypothesis, suppressing alternatives and increasing "confusion rates" (assigning the same prediction to visually distinct but similar pairs).
• Benchmark Limitations: Standard image-level accuracy metrics obscure these failures. Benchmarks like MediConfusion evaluate performance on pairs of confusable images requiring different answers, where "set-level accuracy" (correctly classifying both images in a pair) is the critical metric.

2. Methodology

The authors propose DoubleTake, a two-stage framework comprising Contrastive, Document-Aware Reference Selection and Counterfactual-Contrastive Inference (CCI).

2.1. Contrastive, Document-Aware Reference Selection

Rather than retrieving the most similar images, the system constructs a compact evidence set (a "triad") optimized for discrimination. The selection process uses the ROCO dataset (a large-scale medical image-caption corpus) and involves:

• Deduplication & Gating: Filtering out near-duplicates (similarity > 0.99) and gating candidates by modality (e.g., MRI vs. CT) based on question text.
• Triad Construction: Three specific reference images are retrieved per query:
  1. Anchor ($r_1$): The most similar non-duplicate candidate. Serves as a stable visual baseline.
  2. Hard Negative ($r_2$): Selected from a mid-similarity band (ranks 20–200). It is the candidate most distinct from the anchor in embedding space, forcing the model to probe subtle discriminative dimensions.
  3. Boundary Probe ($r_3$): Selected from a wider band (ranks 200–1000). It is scored by a combination of query relevance, dissimilarity to the anchor, and lexical overlap with the question. This tests robustness against broader, plausible alternatives.
• Document Awareness: The selection protocol explicitly attempts to enforce distinct document provenance ($d_i \neq d_j$) to avoid reinforcing a single source's bias.

2.2. Counterfactual-Contrastive Inference (CCI)

Once the triad is retrieved, CCI aggregates structured pairwise visual comparisons using a confidence-aware reasoning framework:

• Structured Pairwise Comparison: For each reference $r_i$, a vision-language model compares the query $x$ against $r_i$, outputting a predicted answer ($A$ or $B$), a self-reported confidence score ($\alpha_i \in [0, 100]$), and a structured evidence record ($e_i$).
• Confidence-Weighted Aggregation: Votes are weighted by confidence. A margin-based decision rule is applied:
  • Abstention: If total evidence mass ($W$) is below a threshold $t$, or if the decisiveness margin ($M = |W_A - W_B|$) is below a threshold $m$, the system abstains ($\perp$).
  • Decision: If $W \ge t$ and $M \ge m$, the answer with the higher weighted score is selected.
• Adjudication Mechanisms:
  • Text Adjudicator: If the margin is small despite sufficient evidence, a text-only module analyzes the structured evidence records to resolve contradictions.
  • Pair-Level Adjudicator: Since MediConfusion evaluates pairs, a joint reasoning step is triggered if both images in a confusion pair receive the same prediction or if either is ambiguous. This prevents the "collapse" to identical predictions across the pair.

3. Key Contributions

1. Contrastive Reference Selection Framework: A novel protocol that treats retrieval as "evidence construction for discrimination" rather than similarity search. It explicitly balances visual relevance, embedding diversity, and source-level provenance.
2. Counterfactual-Contrastive Inference (CCI): A reasoning framework that aggregates pairwise comparisons using confidence filtering, margin-based decision rules, and faithful abstention.
3. Reproducible Resources: The release of a curated reference bank derived from ROCO and a triad selection protocol to enable systematic study of contrastive retrieval in medical imaging.
4. State-of-the-Art Performance: Demonstration that this approach significantly outperforms direct inference and standard retrieval baselines on the MediConfusion benchmark.

4. Experimental Results

Evaluated on the MediConfusion benchmark (176 confusion pairs, 9 medical categories) using various base models (GPT-4o, GPT-o1, Gemini 2.0 Pro):

• Set-Level Accuracy: CCI achieved 43.75% set-level accuracy with Gemini 2.0 Pro, a substantial improvement over the direct single-shot baseline (29.55%) and a ~15% absolute gain over prior methods.
• Confusion Reduction: The confusion rate (assigning the same answer to both images in a pair) dropped from 67.05% (direct inference) to 36.93% (CCI), representing a ~45% relative reduction.
• Ablation Studies:
  • Removing the Hard Negative or Boundary Probe significantly degraded performance, confirming that contrastive structure is essential.
  • A standard Top-k Nearest-Neighbor baseline performed poorly (30.68% set accuracy), reinforcing that similarity-based retrieval alone cannot resolve confusion.
  • Disabling the Pair-Level Adjudicator caused a sharp drop in set-level accuracy (43.75% $\to$ 27.84%), highlighting the necessity of joint reasoning for confusion pairs.
• Reliability: The system maintained high conditional accuracy (64.20%) with a low abstention rate (3.69%), indicating that performance gains were not driven by aggressive deferral.

5. Significance and Claims

The paper claims that the primary limitation in medical image reasoning under ambiguity is not the quality of visual representations, but how evidence is constructed and aggregated at inference time.

• Shift in Paradigm: The work argues for shifting retrieval from a "similarity" operation to a "discrimination" operation. By explicitly constructing evidence sets that probe decision boundaries (via hard negatives and boundary probes) rather than reinforcing dominant hypotheses, models can better distinguish look-alike conditions.
• Faithful Decision-Making: The framework prioritizes "faithful abstention" over forced, confident predictions in ambiguous cases, aligning with clinical needs where confusing conditions is a critical error.
• Systematic Interaction: The results suggest that improvements arise from the interaction between contrastive evidence construction and confidence-aware inference, rather than either component in isolation.

The authors maintain a modest stance, noting that the approach is benchmark-based, relies on the diversity of the external reference bank, and does not claim immediate clinical readiness without extensive validation and human oversight.