💻 computer science

CXR-Retrieve: Compositional Text-to-Image Retrieval in Chest Radiography

This paper introduces CXR-Retrieve, a structured benchmark and a label-aware contrastive fine-tuning method that significantly improves compositional text-to-image retrieval in chest radiography by ensuring retrieved images satisfy complex clinical constraints, including conjunctions and negations, rather than merely matching keywords.

Original authors: Tomer Erez, Moshe Kimhi, Chaim Baskin, Ehud Rivlin

Published 2026-07-31

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Tomer Erez, Moshe Kimhi, Chaim Baskin, Ehud Rivlin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of looking for a suspect, you are searching through a massive library of X-ray pictures. In the real world, doctors have millions of these images, but most of them are just sitting there with a long, messy paragraph of text attached to them—a radiologist's report. If you want to find a picture of a broken bone, you can't just ask the computer "show me broken bones." You have to read through thousands of reports to find the right one.

To make this easier, scientists have been teaching computers to understand both pictures and words at the same time. Think of this like teaching a robot to speak "Picture-English." The best teachers for this job are models called Vision-Language Models. They are like super-smart librarians who have read every book and looked at every picture in the library, learning how to match a description to an image. Usually, these librarians are trained to match a whole, long story (a full medical report) to a picture. But what happens if you don't want the whole story? What if you just want a picture that has one specific thing, but definitely not another thing? That is where the current librarians start to get confused. They are great at finding the "story," but they are terrible at following strict, short instructions like "Show me a lung with fluid, but make sure there is no pneumonia."

This is the problem a team of researchers from the Technion and Ben-Gurion University decided to tackle. They realized that while computers are getting better at matching long reports to X-rays, they are failing miserably at answering short, precise clinical questions, especially when those questions involve saying "no" to something or combining two different conditions.

The Problem: The "Yes, But No" Confusion

The researchers found that current AI models have a weird blind spot. If you ask them for an image with "Atelectasis" (a collapsed part of the lung) and "no Pneumonia," the AI often ignores the "no" part. It sees the word "Pneumonia" in your request and thinks, "Oh, they want Pneumonia!" and shows you a picture with both problems. It's like asking a chef for a burger with no cheese, and the chef, hearing the word "cheese," decides to put a giant slice of cheese on it anyway because they are too focused on the word itself rather than the instruction.

The team created a new test called CXR-RETRIEVE to prove this. They built a special challenge with 5,159 X-ray images and 145 different search queries. Some queries were simple ("Show me a fracture"), some were combinations ("Show me a fracture and a lung lesion"), and some were tricky negations ("Show me a fracture but no lung lesion"). They found that the best existing models, which are trained to match full reports, would often get the simple ones right but would completely fail the tricky ones. They would retrieve images that matched the words but violated the logic.

The Solution: Teaching the AI to Listen to "No"

To fix this, the researchers didn't just throw more data at the problem; they changed how the AI learns. They introduced a new way of training called label-aware contrastive fine-tuning.

Imagine you are teaching a dog to fetch. If you say "Fetch the ball," the dog runs to the ball. But if you say "Fetch the ball, but don't fetch the stick," a normal dog might get confused and grab both. The researchers taught their AI a new rule: "If you see a picture that matches the 'ball' part but also has the 'stick' part, you must push it away."

They did this by creating a special "scorecard" for every image and text pair.

The Attraction: If an image and a text query agree on what is present (e.g., both say "Edema is here") and what is absent (e.g., both say "No Pneumonia"), the AI is rewarded for bringing them closer together.
The Repulsion: This is the secret sauce. If the text says "No Pneumonia" but the image actually has Pneumonia, the AI is explicitly punished and told to push that image far away. It's like a "Do Not Touch" sign that actively repels the wrong answers.

They also made sure the AI understood that if a report doesn't mention a disease, it's not necessarily a "yes" or a "no"—it's just "unknown." But if the report explicitly says "No Pneumonia," that is a confirmed fact that the AI must respect.

The Results: A Huge Leap in Logic

When they tested their new method, the results were impressive, especially for the hard questions.

For simple searches: Their model was just as good as the best existing models.
For combined searches (A and B): They improved the accuracy by 8.5 percentage points over the previous best model.
For the tricky "No" searches (A and no B): This was the biggest win. They improved the accuracy by 22.0 percentage points.

To put that in perspective, if the old model could only find the right "No Pneumonia" picture 20 times out of 100, the new model found it 42 times out of 100. They also measured how often the AI made the specific mistake of showing a picture with the forbidden disease (called the Hard Negative Retrieval Rate). Their method dropped this error rate significantly, meaning the AI was much less likely to ignore the "no" in your request.

Why This Matters

The researchers suggest that for medical AI to be truly useful, it can't just be a word-matching machine. It needs to understand clinical logic. It needs to know that "Atelectasis and no Pneumonia" is a completely different request than "Atelectasis and Pneumonia." By teaching the AI to actively repel contradictory images and respect explicit "no" constraints, they showed that we can build systems that actually listen to a doctor's specific, short instructions.

This isn't a magic wand that solves every medical problem yet, but it suggests a clear path forward. If we want computers to help doctors find the right past cases to learn from, we have to teach them not just what words mean, but how those words fit together to describe a patient's reality. The paper concludes that reliable medical image retrieval requires training objectives that model not just which findings are mentioned, but how they are asserted—whether they are present, absent, or uncertain.

Technical Summary: CXR-Retrieve

Problem Statement

Large-scale chest radiography archives are difficult to search effectively because studies are typically paired with free-text reports rather than structured clinical annotations. While Vision-Language Models (VLMs) like CLIP offer a natural interface for text-to-image retrieval, current biomedical models (e.g., CXR-CLIP, BioMedCLIP) suffer from a fundamental objective mismatch. These models are primarily optimized for report-to-image matching, aiming to retrieve a study based on its full radiology report.

This training objective fails to address clinical query retrieval, where a user inputs a short, precise constraint (e.g., "atelectasis and no pneumonia"). Current models exhibit two systematic failure modes:

Compositional Failure: They struggle with multi-pathology conjunctions, often retrieving images containing only one of the requested findings.
Negation Failure: Due to an affirmation bias, models incorrectly link negated text tokens (e.g., "no pneumonia") to positive visual features, retrieving images that explicitly contain the forbidden pathology.

Existing metrics based on report-level matching hide these failures, as a retrieved image might match the report textually without satisfying the specific clinical constraints of the query.

Methodology

The authors propose CXR-RETRIEVE, a structured benchmark and a corresponding training framework designed to treat retrieval as a constraint satisfaction problem over clinical findings.

1. The CXR-RETRIEVE Benchmark

Derived from the MIMIC-CXR-JPG dataset, the benchmark consists of:

5,159 test images from the official test split.
145 textual queries spanning three regimes:
- Single-pathology (e.g., "Edema").
- Pairwise conjunctions (e.g., "Edema and consolidation").
- Negation queries (e.g., "Edema and no consolidation").
Relevance Definition: An image is relevant only if it satisfies all asserted constraints (positive findings and explicit exclusions) based on structured pathology labels (CheXpert), rather than matching a paired report.

2. Label-Aware Contrastive Fine-Tuning

The core method involves fine-tuning a dual-encoder CLIP model (initialized from CXR-CLIP) using a novel loss function that prioritizes asserted pathology constraints over token co-occurrence.

Label Kernel ( $K$ ): Instead of comparing raw text embeddings, the model computes a dot product between label vectors derived from the text.
- $+1$ for affirmed findings.
- $-1$ for negated findings.
- $0$ for unmentioned findings (treated as unknown, not absent, to avoid spurious conflicts).
- This kernel rewards shared confirmed absences (crucial for negation learning) and penalizes clinical contradictions (e.g., one caption asserts "present," the other "absent").
Soft Attraction: Pairs with compatible assertions ( $K > 0$ ) are attracted using soft contrastive targets, where the strength of attraction is proportional to the degree of label agreement.
Hard Negative Mining (Repulsion): Pairs with direct clinical contradictions ( $K < 0$ ) are explicitly repelled using a hinge loss. This forces the model to sharply separate images that violate the negation constraint.
Training Data Synthesis: Training captions are synthesized from labels to simulate single, conjunctive, and negation queries. The training mix is balanced (50% single, 25% pair, 25% negation) to ensure robustness across query types.

Key Contributions

CXR-RETRIEVE Benchmark: A structured evaluation suite specifically designed to expose the limitations of current VLMs in handling compositional and negation-aware chest X-ray retrieval.
Conflict-Aware Contrastive Objective: A training method that aligns image-text embeddings based on explicit clinical assertions. It introduces a mechanism to attract compatible profiles (including shared absences) and explicitly repel contradictory pairs.
Comprehensive Evaluation: A detailed comparison against general-domain (CLIP), broad biomedical (BioMedCLIP), and in-domain (CXR-CLIP) baselines, demonstrating that standard report-trained models degrade significantly on complex queries.

Results

The proposed method demonstrates substantial improvements over baselines, particularly on difficult query types:

Conjunctive Queries: Precision@5 improved by 8.5 percentage points over CXR-CLIP (19.6% vs. 11.1%).
Negation Queries: Precision@5 improved by 22.0 percentage points over CXR-CLIP (42.0% vs. 20.0%).
Hard Negative Retrieval Rate (HNRR): The proposed method achieved the lowest HNRR (1.6%), indicating significantly fewer retrievals that violate negation constraints compared to baselines (2.7%–4.9%).
Robustness: The model maintained high performance under paraphrased queries (e.g., "an image with A but without B"), suggesting it learned the concept of negation rather than memorizing specific templates.
Ablation Studies:
- Removing the hard negative mining term resulted in a sharp increase in HNRR (3.7% vs. 1.6%), confirming the necessity of explicit repulsion.
- Initializing from an in-domain checkpoint (CXR-CLIP) was critical for performance on compositional queries; vanilla CLIP initialization performed poorly on these complex tasks.

Significance and Claims

The paper argues that reliable chest X-ray retrieval requires training objectives that model how findings are clinically asserted, not just which findings are mentioned.

Objective Mismatch Resolution: The work highlights that optimizing for full report matching is insufficient for clinical decision support, where users need to retrieve cases satisfying specific, often negative, constraints.
Clinical Logic: The results show that by explicitly modeling label conflicts and shared absences, the model can learn the "clinical logic" binding findings together, rather than relying on superficial token co-occurrence.
Future Directions: The authors note that while this benchmark isolates compositional and negation logic, future work could address less structured queries (abbreviations, regional terminology) and other aspects like spatial awareness and efficiency, which were deliberately excluded to focus on the core logic gap.