💻 computer science

NAST: Improving Negation Handling in Medical Vision-Language Models through Negation-Aware Selective Training

This paper introduces Negation-Aware Selective Training (NAST), an interpretability-guided fine-tuning method that leverages causal tracing effects to selectively modulate layer-wise updates, thereby significantly improving medical vision-language models' ability to distinguish between affirmative and negated clinical statements without compromising general alignment.

Original authors: Ali Abbasi, Mehdi Taghipour, Rahmatollah Beheshti

Published 2026-08-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ali Abbasi, Mehdi Taghipour, Rahmatollah Beheshti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the high-stakes world of modern medicine, doctors rely on a delicate balance between what they see and what they read. When a radiologist examines a chest X-ray, they are not just looking for the presence of disease; they are equally focused on the absence of it. A report stating "no pneumonia" is a critical finding that clears a patient for discharge, while "pneumonia is present" triggers immediate treatment. For decades, computers have struggled to learn this distinction. Artificial intelligence systems designed to read medical images and understand text, known as vision-language models, have become remarkably good at matching pictures to descriptions. They can look at a scan and say, "This shows a broken bone." However, these systems often stumble when the language flips to describe what is missing. They tend to treat the absence of a disease as if it were the presence of one, a confusion that could lead to dangerous medical errors. This gap in understanding negation—the linguistic tool we use to say something is not there—has remained a stubborn blind spot for machines, even as they have grown smarter at other tasks.

A team of researchers set out to solve this specific problem, driven by the realization that current medical AI systems are systematically biased toward positive statements. They discovered that when these models are shown a chest X-ray and asked to choose between a description that says "no pneumonia" and one that says "healthy lungs," they frequently fail to recognize that these two phrases mean the exact same thing in a clinical context. Instead, the models often ignore the negation and guess based on the visual features alone, assuming that because the image looks like a healthy lung, the text must be positive. To prove this was a genuine flaw and not just a random error, the researchers built a specialized testing ground called MedNeg-Bench. This benchmark presented the AI with pairs of questions that were identical in every way except for one crucial detail: one question used a negative phrase like "no edema," while its partner used a positive phrase like "normal fluid levels." The results were stark. Across a wide range of existing medical AI models, the systems performed significantly worse on the negative questions, revealing a deep-seated inability to process the concept of "not" when applied to medical images.

Having identified the problem, the researchers realized that simply feeding the AI more examples of negative sentences would not be enough. In the complex language of medicine, negation is rarely just about something being absent; it is often about where it is absent or how severe it is. A report might say "no large pleural effusion," implying a small one might be there, or "no consolidation in the right lung," leaving the left lung as a possibility. To teach the AI this nuance, the team created a new training dataset called MedNeg-FT. This dataset was constructed by taking structured medical facts and systematically altering them to create "what if" scenarios. They took a confirmed fact, such as "severe edema in the left lung," and generated variations that changed the location, the severity, or the existence of the condition, creating millions of paired examples where the only difference was a single attribute. This allowed the AI to learn that changing a word from "severe" to "mild" or from "present" to "absent" fundamentally changes the meaning, even if the picture looks similar.

The core innovation of the study, however, was not just the data, but the method used to teach the AI how to learn from it. The researchers developed a technique called Negation-Aware Selective Training, or NAST. Instead of forcing the entire AI brain to relearn everything from scratch, they used a method of causal tracing to figure out exactly which parts of the model's internal processing were responsible for understanding negation. They found that the ability to process "no" or "not" was concentrated in specific early layers of the model's text-processing network, rather than being spread evenly throughout. Using this insight, they adjusted the training process to focus the learning effort on those specific layers. It is similar to a student who struggles with a particular type of math problem; rather than making them re-read the entire textbook, a tutor identifies the specific chapter where the confusion lies and focuses the practice there. By directing the AI's attention to the layers that mattered most for negation, the researchers could improve the model's sensitivity to negative statements without disrupting its ability to understand positive ones.

The results of this targeted approach were compelling. When tested on the new benchmark, the AI models trained with NAST showed a dramatic improvement in distinguishing between affirmative and negated statements. The gap between how well they understood "pneumonia is present" versus "no pneumonia" shrank significantly, indicating that the models had finally learned to respect the linguistic operator that negates meaning. Crucially, this improvement did not come at the cost of their general performance. The models remained just as good at matching images to positive descriptions as they were before, proving that the learning was precise and did not degrade their overall knowledge. The study also demonstrated that this method worked across different types of medical AI architectures, suggesting that the way these machines process negation is a shared feature that can be fixed with the right guidance.

This work highlights a fundamental truth about artificial intelligence in healthcare: scaling up data alone is not enough to fix deep reasoning errors. The researchers showed that simply adding more examples of negated sentences to a training set did not solve the problem as effectively as understanding how the model processes information. By using a technique that maps the internal cause-and-effect relationships within the AI, they were able to make a surgical improvement to the system's logic. The findings suggest that for AI to be truly safe and reliable in medicine, it must be able to handle the subtle, critical distinctions of human language, including the power of a single word to reverse a diagnosis. The study concludes that by leveraging these internal signals, we can guide models to become more robust and trustworthy, ensuring that when a machine says a disease is absent, it truly understands what that means.

Technical Summary: NAST – Improving Negation Handling in Medical Vision–Language Models

Problem Statement

Vision–language models (VLMs) have demonstrated strong capabilities in medical image understanding, yet they exhibit a systematic failure in distinguishing affirmative from negated medical statements. In clinical radiology, negation (e.g., "no pneumonia" or "no pleural effusion") is critical for ruling out pathologies and guiding treatment. However, current medical VLMs often rely on shallow correlations between visual features and textual tokens, leading to an "affirmative bias" where models perform significantly better on affirmative descriptions than on semantically equivalent negated descriptions.

Existing benchmarks and datasets often fail to isolate this specific limitation because they do not provide tightly controlled pairs of affirmative and negated statements that share identical visual contexts and semantic meaning. Furthermore, standard fine-tuning approaches often treat negation as a simple binary absence, failing to capture the attribute-level granularity (e.g., location, severity) inherent in clinical negation.

Methodology

The paper proposes Negation-Aware Selective Training (NAST), a framework designed to improve negation sensitivity in CLIP-based medical VLMs through two primary components: a specialized dataset and an interpretability-guided optimization strategy.

1. MedNeg-Bench: A Diagnostic Benchmark

To rigorously evaluate negation handling, the authors constructed MedNeg-Bench, a radiology-specific benchmark.

Construction: Derived from MIMIC-CXR and CheXpert, the benchmark generates paired multiple-choice questions (MCQs). For each chest X-ray, the system creates two sets of options: one containing explicit negations (e.g., "No pneumothorax") and an affirmative counterpart with semantically equivalent phrasing (e.g., "Fully expanded lungs").
Control: The pairs are identical in structure and visual context, differing only in polarity. This design isolates the model's ability to process negation from general vision-language alignment issues.
Validation: All mappings were reviewed by board-certified radiologists to ensure clinical validity.

2. MedNeg-FT: A Contextual Negation Dataset

To enable learning beyond simple condition absence, the authors introduced MedNeg-FT, a fine-tuning dataset based on structured radiological claims.

Granularity: Unlike general negation datasets, MedNeg-FT encodes negation across specific attributes: condition, location (e.g., left vs. right lung), and severity (e.g., mild vs. severe).
Generation: The dataset is generated by taking structured clinical facts and creating counterfactuals by minimally altering exactly one attribute (e.g., changing "severe edema" to "no edema" or "edema in the right lung" to "edema in the left lung"). This yields approximately one million image-text pairs.

3. NAST: Causal Tracing-Guided Optimization

NAST is an adaptation method that uses Causal Tracing Effects (CTEs) to modulate gradient updates during fine-tuning.

Causal Probing: The authors use a causal tracing approach (inspired by Meng et al., 2022) to identify which layers in the text encoder are most responsible for processing negation. By intervening on hidden states and measuring the impact on similarity scores between correct and foil captions, they compute a CTE score for each layer and token position.
Layer-Wise Analysis: Experiments reveal that negation-sensitive signals are concentrated in early layers (0–3) of the CLIP text encoder, with a secondary re-emergence in later layers at the end-of-sequence token.
Selective Training: Instead of updating all parameters equally, NAST scales the gradient updates for Low-Rank Adaptation (LoRA) modules based on their layer's CTE score.
- Layers with high causal contribution to negation receive larger updates.
- Layers with low contribution receive smaller updates.
- This is achieved via a scaling factor $\alpha_\ell$ derived from normalized CTE scores and a sharpness hyperparameter $\beta$ .
Loss Function: The training objective combines standard contrastive loss (for single-caption pairs) and a claim-ranking loss (for multiple-choice sets), both modulated by the CTE-based weights.

Key Contributions

Diagnostic Benchmark (MedNeg-Bench): The introduction of a radiology-specific benchmark that isolates polarity sensitivity, revealing a consistent affirmative bias across state-of-the-art medical VLMs (e.g., BioMedCLIP, MedCLIP, BioViL-T).
Contextual Dataset (MedNeg-FT): A large-scale fine-tuning dataset that captures attribute-level negation (location, severity), moving beyond simple presence/absence to reflect real clinical reporting structures.
NAST Framework: An interpretability-guided adaptation method that utilizes causal tracing to selectively modulate gradient updates. This approach improves negation discrimination without degrading general vision-language alignment.
Empirical Validation: Demonstration that mechanistic interpretability signals can serve as a principled optimization rule for targeted model adaptation in safety-critical medical settings.

Experimental Results

The authors evaluated NAST on MedNeg-Bench and MedNeg-FT against several baselines, including standard CLIP, prompt-based negation handling, and other negation-aware models (NegCLIP, ConCLIP, NegBench).

Contextual Negation Tasks: NAST achieved the highest performance across all metrics (Recall@1, Recall@5, and Claim Accuracy). For instance, on Claim Accuracy, NAST reached 55.6%, outperforming the next best model (NegBench at 48.7%).
Affirmative–Negation Gap: On the independent MedNeg-Bench, NAST reduced the performance gap between affirmative and negated formulations to 2.8%, significantly lower than the baseline CLIP gap of 15.4%. This indicates improved robustness to negation rather than a shift in overall calibration.
Generalization: NAST maintained strong performance on non-negated (affirmative-only) captions, demonstrating that the selective training did not degrade general vision-language alignment.
Cross-Architecture: The method generalized effectively to other contrastive medical VLMs (PubMedCLIP and BioMedCLIP), suggesting that layer-wise causal signals for negation are shared across architectures.
Zero-Shot Transfer: In zero-shot classification on the CheXpert dataset, NAST improved the F1 score for samples with absent (negated) conditions more significantly than the overall F1 score, confirming better recognition of absent findings.

Significance and Claims

The paper claims that NAST demonstrates the value of causal interpretability for targeted model adaptation. Specifically:

Clinical Reliability: Correctly modeling negation is critical for clinical safety, as small linguistic variations in polarity correspond to significant differences in diagnosis (e.g., ruling out pneumothorax).
Beyond Data Scaling: The results suggest that simply scaling data is insufficient to address clinically important reasoning gaps; targeted training strategies based on model internals are necessary.
Parameter Efficiency: The approach shows that improving clinically relevant reasoning does not require full model retraining. Selective, parameter-efficient adaptation (via LoRA guided by CTEs) yields substantial gains, offering a practical pathway for updating deployed models in healthcare environments where computational and regulatory constraints exist.
Mechanistic Insight: The work highlights that negation processing in CLIP-based models is localized to specific layers (primarily early layers), and leveraging this structural insight allows for more effective optimization than uniform fine-tuning.

The authors note limitations, including the exclusive focus on contrastive (CLIP-style) architectures and chest X-ray data, suggesting that future work is needed to extend these findings to generative models and other imaging modalities.