💻 computer science

Exploring Prompt Alignment with Clinical Factors in Zero-Shot Segmentation VLMs for NSCLC Tumor Segmentation

This study demonstrates that in zero-shot vision-language models for NSCLC tumor segmentation, anatomical location is the dominant driver of spatial attention, enabling a model like VoxTell to achieve performance comparable to fine-tuned baselines while highlighting the critical need to evaluate segmentation models based on their alignment with specific prompt dimensions rather than Dice scores alone.

Original authors: Suraj Pai, Thibault Heintz, Cosmin Ciausu, Marion Tonneau, Hugo Aerts, Raymond Mak

Published 2026-05-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Suraj Pai, Thibault Heintz, Cosmin Ciausu, Marion Tonneau, Hugo Aerts, Raymond Mak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but brand-new, robot assistant how to find a specific tumor in a lung scan. Usually, to get a robot to do this, you have to spend months showing it thousands of examples until it memorizes exactly what to look for. This is like training a dog with endless treats.

But this paper explores a different approach: using a Vision-Language Model (VLM). Think of this model as a robot that has already read millions of medical books and seen millions of scans. Instead of retraining it, you just talk to it. You give it a text prompt (a description) and ask, "Find the tumor." This is called "zero-shot" learning because the model hasn't seen this specific patient before, but it tries to figure it out based on your words.

The researchers wanted to know: What exactly in your words makes the robot point to the right spot?

The Experiment: Playing with the Recipe

The team used a model called VoxTell and tested it on 93 lung cancer cases. They treated the text prompts like a recipe, changing one ingredient at a time to see how the robot reacted.

The "Where" vs. The "What":
They found that the most important ingredient in the recipe is location.
- Analogy: Imagine asking the robot, "Find the red ball." If you say "Find the red ball in the kitchen," it looks in the kitchen. If you accidentally say "Find the red ball in the garage," the robot looks in the garage and finds nothing (or the wrong thing).
- The Result: When the researchers changed the location in the text (e.g., from "left lung" to "right lung"), the robot's performance crashed. It completely lost its way. However, if they changed the type of cancer (e.g., from "adenocarcinoma" to "squamous cell") or the stage of the disease, the robot barely noticed. It didn't care much about the medical label; it cared deeply about where to look.
The Specificity Ladder:
They tested how much detail the robot needed.
- Level 1 (Vague): "Tumor." -> The robot guessed, but wasn't great.
- Level 2 (Better): "Lung tumor." -> Much better.
- Level 3 (Best): "Tumor in the upper lobe of the left lung." -> The robot nailed it.
- The Twist: Interestingly, just giving the robot a medical diagnosis (like "Stage 3 Adenocarcinoma") without saying where it is actually made the robot perform worse than just saying "Lung tumor." The robot needs a clear map, not just a medical label.
The "Wrong Patient" Test:
They tried giving the robot a picture of Patient A but the text description of Patient B.
- The Result: The robot realized something was wrong. It either found nothing or found the wrong thing. This proves the robot isn't just blindly looking for "any tumor" in any picture; it is actually listening to the specific details of the prompt to decide what to do.
The "Nonsense" Test:
They asked the robot to find a "liver cyst" in a lung scan.
- The Result: The robot correctly said, "I see nothing here," and drew no lines. It knew that a liver cyst doesn't belong in a lung.

How Did It Compare to the Experts?

The researchers compared this "talk-to-the-robot" method against two other groups:

The "Specialists" (Fine-tuned models): These are robots trained specifically on thousands of lung scans. They are the gold standard.
The "Generalists" (Other zero-shot models): These are robots that haven't been trained on lung scans specifically and just try to guess based on general knowledge.

The Surprise: The "talk-to-the-robot" method (VoxTell) performed almost as well as the highly trained "Specialists." It was significantly better than the other "Generalist" robots.

The Big Takeaway

The paper concludes that for these smart medical robots, location is king.

The robot prioritizes "Where to look" over "What to look for."
If you tell it the wrong place, it fails.
If you tell it the right place, it can find the tumor even if you don't give it a perfect medical diagnosis.

The authors argue that when we test these AI tools, we shouldn't just ask, "Did it find the tumor?" We also need to ask, "Did it listen to the right words?" If the robot only listens to the location and ignores the complex medical details, that's a specific behavior we need to understand before we trust it in a real hospital.

In short: You can get a highly accurate tumor map just by telling the AI exactly where to look, without needing to retrain it for every single new patient. But if you get the location wrong in your sentence, the robot gets lost.

Technical Summary: Exploring Prompt Alignment with Clinical Factors in Zero-Shot Segmentation

1. Problem Statement

Accurate delineation of Gross Tumor Volume (GTV) is critical for radiotherapy planning in non-small-cell lung cancer (NSCLC), yet manual contouring is time-consuming and exhibits significant inter-annotator variability. While deep learning has reduced contouring times, current segmentation paradigms face a gap in interpretability regarding promptable zero-shot Vision-Language Models (VLMs).

Existing segmentation approaches fall into three categories:

Fine-tuned supervised models (e.g., nnUNet).
Zero-shot foundation models (e.g., TotalSegmentator, VISTA3D).
Zero-shot VLMs that accept volumetric imaging and free-text clinical prompts.

While benchmark performance (e.g., Dice Similarity Coefficient) is standard, it is insufficient for clinical deployment. It remains unclear which specific attributes within a clinical prompt (diagnosis, staging, demographics, anatomical location) actually drive the spatial behavior of zero-shot VLMs. A clinically useful VLM must not only achieve high accuracy but also demonstrate that it suppresses output for irrelevant prompts, improves with spatially informative prompts, and fails appropriately when patient-specific information is mismatched, rather than acting as a generic tumor detector.

2. Methodology

The study utilized a held-out internal dataset, HarvardRT, comprising 93 NSCLC cases with planning CTs and expert GTV contours. Free-text clinical prompts were derived from institutional notes, preserving diagnosis, demographic, and staging language.

Experimental Design

The primary focus was on VoxTell, the best-performing zero-shot VLM in the initial screening. The authors conducted four specific alignment analyses on a subset of cases:

Sub-prompt Decomposition: Original prompts were fragmented into controls (diagnosis, demographics, TNM stage, anatomical location, generic "lung tumor," and irrelevant "liver cyst").
Perturbation Robustness: Sentence structures were preserved while swapping single attributes (tumor type, stage, age, sex, location) to measure the change in Dice Similarity Coefficient ( $\Delta$ DSC).
Specificity Ladder: Prompts were varied from Level 0 (generic "tumor") to Level 6 (full prompt plus fabricated detail) to test if increasing clinical detail improves or destabilizes alignment.
Cross-Case Prompt Swaps: Images were paired with prompts from other patients to distinguish patient-specific conditioning from prompt-agnostic detection.

Benchmarking

Nine models were evaluated across the three paradigms:

Fine-tuned: nnUNet (GTV-only), Ahmed et al. (multi-stage), MM-LLM-RO.
Zero-shot Vision: TotalSeg-Nodules, VISTA3D.
Zero-shot VLM: VoxTell, BiomedParse, SAT, CAT.

Performance was measured using the Dice Similarity Coefficient (DSC). Statistical significance was assessed using Friedman tests for overall heterogeneity and pairwise Wilcoxon signed-rank tests with Benjamini–Hochberg correction for 33 pairwise comparisons.

3. Key Results

Prompt Alignment and Robustness

Anatomical Dominance: Anatomical location is the primary driver of VoxTell's spatial attention. 63.4% of location perturbations caused catastrophic drops in performance ( $|\Delta$ DSC| > 0.5), whereas tumor type and stage substitutions had minimal effects (median $|\Delta$ DSC| = 0.009 for type).
Prompt Specificity: Performance improved monotonically from generic queries to full clinical descriptions, with the largest gain occurring at the transition from organ-level ("lung tumor") to lobe-level specificity. However, "diagnosis-only" prompts (L4) were inconsistent, performing well for common histologies but failing for rare types.
Irrelevant Prompt Suppression: Irrelevant prompts (e.g., "liver cyst") correctly yielded zero segmentation.
Patient-Specific Conditioning: Cross-case swaps confirmed that the model relies on patient-specific information. Matched prompt-image pairs achieved a mean DSC of 0.906, while mismatched pairs dropped to 0.406, with 44% of mismatches producing zero output.

Benchmark Performance

VoxTell vs. Fine-tuned: VoxTell achieved a mean DSC of 0.613. This was statistically indistinguishable from the leading fine-tuned models, nnUNet (0.690, $p_{adj} = 0.156$ ) and Ahmed et al. (0.675, $p_{adj} = 0.679$ ).
Zero-Shot Comparison: VoxTell significantly outperformed all other zero-shot VLMs (BiomedParse, SAT, CAT all had DSC < 0.26) and most zero-shot vision models, except TotalSeg-Nodules (0.653), with which it was not significantly different.
Overall Hierarchy: Fine-tuned models generally led the benchmark, followed by VoxTell and TotalSeg-Nodules. Other zero-shot foundation models performed poorly, likely due to distribution mismatches with lung tumor heterogeneity.

4. Key Contributions

Characterization of Prompt Alignment: The study provides the first systematic analysis of which clinical prompt dimensions drive zero-shot VLM segmentation. It establishes that anatomical location is the dominant alignment direction, while histology and staging are largely ignored by the model's spatial attention mechanisms.
Demonstration of Patient-Specific Conditioning: Through cross-case swaps, the authors prove that VoxTell does not function as a generic tumor detector but rather conditions its segmentation on the specific patient context provided in the prompt.
Performance Parity: The work demonstrates that a fully zero-shot VLM can achieve segmentation accuracy statistically comparable to state-of-the-art fine-tuned supervised models on an independent clinical dataset.
Evaluation Framework: The paper argues that VLMs must be evaluated not only by aggregate accuracy (Dice) but also by their alignment behavior—specifically, their ability to suppress irrelevant inputs and respond to spatial cues.

5. Significance and Claims

The paper claims that segmentation VLMs should be evaluated based on both benchmark accuracy and prompt-alignment analyses. The findings suggest that VoxTell leverages clinical language in a selective manner, prioritizing "where to look" (anatomical location) over "what to look for" (histology/stage). This behavior aligns with architectures where text queries localize regions via cross-attention before fusing with spatial features.

The authors conclude that integrating clinical language understanding with volumetric image analysis is a promising direction for scalable radiotherapy contouring. However, they note limitations, including the single-institution dataset, the use of DSC as the sole metric, and the observation that the model's reliance on anatomical location may limit its applicability to complex targets like nodal volumes, where staging and treatment intent are more critical. The study emphasizes that while VoxTell approaches the performance frontier of fine-tuned models, its clinical utility depends on understanding these specific alignment behaviors.