🧬 biology

Open 3D-CT Vision-Language Models Do Not Clear a Pre-Registered Biomarker Gate Under Zero-Shot Prompting at n = 96 NSCLC-Radiogenomics

This study demonstrates that while open 3D-CT vision-language model features contain decodable EGFR mutation signals in NSCLC, zero-shot prompting fails to access this information due to a geometric misalignment between the text-prompt axis and the discriminative feature direction, a limitation overcome only by supervised probing.

Original authors: Adil Karim, Amine Berquedich, El Habib Benlahmar, Sanaa Elfilali, Ahmed Zellou

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Adil Karim, Amine Berquedich, El Habib Benlahmar, Sanaa Elfilali, Ahmed Zellou

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you have a super-smart, all-knowing robot librarian named Merlin. This robot has read millions of medical reports and looked at thousands of CT scans (3D X-rays of the inside of the body). The big question researchers asked was: "If we just ask Merlin a simple question in plain English, can it instantly guess a patient's genetic secrets just by looking at their scan, without ever being taught how to do it?"

This is called "zero-shot" learning. It's like handing a genius a new puzzle and asking, "Solve this!" without giving them the instruction manual.

The researchers set up a strict test with 96 patients who had a specific type of lung cancer. They had a pre-registered "gate" (a finish line) that the robot had to cross to prove it worked: it needed to be right more than 65% of the time, with a very high level of confidence.

The Big Reveal: The Robot Got Lost
When the researchers asked Merlin to guess the genetic mutations (specifically EGFR and KRAS) using simple text prompts like "This patient has a mutant tumor," the robot stumbled. It performed no better than flipping a coin.

For the EGFR mutation, the best guess was only about 0.599 (just under 60%).
For the KRAS mutation, it was even lower, around 0.541.
None of the guesses crossed the finish line of 0.65.

In fact, the robot was so confused that on a separate group of 42 patients, its guesses were basically random noise, sitting between 0.394 and 0.500. The paper explicitly rules out the idea that Merlin simply "doesn't know" the answer. The failure wasn't because the robot was blind; it was because the way they asked the question was the wrong key for the lock.

The Twist: The Treasure Was There All Along
Here is the plot twist that makes this story so interesting. The researchers didn't just give up. They took the exact same 3D scans that Merlin had looked at and asked a different kind of question. Instead of asking Merlin to guess based on a text prompt, they gave the data to a tiny, supervised computer brain (a "linear probe") and said, "Hey, look at these numbers and find the pattern."

Suddenly, the robot's "eyes" (the data it had already processed) were wide open.

When this tiny brain looked at the same Merlin data, it guessed the EGFR mutation correctly 0.748 of the time (about 75%).
This easily crossed the finish line of 0.65.

The "Why": A Misaligned Compass
So, why did the text prompt fail while the data analysis succeeded? The authors explain this with a brilliant geometric analogy.

Imagine the robot's brain is a giant 512-dimensional room filled with information. The genetic mutation (EGFR) is a specific direction in that room, like a hidden treasure chest.

The Supervised Brain is like a person with a map who can walk in any direction in the room to find the chest. They found it easily.
The Zero-Shot Prompt is like a flashlight. The researchers shone the flashlight in a specific direction based on the words they typed ("mutant tumor").
The Problem: The direction the flashlight was pointing was almost perfectly sideways (at a 90-degree angle) to the direction of the treasure chest. The paper calculates that the angle between the text prompt and the genetic signal was nearly 0.02 (which is basically random).

The flashlight was shining on a blank wall, while the treasure was sitting right behind the person holding it. The text prompts the researchers used were so similar to each other that the "difference" between them (the direction the robot looked) was tiny and pointed in the wrong place. The robot wasn't missing the data; the data was just hiding in a part of the room the flashlight couldn't reach.

What About the "Smoking" Distractor?
Before the main test, the researchers worried that the robot might just be guessing based on whether the patient smoked, because smoking is often linked to these mutations. They ran a test on a different type of data first, and it looked like smoking was the only thing the robot could see (scoring 0.708).

However, when they tested this on the actual Merlin data used for the main experiment, the story flipped. On the real data, the genetic signal (0.748) was actually stronger than the smoking signal (0.603). This proves that the robot could see the genetics if asked the right way, and the "smoking" worry was only a problem for the first type of data they tried, not the final one.

The Bottom Line
This paper is a cautionary tale for the future of medical AI. It shows that just because a powerful, open-source robot model exists, it doesn't mean we can just ask it a question in English and get a medical diagnosis.

The Verdict: Zero-shot prompting (asking without training) failed to clear the gate for these lung cancer mutations.
The Hope: The data inside the model does contain the answer. The problem is that our current "text prompts" are like trying to open a door with a butter knife instead of a key.
The Confidence: The authors are very sure about this. They didn't just guess; they ran the numbers 1,000 times to make sure the results weren't a fluke. The signal was real, the method was just misaligned.

In short: The robot has the answer, but we haven't figured out how to ask the question yet.

Technical Summary: Open 3D-CT Vision-Language Models Do Not Clear a Pre-Registered Biomarker Gate Under Zero-Shot Prompting at n = 96 NSCLC-Radiogenomics

Problem Statement
The study addresses the feasibility of using open, pre-trained 3D-CT Vision-Language Models (VLMs) for zero-shot radiogenomic prediction in non-small-cell lung cancer (NSCLC). Specifically, it investigates whether these models can predict molecular biomarkers (EGFR mutation, KRAS mutation, TP53 mRNA, STK11 mRNA) via contrastive image-text alignment without any task-specific training on the target cohort. The research operates under a strict pre-registered protocol with a small internal cohort ( $n=96$ ) and an external hold-out cohort ( $n=42$ ), aiming to determine if zero-shot prompting can clear a rigorous performance gate (AUC > 0.65, bootstrap lower CI > 0.55, and external replication) while accounting for potential confounders like smoking history.

Methodology
The authors pivoted from an originally specified model (CT-CHAT/CT-CLIP) due to inaccessible weights, evaluating two available open alternatives:

Merlin: A 3D-inflated ResNet-152 + Clinical-Longformer CLIP-style model trained on abdominal CT volumes paired with radiology reports.
BiomedCLIP: A 2D ViT-B/16 + PubMedBERT contrastive model trained on a 15M biomedical corpus, used here as a 2D key-slice comparator.

Experimental Design:

Cohorts: An internal NSCLC-Radiogenomics cohort ( $n=96$ ) with CT volumes, lesion masks, RNA-seq, and clinical data; and an external cohort ( $n=42$ ) with EGFR labels and clinical data.
Zero-Shot Protocol: Image features (512-d) were extracted from frozen models. These were scored against binary prompt pairs (positive vs. negative) using two strategies: "bare" (direct disease statement) and "cot_brief" (short rationale with imaging cues). The score was the softmax probability of the positive prompt based on the dot product of image and text embeddings scaled by a learned logit temperature.
Confounder Probe (Phase 1): A pre-registered check to see if "never-smoker" status could be predicted better than the biomarkers on a proxy substrate, which would "foreclose" claims of pure biomarker signal.
Supervised Baselines: To distinguish between feature quality and access mode limitations, the authors trained supervised probes (linear, ridge, and tiny-MLP) on the same frozen image features extracted by the VLMs.
Geometric Analysis: The study analyzed the alignment between the text-prompt axis (the direction the zero-shot score projects onto) and the discriminative axis (the direction a supervised probe finds for the biomarker).

Key Results

Zero-Shot Failure: No zero-shot cell cleared the pre-registered gate.
- Internal Cohort: The best result was for STK11_high using Merlin with "cot_brief" prompts (AUC 0.626), which failed the AUC > 0.65 threshold and had a bootstrap lower CI (0.517) below the 0.55 threshold. EGFR and KRAS results were near chance (e.g., EGFR/Merlin/cot_brief: AUC 0.599).
- External Cohort: All zero-shot EGFR results sat between 0.394 and 0.500 (chance or anti-predictive).
- Model Comparison: Merlin (3D) consistently outperformed BiomedCLIP (2D), but neither cleared the gate.
Supervised Success vs. Zero-Shot Null: Crucially, when the same frozen Merlin features were evaluated using a supervised linear probe, the EGFR signal cleared the gate (AUC 0.748, $p=0.010$ ). This demonstrates that the features contain the biomarker signal, but the zero-shot contrastive access mode fails to retrieve it.
Confounder Reversal: The Phase 1 confounder probe, which initially flagged smoking as a dominant predictor on a proxy CT-FM substrate (AUC 0.708), did not hold on the benchmarked Merlin features. On Merlin, the supervised EGFR signal (AUC 0.748) exceeded the smoking signal (AUC 0.603), indicating the confounder foreclosure was substrate-specific.
Geometric Misalignment: The study identified the mechanism of failure: the zero-shot score projects the image embedding onto a single text-defined axis ( $\hat{v} = \hat{t}_{pos} - \hat{t}_{neg}$ ). The EGFR-discriminative direction ( $\hat{w}$ ) found by supervised learning is nearly orthogonal to this text axis (cosine similarity $\approx 0.018$ ). Consequently, projecting onto the text axis discards the biomarker signal, while removing the text axis leaves the signal intact (AUC 0.748).

Key Contributions

Benchmarking Zero-Shot Limits: Provides a reproducible, pre-registered benchmark showing that current open 3D-CT VLMs, under zero-shot prompting at small sample sizes ( $n=96$ ), cannot achieve clinically relevant biomarker prediction for EGFR or KRAS.
Decoupling Features from Access Mode: Demonstrates that a zero-shot null result does not necessarily imply a lack of biological signal in the features. The failure is attributed to the "access mode" (contrastive prompting) rather than the feature quality.
Geometric Diagnosis: Offers a geometric explanation for the failure, showing that the discriminative direction for biomarkers lies in the subspace orthogonal to the text-prompt axis, exacerbated by the near-collinearity of positive and negative prompts and the image-text modality gap.
Confounder Verification: Highlights that confounder relationships (e.g., smoking vs. biomarker) are substrate-specific and must be verified on the actual features used, not just on proxy substrates.

Significance and Claims
The paper claims that while open radiology VLMs possess the underlying feature representations to decode EGFR signals (as proven by supervised probes), the zero-shot contrastive prompting mechanism is structurally incapable of accessing these signals at small cohort sizes. The "zero-shot null" is a limitation of the prompting interface, not the model's learned representation. The authors conclude that for small-cohort radiogenomics, zero-shot prompting currently yields performance near chance (AUC 0.55–0.60) for EGFR/KRAS, and that the discriminative information resides in directions of the embedding space that standard text prompts do not align with. The study advocates for interpreting zero-shot failures through representation geometry rather than assuming a lack of biological signal.

Technical Summary: Open 3D-CT Vision-Language Models Do Not Clear a Pre-Registered Biomarker Gate Under Zero-Shot Prompting at n = 96 NSCLC-Radiogenomics

More like this