💻 computer science

When Does Synthetic CT Transfer? A Label-Free Donor/Host Diagnostic for Medical Vision-Language Model Routing on Real Lung CT

This paper demonstrates that the transferability of medical vision-language models from synthetic to real lung CT data can be predicted label-free by distinguishing between donor-driven features (which transfer) and host-driven features (which do not), enabling the creation of a training-free diagnostic router that accurately selects the best model without requiring real-world labels.

Original authors: Fakrul Islam Tushar

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fakrul Islam Tushar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to hire a team of five different doctors to diagnose lung nodules from CT scans. You have a huge problem: you don't have the "answer key" (real patient labels) to test them on the real world yet. However, you do have a perfect, computer-generated simulation of lungs with exact answers.

The paper asks a simple but tricky question: Can we trust the results we get from the simulation to tell us who the best doctor is for the real world?

Usually, the answer is "maybe not." Simulations are often too perfect, and real life is messy. But this paper discovers a clever trick to predict exactly when the simulation will work and when it won't, without ever needing to look at a real patient's answer key first.

Here is how they did it, explained with everyday analogies:

1. The "Donor" vs. The "Host" (The Core Discovery)

The authors realized that a medical AI's skill comes from two different sources, like a transplant patient:

The Donor (The Nodule): This is the specific thing being looked at (the lung spot). If a doctor is good at spotting the shape or size of a spot, that skill belongs to the "donor." It travels with the spot, no matter whose body it is in.
The Host (The Surrounding Anatomy): This is the rest of the lung and chest. If a doctor is good at figuring out which part of the lung a spot is in, that skill belongs to the "host." It depends entirely on the specific body the spot is sitting in.

The Big Finding:

Donor skills transfer: If a model is great at measuring the size of a nodule in a fake lung, it will likely be great at measuring it in a real lung. The "skill" moves with the object.
Host skills do not transfer: If a model is great at naming the lobe of a lung in a fake body, it might fail completely in a real body. The "skill" gets stuck in the fake body because real lungs look different.

2. The "TrialCouncil" (The Smart Manager)

To test this, the authors built a system called TrialCouncil. Think of this as a manager who hires the five doctors based only on their performance in the simulation.

The manager has three rules (a "gate") for how to handle a new real-world case:

SELECT: If the simulation shows one doctor is clearly the best at a specific task (like measuring size), the manager picks that doctor and ignores the others.
BLEND: If the simulation shows the doctors are all decent but no single one dominates, the manager takes a weighted vote (like a committee).
DEFER: If the simulation shows that all the doctors tend to make the same mistake together (correlated failure), the manager says, "I don't know," and refuses to guess. This is safer than guessing wrong.

3. The Results: What Worked and What Didn't

The team tested this on three tasks:

Task A: Is there a nodule? (Presence)
- Prediction: This is a "Donor" task.
- Result: The simulation predicted the best model perfectly. The manager successfully picked the right doctor for the real world.
Task B: How big is the nodule? (Size)
- Prediction: This is a "Donor" task.
- Result: Perfect prediction. The simulation told them exactly who was best, and that person was the best in real life too.
Task C: Which lobe is it in? (Lobe)
- Prediction: This is a "Host" task.
- Result: The simulation failed to predict the winner. The doctor who looked best in the fake lung was not the best in the real lung. This confirmed the theory: skills based on the surrounding anatomy don't transfer well from fake to real.

4. Why This Matters (The "Label-Free" Magic)

Usually, to know if an AI is good, you need real data with answers (labels). This paper says: "No, you don't."

By looking only at the computer-generated data, they could predict:

"Hey, for measuring size, we can trust the simulation to pick our winner."
"Hey, for naming lung lobes, the simulation is lying to us; don't trust it."

They built a system that acts on this prediction. It uses the simulation to choose the best model for the tasks that do transfer, and it knows to be careful (or abstain) on the tasks that don't.

Summary Analogy

Imagine you are trying to pick the best driver for a race.

The Simulation: A video game track.
The Real Race: A muddy, rainy dirt track.

If you test drivers on the video game:

Donor Skill: How fast they can turn a corner. If they are fast in the game, they are fast on the mud. (This transfers).
Host Skill: How well they know the specific landmarks of the video game track. If they know the game track perfectly, they might get lost on the mud track because the landmarks are different. (This does not transfer).

This paper built a "Manager" that looks at the video game results and says: "Okay, for turning corners, I'll pick Driver A. But for knowing the landmarks, I know the game results are useless, so I won't trust them."

The Bottom Line: You can predict which AI skills will survive the jump from fake data to real life, and you can do it without ever needing to see a single real patient's answer key.

Technical Summary: When Does Synthetic CT Transfer?

Problem Statement
Synthetic medical images offer exact labels and controlled interventions, making them valuable for measuring the competence of medical Vision-Language Models (VLMs) in zero-shot settings. However, a critical gap exists: a competence measurement derived from synthetic data is only useful if it survives the transition to real clinical data. Real-world medical imaging often lacks the ground-truth labels required to verify this transfer. The paper addresses the open question of whether competence rankings observed on synthetic data can predict performance on real CT scans without access to real labels, and if so, under what conditions this transfer holds.

Methodology
The authors propose a mechanism to predict transferability by decomposing model competence into two axes on synthetic "digital twins":

Donor-driven competence: Properties inherent to the transplanted nodule (e.g., presence, size) that remain constant regardless of the surrounding anatomy.
Host-driven competence: Properties dependent on the surrounding anatomy (e.g., lobe localization) that change when the host environment changes.

The hypothesis is that donor-driven competence transfers from synthetic to real CT because the "donor" (nodule) is consistent, whereas host-driven competence may not transfer because the move from synthetic to real CT constitutes a change of "host."

To operationalize this, the authors introduce TrialCouncil, a training-free routing system comprising five VLMs (BiomedCLIP, MedGemma, Lingshu, Qwen2.5-VL, LLaVA-Med). The council operates via a Competence Gate calibrated entirely on synthetic CT (using the iTrialSpace dataset) and applied unchanged to real CT. The gate reads the synthetic competence ordering ( $Q$ ) and routes each task-condition cell to one of three behaviors:

Select: If one member dominates the competence ordering, route to that model.
Blend: If competence is distributed, take a weighted plurality vote based on synthetic competence weights.
Defer: If all competent members fail together (correlated failure), abstain from answering.

Crucially, the system uses no real labels for calibration, no model weight updates, and relies solely on the ordering of synthetic performance to make routing decisions.

Key Contributions

A Label-Free Transfer Criterion: A synthetic-only donor/host decomposition that predicts, before seeing any real data, whether a competence ordering will transfer. This is grounded in a mechanistic explanation rather than outcome fitting.
Three-Task Demonstration: Validation across three lung-CT tasks (nodule presence, size, and lobe localization) spanning the donor-to-host axis.
TrialCouncil Instrument: A routing policy that recovers the best fixed model label-free wherever the diagnostic predicts transfer, and correctly identifies the boundary where transfer fails.
Quantification of Correlated Failure: Evidence that medical VLMs fail on the same cases far more often than independence predicts, necessitating deferral strategies over simple voting.

Results
The study was evaluated across five public VLMs, four spatial-guidance conditions (plain, bounding box, contour, both), and seven real lung-CT datasets.

Transfer of Ordering: For donor-driven tasks (presence and size), the synthetic competence ordering strongly predicts real-world performance ( $R^2 \ge 0.96$ ). The synthetic-best member is consistently the real-best member. Consequently, the "transfer gap" (the performance difference between synthetic-calibrated and real-calibrated routing) is effectively zero (0.000) for these tasks.
The Boundary (Lobe): For the host-driven task (lobe localization), the ordering does not transfer ( $R^2 = 0.83$ ). The synthetic-best member is not the real-best member, and a significant transfer gap exists. The diagnostic correctly flags this task as transfer-risky.
TrialCouncil Performance:
- On size, TrialCouncil acts as a pure selector, matching the synthetic-best member (MedGemma) and tripling the macro-F1 over plurality voting.
- On lobe, where ordering is unstable, TrialCouncil uses blending to outperform the synthetic-best member, though it cannot match the oracle real-best (BiomedCLIP) without real labels.
- On unaided presence, the system identifies a "correlated-failure core" where all models fail together. By deferring in these cases, it lowers selective risk significantly compared to confidence-based or entropy-based deferral.
Robustness: The results hold under "leave-source-out" calibration, ruling out data leakage from shared public sources between synthetic and real datasets.

Significance and Claims
The paper claims its primary contribution is not the router itself, but the finding that transfer is predictable, label-free, and derivable from synthetic data alone.

The authors argue that synthetic measurements are useful only if they survive the move to real data. By decomposing competence into donor and host axes, they demonstrate that one can predict in advance which synthetic measurements will transfer. The "diagnostic" localizes the boundary of transfer (identifying lobe localization as the failure point) without ever seeing a real label.

The significance lies in offering an alternative to collecting target labels and fine-tuning: if a faithful simulator exists, one can calibrate a decision policy on synthetic data and deploy it unchanged. The paper modestly notes that the council reallocates existing competence rather than creating new accuracy, and that absolute performance ceilings (e.g., low macro-F1 on size) remain a challenge. The work establishes a framework for understanding when synthetic-to-real transfer is valid, providing a label-free reason to trust or distrust synthetic calibration for specific tasks.