💻 computer science

When Can Test-Time Adaptation Help Zero-Shot CT Vision-Language Models?

This paper investigates the conditional effectiveness of test-time adaptation for zero-shot 3D CT vision-language models and introduces CARVE, a novel method that estimates positive-label cardinality and employs memory-efficient multi-view optimization to improve multi-label prediction reliability under distribution shifts.

Original authors: Ailar Mahdizadeh, Puria Azadi Moghadam, Xiangteng He, Leonid Sigal

Published 2026-07-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ailar Mahdizadeh, Puria Azadi Moghadam, Xiangteng He, Leonid Sigal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot doctor that can look at a 3D picture of a human body (like a CT scan) and read a list of medical conditions to see if they are present. This robot hasn't been taught by a human teacher with answer keys; instead, it learned by reading millions of medical reports and matching them to pictures on its own. This is called a "zero-shot" model. It's amazing because it can be sent to a new hospital immediately, even if that hospital uses different machines or has different patients.

However, there's a catch. When this robot moves to a new hospital, the pictures might look slightly different—maybe the scanner is from a different brand, or the patients are from a different region. This is called "distribution shift." To fix this, scientists use a trick called "Test-Time Adaptation" (TTA). Think of TTA as giving the robot a quick "brain warm-up" right before it looks at a new patient. It looks at the new, unlabeled scan and tweaks its internal settings to match the new environment, hoping to get better at spotting diseases without needing a human to grade its work first.

But here is the big question: Does this warm-up always help? Sometimes, trying to fix a robot that is already confused just makes it more confused. This paper asks exactly that: When does this quick warm-up actually help a 3D medical robot, and when does it fail? The researchers didn't just assume it works; they tested it to find the specific conditions where it succeeds and where it hits a wall.

The Detective Work: When Does the Warm-Up Work?

The researchers, led by Ailar Mahdizadeh and her team, treated this like a detective story. They wanted to know if Test-Time Adaptation (TTA) is a magic bullet or a conditional tool. They studied "Vision-Language Models" (VLMs)—AI systems that connect 3D CT scans (the pictures) with text prompts (the disease names).

First, they discovered a few hard rules. The robot's "eyes" (the part of the AI that sees the 3D scan) need to see the picture in its full, original depth. If you squish the 3D picture into a flatter version, the robot gets confused immediately, and no amount of warm-up can fix it. They found that if you reduce the depth of the scan too much, the robot's accuracy drops by more than 0.12 before it even starts trying to adapt. So, the first rule is: Keep the 3D picture 3D.

Second, the robot needs to start with a decent brain. If the robot is completely lost when it first arrives at the new hospital (meaning its base predictions are random, like flipping a coin), no amount of warm-up can teach it to see clearly. The researchers found that if the base model is already good at separating "sick" from "healthy," then adaptation helps. But if the base model is failing, adaptation can't create magic out of thin air.

The Problem with the Old Rules

The team realized that the old ways of doing this "warm-up" were broken for medical scans. Most existing methods were designed for natural images (like photos of cats or cars) where the AI has to pick one answer (e.g., "Is this a cat or a dog?"). They use a method called "entropy minimization," which basically tells the AI: "Be very confident about your answer."

But medical scans are different. A single patient can have multiple problems at the same time—like having both pneumonia and a broken rib. This is called "multi-label" prediction. The old methods tried to force the AI to be confident about just one thing, which caused it to ignore the other diseases present. It was like a teacher telling a student who got three questions right to pick only one answer to be confident about, forcing them to forget the other two correct answers.

Enter CARVE: The Smart Adapter

To fix this, the team invented a new method called CARVE (Cardinality-Aware Retained-View Entropy). Imagine CARVE as a very careful editor.

Counting the Problems: Instead of guessing, CARVE looks at the robot's initial guesses and estimates how many diseases are likely present in that specific scan. It asks, "Does this patient have one problem, two, or three?"
The "Weak View" Trick: Scanning a full 3D body is heavy and slow. To save time and memory, CARVE creates several slightly different, "weak" versions of the scan (like looking at a photo through slightly foggy glasses). It checks which versions the robot is most confident about and keeps only the best ones.
The Top-K Goal: Instead of forcing the robot to be confident about just one disease, CARVE tells it: "Be confident about your top k guesses," where k is the number of problems you estimated in step one. This ensures that if a patient has three diseases, the robot learns to spot all three, rather than collapsing into just one.

What They Found

The results were clear and consistent. CARVE worked best when the robot was already somewhat smart (discriminative) and the 3D pictures were kept in their full depth.

On internal tests (where the data was similar to what the robot learned on), CARVE boosted the robot's ability to spot diseases significantly. For example, on one test, it improved the accuracy score from 0.713 to 0.749.
On external tests (where the data came from a totally different hospital with different scanners), the robot's starting brain mattered most. If the robot started out confused, even CARVE couldn't fix it completely. However, when the robot started with a decent brain, CARVE still managed to squeeze out extra performance, improving scores from 0.499 to 0.533 in very difficult scenarios.

The paper explicitly rules out the idea that adaptation is always helpful. They showed that if the input data is squished (reduced depth) or if the base model is terrible, adaptation fails. They also proved that standard methods (like TENT or RLCF) often make things worse in multi-label medical settings because they crush the "co-occurring" diseases into a single choice.

In short, the authors found that Test-Time Adaptation is a powerful tool, but it's not a magic wand. It works like a fine-tuning screwdriver: it's great for tightening up a machine that is already built well, but it can't fix a machine that is broken or built with the wrong parts. For 3D CT scans, the key is keeping the 3D structure intact and using a method like CARVE that respects the fact that patients often have more than one problem at a time.

Technical Summary: When Can Test-Time Adaptation Help Zero-Shot CT Vision-Language Models?

Problem Definition
Zero-shot 3D CT Vision-Language Models (VLMs), such as CT-CLIP and fVLM, offer a promising avenue for cross-institutional deployment by classifying abnormalities from text prompts without task-specific labels. However, these models face significant challenges when deployed in new clinical settings due to distribution shifts caused by differences in scanner vendors, reconstruction protocols, and demographics. While Test-Time Adaptation (TTA) allows models to update on unlabeled target data without source data or annotations, existing TTA methods are ill-suited for this specific domain. Current approaches target multi-class softmax prediction on natural images or single-label medical segmentation. They fail to address unsupervised multi-label adaptation for zero-shot 3D CT VLMs, where a single scan contains multiple co-occurring abnormalities scored as independent Bernoulli probabilities rather than a shared softmax distribution. Furthermore, the conditions under which TTA is beneficial for these models remain poorly understood.

Methodology: CARVE
The authors introduce CARVE (Cardinality-Aware Retained-View Entropy), the first TTA method designed specifically for the prompt-pair multi-label regime of zero-shot 3D CT VLMs. The method operates through three primary stages:

Memory-Efficient View Selection: To address the computational constraints of volumetric inference, CARVE generates $V$ weakly augmented 3D views of the input scan. It scores these views based on mean binary entropy across labels without computing gradients. It then retains only the top $K$ lowest-entropy views (the "reliable" subset) for the adaptation step, avoiding the memory cost of backpropagating through all augmented views.
Sample-Specific Cardinality Estimation: Unlike natural image multi-label TTA which relies on retrieving paired captions to estimate label counts, CARVE estimates the number of positive abnormalities ( $\hat{k}$ ) directly from the prompt-pair probabilities. It sums the Bernoulli probabilities for all labels and clips the result to a valid range $[k_{min}, k_{max}]$ .
Cardinality-Aware Objective: CARVE replaces standard entropy minimization with a top- $\hat{k}$ objective. This loss function sharpens the probabilities of the estimated $\hat{k}$ most likely positive labels while suppressing the remaining labels. This approach prevents the collapse of co-occurring abnormalities (a failure mode of cross-label entropy) and avoids the weakness of per-label binary entropy near the decision boundary ( $p \approx 0.5$ ).

The adaptation is episodic and source-free: for each test scan, the model resets its visual normalization parameters to the source values, performs a few gradient steps using the CARVE loss on the retained views, and then predicts on the original unperturbed volume.

Key Contributions

Diagnostic Framework: The authors establish that TTA for zero-shot 3D CT VLMs is conditional, not universally beneficial. Adaptation requires two preconditions: (1) the volumetric input pipeline must preserve the depth structure the encoder was pretrained on (reducing depth degrades performance significantly before adaptation begins), and (2) the base representation must already be discriminative enough to separate present from absent abnormalities in the target cohort.
Identification of Objective Mismatch: The paper identifies a structural mismatch between standard entropy minimization and prompt-pair multi-label CT prediction. Standard entropy either collapses co-occurring positives or becomes uninformative near the decision boundary, failing to preserve the scan-level cardinality of abnormalities.
CARVE Algorithm: The introduction of a cardinality-aware, memory-efficient TTA method that estimates label counts directly from probabilities and optimizes a top- $\hat{k}$ objective, enabling the preservation of multiple co-occurring abnormalities.

Results
Experiments were conducted on internal (CT-RATE) and external (RAD-ChestCT) datasets using CT-CLIP and fVLM backbones, covering multi-label, three-class, and binary settings.

Preconditions: Reducing input depth (e.g., from $z=240$ to $z=40$ ) caused a significant drop in AUROC (>0.12) for the unadapted base model, confirming that input integrity is a prerequisite for TTA. Under severe external shift (RAD-ChestCT), the choice of backbone (e.g., fVLM vs. CT-CLIP) had a larger impact on performance than the choice of adaptation objective, indicating that TTA cannot recover from a non-transferable base representation.
Performance: When the base model was already discriminative, CARVE provided the most consistent improvements. For example, on internal CT-RATE (Zero-shot), CARVE improved AUROC from 0.713 (No-TTA) to 0.749. On external RAD-ChestCT, it improved AUROC from 0.499 to 0.533.
Comparison: Standard entropy-based methods (TENT, RLCF) and existing multi-label TTA (ML-TTA) often failed to improve performance or degraded it by shifting the operating point without preserving co-occurring labels. CARVE's gains were attributed to its ability to estimate and preserve the correct number of positive labels.

Significance and Claims
The paper claims to establish multi-label TTA for zero-shot 3D CT VLMs as a distinct problem requiring a cardinality-aware approach. The authors modestly assert that their results demonstrate when adaptation helps: specifically, when the input pipeline preserves volumetric structure and the base model already possesses a transferable, discriminative representation. They emphasize that CARVE is effective at refining these existing signals by preserving co-occurring abnormalities, but it cannot create discrimination from a failed representation. The work provides a diagnostic protocol for deployment and a specialized objective (CARVE) that aligns with the Bernoulli nature of prompt-pair medical diagnosis.

The Detective Work: When Does the Warm-Up Work?

The Problem with the Old Rules

Enter CARVE: The Smart Adapter

What They Found

Technical Summary: When Can Test-Time Adaptation Help Zero-Shot CT Vision-Language Models?

More like this