💻 computer science

Toward Training-Free Zero-Shot Anomaly Detection in 3D Medical Images: A Batch-Based Approach Using 2D Foundation Models

This paper introduces CS3F, a training-free, batch-based framework that leverages 2D foundation models and a coarse-to-fine tokenization strategy to enable zero-shot anomaly detection in 3D medical images by identifying subjects with unique volumetric features lacking analogues in a reference cohort.

Original authors: Tai Le-Gia

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tai Le-Gia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a doctor trying to find a tiny, hidden tumor in a 3D scan of a patient's brain or lungs. Usually, to teach a computer to do this, you need to show it thousands of examples of "sick" brains with the tumors already marked by experts. But what if you don't have those marked examples? What if the patient has a rare disease you've never seen before, or the hospital uses a different type of scanner than the one the computer was trained on?

This is the problem the paper solves. The authors created a new method called CS3F (Cross-Subject Anomaly Scoring for 3D volumes via Foundation models). It's like a "detective" that can spot weirdness in a 3D medical scan without ever needing to be taught what a disease looks like.

Here is how it works, broken down into simple concepts:

1. The "Group Photo" Detective (Batch-Based Approach)

Imagine you have a group of 100 people standing in a room. You ask them to describe themselves.

The Normal People: Most people look and act very similar to each other. If you pick one person, you can easily find 10 others who look just like them.
The Odd One Out: One person is wearing a bright clown nose and a tutu. If you try to find someone else in the room who looks like that person, you can't. They are unique.

CS3F works the same way. Instead of looking at one patient in isolation, it looks at a whole "batch" (group) of patients at the same time. It assumes that healthy body parts (like normal brain tissue) will look very similar across different people. If a specific spot in a patient's brain looks nothing like the corresponding spot in any of the other patients in the group, the system flags it as "anomalous" (suspicious). It doesn't need to know what the disease is; it just knows that this spot doesn't fit the pattern of "normal."

2. The "2D Camera" Problem

The paper uses powerful AI models called "Foundation Models" (like DINOv2). These are like super-smart cameras that have been trained on millions of regular 2D photos (like pictures of cats, cars, and landscapes). They are amazing at spotting details in flat, 2D images.

However, medical scans are 3D (they are thick blocks of data, like a loaf of bread, not just a slice of bread).

The Challenge: You can't just feed a whole 3D loaf of bread into a camera designed for 2D photos.
The Solution: CS3F slices the 3D loaf into thin 2D slices (like cutting the bread). It runs the 2D camera on every slice to understand the details. Then, it glues the information from neighboring slices back together to understand the 3D shape.

3. The "Blurry vs. Sharp" Trade-off (Coarse-to-Fine)

When the system glues the slices back together, it faces a dilemma:

The Big Picture (Coarse): If you group a large chunk of slices together, the image is stable and clear. You can easily tell if a whole region looks normal. But, if there is a tiny tumor (like a small speck of dust), it might get "averaged out" and disappear in the big chunk.
The Fine Detail (Fine): If you look at tiny, individual slices, you can see the small speck of dust. But, looking at tiny details is computationally expensive (it takes a lot of time and computer power) and can be noisy (you might mistake a normal shadow for a speck).

The "Coarse-to-Fine" Trick:
CS3F uses a smart routing system.

First Pass (Coarse): It quickly scans the whole brain with "big eyes" to find general areas that look suspicious.
Second Pass (Fine): It only zooms in with "microscope eyes" on those specific suspicious areas to find the tiny details.
This saves a massive amount of time and computer power while still catching the small tumors.

4. The "Three-Angle" View

To make sure it doesn't miss anything, CS3F looks at the 3D volume from three different angles:

Axial: Looking from the top down.
Coronal: Looking from the front.
Sagittal: Looking from the side.

It combines the clues from all three angles. If a spot looks weird from the top, the front, and the side, it's definitely a problem. This makes the detection much more reliable than just looking at one slice.

What Did They Test?

The authors tested this "detective" on:

Brain Scans (MRI): Looking for metastases (cancer that spread to the brain), gliomas (brain tumors), and strokes.
Lung Scans (CT): Looking for lung cancer.

The Results:

It worked without needing any pre-labeled "sick" data.
It found anomalies better than other "zero-shot" methods that try to use text prompts (like asking the AI "Is this a tumor?").
It performed almost as well as methods that were trained on healthy data, but without the hassle of collecting that data.
It worked on both brain and lung scans, proving it can handle different organs and different types of scanners.

The Bottom Line

CS3F is a clever, "training-free" way to find medical anomalies. It treats a group of patients like a crowd of people, looking for the one who doesn't fit the crowd. By using a smart "zoom-in" strategy and looking from multiple angles, it can spot hidden problems in 3D medical scans using powerful 2D AI tools, all without needing a teacher to show it what diseases look like first.

Important Note from the Paper:
The authors are careful to say that while this works well for finding where the problem is, it still relies on the AI being able to "see" the difference between healthy and sick tissue. If a tumor is very faint or looks exactly like healthy tissue in the AI's eyes, the system might miss it. It is a tool for screening and curation, not a replacement for a doctor's final diagnosis.

Technical Summary: CS3F – Training-Free Zero-Shot Anomaly Detection in 3D Medical Images

Problem Statement

Zero-shot anomaly detection (ZSAD) is critical for medical imaging, where clinical systems must handle heterogeneous acquisition protocols, changing patient populations, and pathologies lacking annotated training data. While existing ZSAD methods for 2D images leverage foundation models (e.g., CLIP, DINOv2), their direct extension to 3D medical volumes faces three primary obstacles:

Lack of 3D Foundation Models: Large-scale, native 3D medical foundation models are less mature than their 2D counterparts.
Computational Intractability: Volumetric images generate orders of magnitude more tokens than 2D images, making the quadratic complexity of cross-subject mutual similarity searches computationally prohibitive.
Contextual Loss: Slice-wise processing discards through-plane contextual information, while fixed-scale depth aggregation can attenuate focal lesion signals if the lesion occupies only a small fraction of the aggregation window.

Current unsupervised methods often require curated normal training cohorts, which are costly to assemble and may not match the operating distribution due to domain shifts.

Methodology: CS3F Framework

The authors propose CS3F (Cross-Subject Anomaly Scoring for 3D volumes via Foundation models), a training-free, batch-based framework that adapts 2D foundation models to 3D anomaly localization.

1. Core Assumption: The Doppelgänger Assumption

The framework relies on the statistical observation that within a batch of volumes from the same anatomical domain, normal anatomical structures recur with consistent appearance and spatial organization, whereas pathological structures are heterogeneous and rare. Consequently, normal tokens find close cross-subject analogues, while anomalous tokens do not.

2. Volumetric Token Construction

Instead of using a 3D encoder, CS3F utilizes frozen 2D vision transformers (e.g., DINOv2):

Multi-Axis Slicing: Each 3D volume is decomposed along the sagittal, coronal, and axial axes.
Slice Encoding: Slices are resized and encoded by the 2D foundation model to extract patch features.
Depth Pooling: Neighboring slice features are aggregated via average pooling and $L_2$ normalization to form localized volumetric tokens. This captures depth-wise anatomical context without 3D pre-training.
Random Projection: High-dimensional token features are projected into a lower-dimensional space (e.g., $d=128$ ) using a random projection matrix to reduce memory and computational costs while preserving local neighborhood geometry (Johnson-Lindenstrauss lemma).

3. Cross-Subject Scoring

Anomaly scores are derived from mutual similarity within the test batch:

For a query token, the distance to the nearest tokens in all other volumes in the batch is computed.
The anomaly score is the average of the $K$ smallest distances. Tokens lacking close analogues (high distances) are assigned higher anomaly scores.
Axis-Wise Fusion: Scoring is performed independently for each anatomical axis, and the resulting anomaly maps are averaged to fuse complementary evidence from orthogonal planes.

4. Coarse-to-Fine (C2F) Tokenization Strategy

To address the attenuation of focal lesions caused by large pooling kernels and the computational cost of exhaustive fine-scale matching, CS3F introduces a C2F routing mechanism:

Coarse Pass: A coarse pooling configuration ( $q_c$ ) generates large tokens for robust global anatomical matching.
Routing: For each query coarse token, the top- $L$ nearest coarse tokens in reference volumes are identified.
Fine Pass: Fine-scale matching ( $q_f$ ) is restricted only to the fine tokens contained within these selected coarse regions.
Multi-Scale Fusion: The final anomaly map combines the stable coarse map with the high-resolution fine map, balancing global consistency with focal lesion sensitivity.

Key Contributions

CS3F Framework: A training-free, batch-based ZSAD framework for 3D medical images that utilizes frozen 2D foundation models to generate volumetric representations without lesion annotations or curated normal training data.
Axis-Wise Score Fusion: A redesigned pipeline that fuses anomaly scores at the map level rather than the feature level. This reduces memory usage, preserves axis-specific spatial evidence, and supports non-cubic volumes (e.g., lung CT).
Coarse-to-Fine Multi-Scale Tokenization: A strategy motivated by the analysis of pooling-induced lesion attenuation. It enables fine-resolution scoring for focal lesions while avoiding the prohibitive cost of exhaustive mutual similarity searches.
Cross-Organ Validation: Demonstration of the framework's generalizability beyond atlas-aligned brain MRI to non-standardized, non-cubic lung CT data.

Experimental Results

The framework was evaluated on brain MRI (metastases, glioma, stroke) and lung CT (lung cancer).

Anomaly Detection (Patient-Level): CS3F variants consistently outperformed CLIP-based zero-shot baselines (WinCLIP, AnomalyCLIP, APRIL-GAN) across all benchmarks. CS3F-MS (Multi-Scale) achieved patient-level AUROC >84% on all datasets, whereas CLIP-based methods showed significant performance drops on specific pathologies (e.g., dropping to ~63% on stroke). CS3F also remained competitive with reconstruction-based unsupervised methods (DAE, pDDPM), which achieved high detection scores but often suffered from poor segmentation due to domain gaps.
Anomaly Segmentation (Voxel-Level): CS3F outperformed CLIP-based methods in Dice and IoU scores. The multi-scale approach (CS3F-MS) provided a balance between the stability of coarse tokens and the sensitivity of fine tokens.
Lesion-Wise Analysis: Under a fixed 99% True Negative Rate (TNR), fine tokenization (CS3F-F) significantly improved the detection of small and medium lesions (e.g., increasing small-lesion LTPR from 11.5% to 34.6% on ATLAS T1w) but at the cost of increased false positives. CS3F-MS offered a controlled trade-off, improving sensitivity over coarse-only methods while maintaining manageable false-positive rates.
Cross-Organ Generalization: On lung CT, where volumes were not atlas-registered and were non-cubic, CS3F maintained effectiveness, outperforming CLIP-based baselines in both detection and localization.
Efficiency: The C2F routing reduced computational overhead significantly compared to exhaustive fine-scale matching. CS3F-C (coarse-only) processed volumes in ~3.2s on a consumer GPU, while CS3F-MS took ~5.5s.

Significance and Claims

The paper claims that frozen 2D foundation models can effectively support anomaly localization in 3D medical images when paired with a suitable volumetric tokenization and scoring pipeline.

Practicality: CS3F offers a practical pathway for cohort-level screening, large-scale dataset curation, and automated quality control by eliminating the need for separately curated healthy-control datasets and manual lesion annotations.
Limitations: The authors acknowledge that the method is constrained by the "Doppelgänger Assumption," requiring a batch of subjects for inference rather than single-subject processing. Furthermore, performance on small, subtle, or low-contrast lesions is limited by the foundation model's ability to encode them as separable outliers in the feature space. The paper notes that while C2F tokenization improves sensitivity, it cannot fully overcome the lack of discriminative signals in the underlying encoder for certain modalities (e.g., T1w MRI for low-contrast metastases).
Future Direction: The authors suggest that lightweight adaptation strategies on auxiliary medical data and multi-modal fusion could further improve lesion-normal separability while preserving the zero-shot character of the target task.