⚡ electrical engineering

An Open Multi-Center Whole-Body FDG PET/CT Foundation Model for Tumor Segmentation

This paper introduces an open-source, multi-center foundation model for whole-body FDG PET/CT tumor segmentation that utilizes early cross-modal fusion and a specialized masked autoencoding objective to achieve high performance with minimal labeled data, thereby reducing the reliance on extensive manual annotations in clinical oncology.

Original authors: Xiaofeng Liu, Qianru Zhang, Thibault Marin, Menghua Xia, Chi Liu, Georges El Fakhri, Jinsong Ouyang

Published 2026-05-22

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaofeng Liu, Qianru Zhang, Thibault Marin, Menghua Xia, Chi Liu, Georges El Fakhri, Jinsong Ouyang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to spot tumors in a patient's body using medical scans. Doctors usually look at two types of images together: a CT scan (which shows the body's anatomy, like a detailed 3D map of bones and organs) and a PET scan (which shows metabolism, lighting up areas where cells are burning energy, often indicating cancer).

For a long time, computers have been bad at combining these two views effectively. Most existing AI models were like students who only studied for one specific test. They were trained on small groups of patients from just one hospital, and they often treated the CT and PET scans as two separate subjects, only trying to combine them at the very end of their "thinking" process. This made them fragile; if the computer saw a scan from a different hospital or a different machine, it often got confused.

This paper introduces a new solution: a "Foundation Model" for PET/CT scans. Think of this not as a student studying for one test, but as a medical student who has read thousands of textbooks from four different countries before ever seeing a real patient.

Here is how they built it and why it works, explained simply:

1. The "Library" of Scans

Instead of using a small dataset from one place, the researchers gathered 4,997 scans from four different public datasets (from Germany, Australia, the USA, and Vietnam).

The Challenge: These scans were all different sizes and taken with different machines. It's like trying to build a puzzle where some pieces are from a 1000-piece set and others are from a 500-piece set, and the picture quality varies.
The Fix: They created a "harmonization pipeline." Imagine a master editor who takes all these different puzzle pieces, trims the edges, and resizes them so they all fit into the same physical grid (2.0 x 2.0 x 3.0 mm per piece). Crucially, they didn't just stretch the images to look the same; they preserved the actual physical size of the tumors so the computer learns the true scale of things.

2. The "Early Marriage" of CT and PET

Most previous AI models treated the CT and PET scans like two people sitting at opposite ends of a table, only talking to each other after they had each written their own report.

The New Approach: This model forces the CT and PET scans to "hold hands" from the very first moment. They are stitched together side-by-side (channel-wise concatenation) before the computer even starts analyzing them.
The Analogy: It's like teaching a child to recognize a dog by showing them a picture of a dog and the sound of a bark simultaneously, rather than showing them a picture, asking "What is this?", then playing a sound and asking "What is this?" separately. By mixing the anatomical (CT) and metabolic (PET) information immediately, the model learns how the body's structure and its energy use relate to each other right from the start.

3. The "Blindfold" Game (Masked Autoencoding)

To teach this model without needing a human to draw outlines on every single tumor (which is expensive and slow), they used a technique called Masked Autoencoding.

The Game: Imagine you show the computer a picture of a tumor, but you cover up 50% of it with a blindfold. The computer's job is to guess what is hidden underneath based on the visible parts and the other scan type (e.g., if the CT is hidden, it looks at the PET to guess the shape).
The Innovation: Usually, when computers cover up parts of an image, they fill the blank space with a generic "placeholder" token (like a blank gray square). The researchers realized this creates "artificial edges" that confuse the computer. Instead, they used Zero-Mean Imputation. Since they had already normalized the data (adjusted the brightness so the average is zero), they simply filled the hidden spots with zero.
Why it matters: It's like filling a hole in a wall with the exact same color of plaster that was already there, rather than putting a gray patch on it. This keeps the "texture" of the image smooth, allowing the computer to learn better without getting distracted by fake boundaries.

4. The Results: Learning with Less

The researchers tested this new model on a task called "tumor segmentation" (drawing a line around the tumor).

The "Few-Shot" Miracle: Usually, AI needs thousands of labeled examples to learn well. This model, however, was trained on the massive "library" of 4,997 scans first. When they tested it with only 10% of the usual labeled data (just 70 scans), it performed just as well as other models trained on 100% of the data.
The "Extreme" Test: They even tried a "5-shot" test, where the computer only saw 5 labeled examples. The new model still worked, while older models failed completely.
The Takeaway: Because the model learned general rules about how tumors look in both CT and PET from a huge, diverse dataset, it didn't need to be re-taught everything from scratch for every new hospital.

Summary

The paper presents an open-source "brain" for medical imaging that has read thousands of diverse scans. By teaching it to look at CT and PET scans together from the very beginning and by using a clever "fill-in-the-blanks" training method, they created a system that is:

Smarter: It understands the relationship between body structure and metabolism better.
More Efficient: It can learn to spot tumors with very few labeled examples, saving doctors time.
More Robust: It works well even when the scans come from different hospitals or machines.

The authors note that while this is a powerful tool, it currently requires both CT and PET scans to be present (it can't handle missing data yet) and that the largest version of the model sometimes needs more labeled data to fine-tune perfectly than what was available in their test. However, it provides a strong, open foundation for future automated cancer imaging.

Technical Summary: An Open Multi-Center Whole-Body FDG PET/CT Foundation Model for Tumor Segmentation

Problem Statement
While the synergistic interpretation of Computed Tomography (CT) and Positron Emission Tomography (PET) is standard in oncologic imaging for lesion detection and staging, existing deep learning approaches face significant limitations. Most current methods are task-specific, trained on small, single-center cohorts, and rely on dual-branch fusion schemes that delay cross-modal interaction. These approaches often fail to generalize across different institutions, scanner vendors, and acquisition protocols due to a lack of diverse training data and the underutilization of early spatial correspondence between PET and CT. Furthermore, standard Masked Autoencoder (MAE) implementations, which use learnable mask tokens, can introduce non-physical intensity discontinuities that destabilize hierarchical encoder-decoder architectures common in medical imaging.

Methodology
The authors propose an open-source foundation model pre-trained on a harmonized corpus of 4,997 whole-body FDG PET/CT scans aggregated from four public datasets (AutoPET IV, DeepPSMA, SPADE, and ViMedPET).

Data Curation and Harmonization: Instead of resizing images to fixed matrix sizes (which distorts anatomy), the authors resampled all scans to a unified physical voxel spacing of $2.0 \times 2.0 \times 3.0$ mm $^3$ . They applied volume-wise z-score normalization to standardize CT Hounsfield Units and PET Standardized Uptake Values (SUVs). Misregistrations in specific datasets were corrected via rigid registration.
Architecture: The framework employs hierarchical UNet-shaped backbones, specifically evaluating SwinUNETR-v2 (in small, base, and large configurations) and nnUNet-v2. A key architectural innovation is early channel-wise concatenation of PET and CT inputs at the first embedding layer. This allows anatomical and metabolic features to interact immediately, rather than being processed by separate encoders and fused later.
Pre-training Objective (Zero-Mean Imputation MAE): To address the incompatibility of learnable mask tokens with convolutional skip connections, the authors replaced them with zero-mean imputation. Since the input data is z-score normalized, masked patches are filled with zeros (the expected mean of the background).
Loss Function: The model utilizes a weighted global reconstruction loss. Unlike standard MAE which ignores visible regions, this loss includes a weighted term for visible voxels ( $\lambda = 0.2$ ) to stabilize the training of skip connections and preserve high-frequency structural information, while primarily focusing on reconstructing the masked regions.

Key Contributions

Large-Scale Multi-Center Corpus: The creation and harmonization of a 4,997-scan 3D whole-body FDG PET/CT pre-training dataset, significantly expanding the scale and diversity available for PET/CT representation learning.
Novel Pre-training Framework: The introduction of a masked autoencoding strategy tailored for hierarchical CNN/Transformer backbones, featuring independent PET/CT masking, zero-mean imputation, and a weighted global reconstruction loss.
Demonstrated Label Efficiency: Empirical evidence showing that the resulting foundation models achieve strong performance with minimal labeled data, outperforming separated-modality pre-training approaches.

Results
The models were evaluated on downstream AutoPET lesion segmentation tasks under various data regimes:

Fine-Tuning with Limited Data: When fine-tuned with only 10% of the labeled training data (70 scans), the MAE-initialized models achieved performance comparable to models trained from scratch on the full dataset (700 scans). For instance, the SwinUNETR-v2-base model achieved a Dice score of 0.6132 with 10% data (MAE-initialized) compared to 0.4889 (scratch), and this was competitive with the full-data scratch baseline (0.6414).
Few-Shot Linear Probing: Under extreme 5-shot conditions (freezing the encoder and most of the decoder, training only the final layer), joint PET/CT pre-training significantly outperformed separated-modality pre-training. The SwinUNETR-v2-base achieved a Dice score of 0.3557 with joint pre-training versus 0.2163 with separated pre-training, whereas scratch initialization collapsed to ~0.09.
Model Scaling: While the SwinUNETR-v2-large variant achieved the lowest pre-training reconstruction loss and best linear probing scores, the SwinUNETR-v2-base variant demonstrated the best balance for full-parameter fine-tuning, avoiding overfitting on the limited downstream dataset.

Significance and Claims
The paper claims to present the first open-source PET/CT foundation model pre-trained on a large multi-center corpus. The primary significance lies in demonstrating label efficiency and cross-modality representation learning. The authors argue that large-scale multi-center pre-training provides transferable anatomical and metabolic priors that substantially reduce the need for large-scale manual annotations in clinical practice.

The work suggests that early fusion of modalities yields a more useful latent space than delayed fusion. Furthermore, the study highlights that while large-capacity models learn robust representations, intermediate-capacity models (like SwinUNETR-v2-base) may offer a more practical balance between representational power and the constraints of available annotated data for downstream clinical tasks. The authors conclude that this foundation model provides a robust, open-source basis for advancing automated oncologic imaging, though they acknowledge limitations regarding pixel-level metrics not fully capturing diagnostic uncertainty and the current requirement for both PET and CT modalities to be present.