💻 computer science

Scaling Self-Supervised and Cross-Modal Pretraining for Volumetric CT Transformers

SPECTRE is a scalable, fully transformer-based foundation model for volumetric CT that leverages self-supervised and vision-language pretraining on open datasets to overcome 3D imaging challenges and achieve state-of-the-art performance in both zero-shot and fine-tuned medical imaging tasks.

Original authors: Cris Claessens, Christiaan Viviers, Giacomo D'Amicantonio, Egor Bondarev, Fons van der Sommen

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Cris Claessens, Christiaan Viviers, Giacomo D'Amicantonio, Egor Bondarev, Fons van der Sommen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to "see" and "understand" 3D medical scans (like CT scans of the chest or abdomen). This is a massive challenge because a CT scan isn't just a flat picture; it's a thick, blocky stack of thousands of slices, creating a huge 3D volume of data.

The paper introduces SPECTRE, a new AI model designed specifically to master this 3D world. Here is how it works, explained through simple analogies.

1. The Problem: The "Too Big to Fit" Puzzle

Standard AI models (like those that recognize cats in photos) are great at 2D images. But if you try to feed them a 3D CT scan, it's like trying to stuff a whole library of books into a single mailbox. The data is too huge, and the computer runs out of memory.

Also, CT scans are "weird" compared to photos:

They are lumpy: The distance between slices is often different from the distance between pixels on a slice (like a brick that is tall but thin).
The labels are messy: Doctors write reports in free-flowing text, not neat checkboxes. Sometimes a report mentions five different diseases at once, making it hard for the AI to know what is "positive" and what is "negative."

2. The Solution: The "Two-Story House" Architecture

SPECTRE solves the size problem by building a two-story house instead of a skyscraper.

The Ground Floor (Local Vision): Imagine looking at a CT scan through a small window. The model looks at small chunks of the scan (like a 3D block of tissue) to understand the fine details. It learns what a cyst, a tumor, or a healthy lung looks like up close. This is the "Local Transformer."
The Second Floor (Global Vision): Once the ground floor has summarized the details of each chunk, it sends a "report card" to the second floor. The second floor looks at all these report cards together to understand the whole patient. It connects the dots: "The lung has a spot, and the liver has a cyst, so this patient might have a specific condition." This is the "Global Transformer."

By splitting the work, SPECTRE can process huge 3D scans without the computer crashing.

3. The Training: Learning Without a Teacher, Then Learning from a Doctor

The model learns in two distinct phases, like a student in school.

Phase 1: The "Self-Study" (Self-Supervised Learning)
Imagine a student who has a library of 200,000 CT scans but no textbooks and no teacher.

The model plays a game of "Hide and Seek." It covers up parts of the scan and tries to guess what's underneath based on the surrounding pixels.
It also looks at the same scan from different angles and distances, learning that a kidney is still a kidney whether it's zoomed in or out.
Result: The model becomes an expert at the shape, texture, and geometry of human anatomy without ever reading a single word of medical text.

Phase 2: The "Language Class" (Vision-Language Alignment)
Now, the model has a teacher (the radiology reports).

The model is shown a CT scan and the doctor's written report side-by-side.
It learns to match the visual "blob" in the scan with the word "tumor" or "inflammation" in the text.
The Twist: Because doctors' reports are messy and often list multiple things, the model uses a special technique (SigLIP) that is more forgiving. It doesn't demand a perfect 1-to-1 match; it learns that a scan might relate to several different descriptions.
Result: The model now understands not just what the anatomy looks like, but what it means clinically.

4. The Result: The "Universal Translator"

The paper tested SPECTRE against 10 other top AI models. Here is what happened:

The "Zero-Shot" Test: They gave SPECTRE a new task it had never seen before (like predicting if a patient would survive two years) and asked it to guess using only its pre-trained knowledge. SPECTRE won more often than any other model. It was like a student who hadn't studied for a specific exam but still got an A because they understood the fundamental principles so well.
The "Retrieval" Test: They asked the model: "Find me all the scans that match this specific sentence in a report." SPECTRE was much better at finding the right images than previous models, proving it truly understands the connection between language and 3D images.
The "Segmentation" Test: They asked it to draw outlines around organs (like the liver or kidneys). Even though SPECTRE wasn't specifically trained to draw, its understanding of the 3D structure was so good that it drew better outlines than models designed only for drawing.

Why This Matters

Before SPECTRE, building a smart 3D medical AI usually required:

Private, expensive data that only big hospitals had.
Massive amounts of manual labeling (doctors drawing on every scan).

SPECTRE proves you can build a world-class, general-purpose 3D medical brain using only publicly available data and open-source code. It's like taking a generic, open-source language model and teaching it to speak "Medical 3D" fluently, so any hospital can use it to detect diseases faster and more accurately.

In short: SPECTRE is a smart, open-source AI that learns to see inside the human body in 3D by first mastering the shapes of anatomy and then learning to read the doctors' notes, all without needing expensive private data.

1. Problem Statement

The paper addresses the challenges in developing scalable, general-purpose foundation models for volumetric Computed Tomography (CT) imaging. While Vision Transformers (ViTs) have succeeded in 2D natural images and 2D medical imaging, their application to 3D CT scans faces unique hurdles:

Computational Complexity: Naive 3D patching leads to token counts scaling cubically with resolution, while self-attention scales quadratically, making global attention computationally intractable for large volumes.
Geometric Anisotropy: CT scans often have anisotropic voxel spacing (e.g., slice thickness differs from in-plane resolution) and variable fields of view (FOV), which standard isotropic positional encodings fail to represent accurately.
Weak and Noisy Supervision: Clinical labels are often sparse, hierarchical, or derived from free-text reports, creating a tension between dense spatial objectives (like reconstruction) and global semantic alignment.
Data Scarcity & Privacy: High-quality 3D medical datasets are often private or small, hindering the training of large-scale models without relying on synthetic data or private hospital data.

2. Methodology: The SPECTRE Framework

The authors propose SPECTRE (Self-Supervised & Cross-Modal Pretraining for CT Representation Extraction), a fully transformer-based foundation model trained exclusively on openly available datasets.

A. Architecture: Two-Stage Hierarchical Transformer

To handle the computational cost of 3D attention, SPECTRE employs a two-stage design:

Local Vision Transformer ( $ViT_\ell$ ):
- Processes the CT volume by partitioning it into non-overlapping 3D patches (e.g., $16 \times 16 \times 8$ voxels).
- Uses Windowed Attention restricted to local 3D windows to capture fine-grained, geometry-aware features.
- Aggregates patch tokens within each window into a compact window-level representation (using a learnable [cls] token).
Global Vision Transformer ( $ViT_g$ ):
- Takes the aggregated window-level tokens from $ViT_\ell$ as input.
- Applies Full Self-Attention across the entire scan (scan-level context) to model long-range dependencies.
- Because the number of windows ( $G$ ) is much smaller than the number of patches ( $m$ ), this stage remains computationally efficient while capturing global semantics.

Key Architectural Innovations:

3D Rotary Positional Embeddings (RoPE): Instead of learned absolute positions, SPECTRE uses 3D RoPE to inject continuous axial coordinates. This preserves relative position information across different resolutions and window sizes.
Anisotropic Tokenization: Patch depths are adjusted (e.g., $8$ voxels vs. $16$ in-plane) to match typical CT voxel spacing, explicitly encoding geometric anisotropy.
Stochastic Shifts: During pretraining, random global rescaling is applied to coordinates to improve robustness to variable voxel spacings and FOVs.

B. Pretraining Strategy: Two-Stage Pipeline

The model is trained in two complementary stages using a massive corpus of ~230,000 CT scans:

Stage 1: Self-Supervised Learning (SSL) with DINOv3:
- Objective: Learn robust, geometry-aware visual features without text.
- Method: Uses a student-teacher framework (DINOv3 style) with masked patch modeling (iBOT) and self-distillation.
- Augmentation: Multi-crop strategy (global and local views) with random flips, intensity rescaling, and noise injection.
- Losses: Combines DINO (global consistency), iBOT (masked token prediction), and KoLeo (regularization to prevent embedding collapse).
Stage 2: Vision-Language Alignment (VLA) with SigLIP:
- Objective: Inject clinical semantics by aligning CT volumes with radiology reports.
- Method: Uses SigLIP (Sigmoid Loss for Language Image Pretraining) instead of standard InfoNCE/CLIP loss.
- Why SigLIP? It uses binary cross-entropy with symmetric sigmoids, which is more robust to the "many-to-many" nature of medical reports (one scan can have multiple findings) and performs better with smaller batch sizes.
- Text Encoder: Uses Qwen3-0.6B with LoRA adapters to encode radiology reports (Findings and Impressions sections).
- Data Augmentation: Reports are paraphrased using an LLM to create multiple semantically equivalent versions, enhancing robustness to stylistic variations.

3. Key Contributions

First Fully Transformer-Based 3D CT Foundation Model: SPECTRE is a pure 3D ViT architecture (no CNN backbones) designed specifically for volumetric medical data, demonstrating that transformers can scale to 3D medical imaging.
Scalable 3D Attention Mechanism: The hierarchical local-global design makes large-scale 3D attention computationally tractable, solving the token explosion problem inherent in 3D volumes.
Geometry-Aware Design: Explicit handling of anisotropic voxel spacing via 3D RoPE and anisotropic patching, allowing the model to generalize across different scanner protocols without retraining.
Open and Generalizable: The model is trained exclusively on public datasets (NLST, CT-RATE, Merlin, etc.), proving that high-performance representations do not require private data. All code and weights are open-source.
Two-Stage Pretraining: Successfully combines SSL for structural understanding with VLA for clinical semantic grounding, outperforming models that use only one approach.

4. Experimental Results

SPECTRE was evaluated across multiple benchmarks, consistently outperforming prior state-of-the-art (SOTA) foundation models.

Zero-Shot Biomarker Prediction (kNN on Frozen Embeddings):
- Evaluated on 6 datasets (LUNA16, DLCS, NSCLC-Radiomics, etc.) for malignancy and survival prediction.
- Result: SPECTRE achieved the highest AUC on 4 out of 6 benchmarks, surpassing models like CT-CLIP, Merlin, and MedImageInsight. This indicates superior transferability of volumetric representations.
Semantic Segmentation (Encoder-Only):
- Integrated into nnU-Net as a drop-in encoder replacement (SEoMT).
- Result: Outperformed other transformer-based models (SwinUNETRv2, UNETR, nnFormer) and was competitive with the convolutional-based nnU-Net ResEnc L, despite lacking a heavy decoder.
Zero-Shot Text-to-Image Retrieval:
- Evaluated on CT-RATE and Merlin datasets.
- Result: Significantly outperformed CT-CLIP and OpenCLIP. Notably, SPECTRE excelled in retrieving images based on "Impressions" (interpretive text), whereas other models struggled, suggesting better integration of clinical reasoning.
Robustness: Demonstrated insensitivity to variations in voxel spacing and anisotropy, and maintained performance even with noisy or truncated text reports.

5. Significance

Paradigm Shift: SPECTRE demonstrates that pure transformer architectures can be successfully scaled to 3D medical imaging, moving away from hybrid CNN-Transformer designs.
Democratization of Medical AI: By training on public data and releasing the model openly, SPECTRE lowers the barrier to entry for institutions without access to massive private datasets or compute clusters.
Clinical Utility: The model's ability to align 3D volumes with free-text reports enables powerful downstream applications like automated report generation, retrieval-based diagnosis, and zero-shot biomarker discovery.
Future Direction: The work establishes a scalable path for general-purpose 3D medical foundation models, suggesting that future improvements should focus on decoder architectures for specific tasks while leveraging these powerful, pre-trained encoders.