🤖 AI

Anatomy Contextualized Adaption of CT Foundation Models

The paper introduces Anatomy Contextualized Adaptation (ACA), a lightweight framework that efficiently adapts frozen CT foundation models to achieve anatomy-level vision-language alignment while preserving global context, outperforming existing methods in zero-shot tasks with minimal training time.

Original authors: Roshan Kenia, Stephanie L McNamara, William Lotter

Published 2026-07-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Roshan Kenia, Stephanie L McNamara, William Lotter

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to read medical scans, like a CT scan of a human body. This isn't just a flat picture; it's a 3D block of data, like a giant, invisible Lego tower made of thousands of tiny slices. In the world of artificial intelligence, there are "foundation models"—these are like giant, pre-trained brains that have already looked at millions of these Lego towers. They are incredibly good at spotting general patterns, kind of like how a human can look at a blurry photo and say, "That's a dog."

However, there's a catch. These giant brains usually look at the entire Lego tower as one big, blurry blob. They might tell you, "There's something wrong in this whole section," but they struggle to point a finger and say, "The liver is too big, but the kidneys are fine." To fix this, scientists have tried to teach the robot to look at individual organs one by one. But doing this from scratch is like trying to build a new car engine every time you want to drive to the store; it takes forever and uses up a massive amount of energy. The big question is: Can we take that already-smart, pre-trained brain and teach it to look at the tiny details without rebuilding the whole thing?

This is exactly what the paper "Anatomy Contextualized Adaption of CT Foundation Models" tackles. The authors, Roshan Kenia, Stephanie L McNamara, and William Lotter, introduce a clever new framework called ACA (Anatomy Contextualized Adaption). Think of ACA as a lightweight "translator" or a "smart adapter" that plugs into the giant, frozen brain. Instead of retraining the whole brain, ACA takes the giant brain's general knowledge and slices the 3D scan into 44 different anatomical parts (like the liver, heart, lungs, and kidneys) using a tool called TotalSegmentator.

Here is where the magic happens. Usually, if you just look at a liver in isolation, you might miss the context. Is it big because it's sick, or just because the person is tall? ACA solves this by using a special "inter-anatomy transformer." Imagine this as a group chat where every organ gets to talk to every other organ. The liver says, "Hey, I'm looking a bit swollen," and the heart replies, "Well, I'm normal size, so maybe you are." This allows the model to understand the relationship between organs, not just the organs themselves. It combines the "big picture" view of the whole scan with the "fine-grained" details of each part.

The researchers tested this on two massive datasets of CT scans and radiology reports: Merlin (focused on the abdomen) and CT-RATE (focused on the chest). They compared their new ACA method against the original giant brains and other fine-grained methods. The results were promising: ACA consistently outperformed the others in identifying specific medical findings without needing to see any labeled examples first (a "zero-shot" test). For instance, on the Merlin dataset, the average accuracy score (AUROC) jumped from about 0.7729 with the original model to 0.8213 with ACA. On the CT-RATE dataset, it went from 0.7082 to 0.7311.

One of the most interesting findings is how the model "thinks." When the authors looked at the attention weights (which show which organs the model is focusing on), they saw that the model learned plausible connections. For the abdominal dataset, the stomach, small bowel, and colon paid a lot of attention to each other, which makes sense because they are connected in a line. For the chest dataset, the lungs and heart were the main focus. The model even figured out that to spot an enlarged spleen (splenomegaly), it needed to compare the spleen's size to the surrounding organs, something the older "look-at-everything-independently" models failed to do.

The paper suggests that this approach is a practical, low-cost way to make medical AI smarter. It proves that you don't need to burn millions of dollars in computing power to get better results; you just need to add a smart, lightweight layer that helps the AI understand how body parts relate to one another. While the authors note that their comparisons to some other methods were done under specific constraints (using frozen models rather than retraining from scratch), the results strongly suggest that this "contextualized adaptation" is a powerful step forward for making AI doctors more precise and reliable.

Technical Summary: Anatomy Contextualized Adaption of CT Foundation Models

Problem Statement

CT vision-language foundation models (e.g., Merlin, CT-CLIP) have demonstrated strong zero-shot performance across downstream tasks by training on whole-volume representations. However, these models typically condense an entire 3D CT volume into a single embedding, which dilutes fine-grained anatomical signals. Conversely, existing fine-grained vision-language pre-training (FVLP) approaches align anatomy-specific visual features with specific text descriptions but often discard the global context provided by whole-volume models. Furthermore, current FVLP methods generally require training the entire model from scratch, making them computationally expensive and difficult to scale. There is a need for a framework that adapts frozen foundation models to achieve anatomy-level alignment while preserving and enhancing global contextual information without the cost of full retraining.

Methodology: Anatomy Contextualized Adaptation (ACA)

The authors introduce Anatomy Contextualized Adaptation (ACA), a lightweight framework designed to adapt frozen CT foundation model representations for anatomy-level vision-language alignment. The methodology consists of three core components:

Embedding Construction:
- Visual: The framework uses TotalSegmentator to decompose CT volumes into 44 anatomical structures. These segmentation masks are projected onto the spatial feature maps of a frozen foundation model backbone (either Merlin's I3D ResNet152 or CT-CLIP's CT-ViT). Features corresponding to each anatomical structure are mean-pooled to create anatomy-level embeddings.
- Text: Anatomy-specific findings are extracted from radiology reports using a Large Language Model (Qwen3-4B-Instruct). These findings are encoded using the frozen text encoder of the respective foundation model to produce anatomy-level text embeddings. A separate scan-level text embedding is generated from the full report.
- Positional Encoding: Inter-anatomy spatial position encodings are computed based on the $\ell_2$ distance between the voxel-space centroids of anatomical structures, normalized by the volume diagonal.
Inter-Anatomy Transformer:
- To capture cross-anatomy relationships, the anatomy embeddings are passed through a learnable inter-anatomy transformer.
- Before entering the transformer, each embedding is augmented with learned spatial position encodings and learnable anatomy type embeddings.
- This module contextualizes each anatomy's embedding by attending to the full set of structures present in the scan, enabling the model to learn relationships between organs (e.g., relative size comparisons).
Combined Loss Function:
- Anatomy-Level Loss ( $\mathcal{L}_{anatomy}$ ): A contrastive loss that aligns each contextualized anatomy embedding with its corresponding extracted text description. This loss employs a soft target matrix to handle false negatives, treating "normal" instances from different patients as positive pairs to avoid penalizing semantically identical healthy structures.
- Scan-Level Loss ( $\mathcal{L}_{scan}$ ): A contrastive loss that aligns the mean-pooled scan-level visual embedding (derived from the contextualized anatomy embeddings) with the full radiology report embedding. This provides a global supervision signal to ensure the model maintains a coherent view of the entire scan.
- The total objective is the sum of these two losses ( $\mathcal{L}_{total} = \mathcal{L}_{anatomy} + \lambda \mathcal{L}_{scan}$ ).

Key Contributions

Lightweight Adaptation: ACA adapts frozen foundation models using only lightweight trainable modules (the transformer and projection heads), avoiding the computational cost of end-to-end retraining required by existing FVLP methods.
Contextualized Fine-Grained Alignment: Unlike previous fine-grained methods that treat anatomies independently, ACA introduces an inter-anatomy transformer that explicitly models cross-anatomy relationships, addressing the limitation of ignoring global context in fine-grained approaches.
Dual-Granularity Supervision: The framework uniquely combines anatomy-level contrastive learning with scan-level report alignment, allowing the model to learn both specific organ pathology and global clinical context simultaneously.

Results

The framework was evaluated on two large-scale datasets: Merlin (abdominal CT) and CT-RATE (chest CT), using their respective frozen foundation models as backbones.

Performance Gains: ACA consistently outperformed both the frozen foundation model baselines and existing fine-grained adaptation methods (including MLP, fVLM, and ViSD-Boost adaptations) in zero-shot finding classification.
- On the Merlin dataset, ACA improved the in-distribution macro-average AUROC from 0.7729 (Merlin baseline) to 0.8213.
- On the CT-RATE dataset, it improved the in-distribution AUROC from 0.7082 (CT-CLIP baseline) to 0.7311.
- Significant gains were observed in out-of-distribution settings as well (e.g., Merlin-trained ACA on CT-RATE test set: 0.7400 vs. 0.6919 baseline).
Ablation Studies: Removing either the anatomy-level loss or the scan-level loss resulted in decreased performance, confirming that both local anatomical alignment and global context supervision are necessary.
Anatomy-Guided Pooling: An inference-time strategy that restricts pooling to anatomically relevant structures for specific findings further improved performance, particularly for models without scan-level supervision.
Learned Associations: Attention weight analysis revealed that the transformer learned plausible cross-anatomy associations (e.g., stomach-small bowel-colon continuity in Merlin; lung-esophagus in CT-RATE) purely from contrastive training, reflecting the anatomical scope of the training data.
Case Study: In detecting splenomegaly, ACA correctly identified pathological enlargement by comparing the spleen's size relative to surrounding organs, whereas independent fine-grained baselines failed to make this relative assessment.

Significance and Claims

The paper claims that ACA provides a practical and compute-efficient path for improving representation learning in medical imaging. By demonstrating that frozen foundation models can be effectively adapted to anatomically grounded vision-language tasks with less than one hour of training (once embeddings are cached), the authors argue that full end-to-end retraining is not strictly necessary to achieve state-of-the-art fine-grained performance.

The authors position ACA as a solution that bridges the gap between the broad, global context of whole-volume models and the specificity of fine-grained alignment. They note that while their comparisons to other fine-grained methods (fVLM, ViSD-Boost) are conducted under a matched "frozen embedding" setting (rather than comparing against the original end-to-end trained versions), the results support the efficacy of their adaptation strategy. The work is limited to zero-shot finding classification, with segmentation and report generation left for future work.