🤖 AI

OrganLens: Organ-Specific Representation Learning for CT Foundation Models

OrganLens introduces a scalable, self-supervised framework that conditions a shared CT encoder on organ identity to generate 11 distinct organ-specific representations without requiring external segmentation masks, significantly outperforming existing foundation models in downstream tasks such as cardiomegaly detection and lung cancer mortality prediction.

Original authors: Zhixuan Ge, Anqi Li, Sadeer Al-Kindi, Hanwen Xu, Wei Qiu

Published 2026-07-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhixuan Ge, Anqi Li, Sadeer Al-Kindi, Hanwen Xu, Wei Qiu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a giant, three-dimensional puzzle made of light and shadow. This is a CT scan, a special kind of medical photograph that slices through the human body to reveal its inner workings. For decades, computers have been getting very good at looking at these puzzles. They use "foundation models," which are like super-smart students that have read millions of these scans to learn what a healthy body looks like. When these students look at a new scan, they usually give you one big, general summary of the whole picture. It's like a teacher looking at a whole classroom and saying, "This class is doing well," without noticing that one specific student is struggling with math while another is acing art.

But in medicine, the details matter. A doctor doesn't just want to know if the "chest" is okay; they need to know if the heart is enlarged or if the lungs have a hidden spot. The problem is that when a computer looks at the whole chest at once, the signal from the heart gets mixed up with the signal from the lungs, the ribs, and the muscles. It's like trying to hear a single violin in a full orchestra playing at full volume; the specific note gets drowned out. Scientists have been trying to teach computers to focus on just one instrument at a time, but previous attempts were like asking the computer to cut the violin out of the orchestra entirely, which loses the context of how the music fits together, or asking it to sort the notes after the song is already over, which is too late to change how the song sounds.

This is where a new idea called OrganLens comes in. Think of OrganLens not as a pair of scissors that cuts the picture apart, but as a magical pair of glasses with a special dial. When you turn the dial to "Heart," the computer doesn't just look at the heart; it learns to listen to the heart while still seeing the rest of the body in the background. It's like having a translator who can focus on one person in a crowded room, understanding their specific words and tone, while still knowing exactly where they are standing relative to everyone else.

The researchers behind OrganLens built a system that can take a single CT scan and, without needing a human to draw lines around the organs first, generate 11 different "personalities" for that same image. One personality is the "Heart Expert," another is the "Lung Expert," and so on. They trained this system using a clever trick: they showed the computer a picture of an organ and asked it to guess where that organ was, then used that guess to weigh the importance of different parts of the image. If the computer thinks a patch of pixels belongs to the heart, it listens to that patch more closely when answering heart-related questions.

The results are quite promising. When the researchers tested this new "dial" on real medical data, they found that the organ-specific views were much better at spotting problems than the old "one-size-fits-all" views. For example, when looking for an enlarged heart, the "Heart Expert" view was correct 95.3% of the time, a big jump from the previous best of 91.0%. Similarly, when predicting the risk of lung cancer death, the "Lung Expert" view improved the accuracy by 14.2% compared to the standard models. The system was also surprisingly good at finding the right picture when given a text description, suggesting that by understanding the specific parts, the computer understands the whole story better.

However, the authors are careful to say that while this is a powerful new tool for research, it is not yet a magic wand for the clinic. They tested it on several large groups of patients and found that the "organ-specific" approach consistently provided clearer signals for specific diseases than the general approach. But they also note that this was a retrospective study, meaning they looked at past data, and the system still needs to be tested in real-time hospital settings to prove it helps doctors make better decisions for living patients. For now, OrganLens offers a fascinating new way to look at the human body: not as a single, blurry blob, but as a collection of distinct, focused stories, all waiting to be heard.

Technical Summary: OrganLens

Problem Statement

Computed Tomography (CT) examinations capture multiple organ systems within a single volume, yet many biomedical inquiries—ranging from abnormality detection to prognostic prediction and longitudinal monitoring—focus on specific organs. Existing CT foundation models typically generate a single, volume-level representation. This approach dilutes organ-specific evidence because it must integrate features from organs of varying sizes, appearances, and clinical relevance, often obscuring the signal required for specific tasks.

Current anatomy-aware approaches attempt to address this through three primary methods, each with limitations:

Pre-segmented inputs: Encoding isolated organ volumes removes clinically relevant surrounding context necessary for detecting abnormalities.
Token disentanglement: Separating images into organ-wise token groups without explicit conditioning on a selected organ can retain irrelevant features and fail to concentrate the signal.
Text-derived supervision: Localizing anatomy via report-region alignment often lacks the spatial precision to distinguish adjacent organs.

Furthermore, training separate encoders for each organ is computationally inefficient and prevents representation sharing. The core challenge is to develop a shared encoder that can adapt its representation to a selected organ, preserve surrounding anatomical context, and produce anatomy-weighted representations without requiring manual segmentation masks at inference.

Methodology

The authors introduce OrganLens, a framework for organ-specific representation learning built upon a shared CT-pretrained DINOv2 encoder. The model produces 11 distinct organ-specific representations (heart, lung, aorta, liver, spleen, kidneys, pancreas, stomach, intestine, esophagus, and trachea) from the same CT volume using a single set of parameters.

The framework relies on three key components during self-supervised pretraining:

Organ-Identity Conditioning:
A learned embedding for a specific organ identity is added as an offset to the CLS token of the Vision Transformer (ViT). This conditions the shared encoder to adapt its features to the selected organ without altering the backbone architecture. The strength of this offset is learned during optimization.
Organ-Guided Distillation:
Adapting the DINOv2 student–teacher framework, the model performs self-distillation using views constructed from the CT slice and the corresponding organ mask.
- Cropping: Global and local crops are sampled such that they retain the foreground anatomy specified by the organ identity.
- Training: The student processes both global and local crops, while the teacher (updated via exponential moving average) processes only global crops. This ensures both networks learn features relevant to the specific organ identity.
Anatomy-Mask Supervision:
To ensure patch features localize the queried organ, an auxiliary decoder predicts a spatial mask for the selected organ from the student's patch features.
- Loss: The decoder is trained using a combination of pixel-wise binary cross-entropy and a differentiable Dice-based loss against ground-truth masks (derived from TotalSegmentator pseudo-labels during pretraining).
- Inference Mechanism: Crucially, at inference, the ground-truth mask is not required. The trained decoder predicts a soft spatial mask, which serves as weights for pooling patch features.

Representation Construction:
At inference, for a given organ identity $o$ and CT volume $V$ :

The encoder produces patch features and a CLS feature conditioned on $o$ .
The anatomy decoder predicts a spatial probability map for the organ.
Patch features within each slice are pooled using these soft weights to create a slice representation.
Slice representations are aggregated across the volume, weighted by the predicted organ area in each slice, to form the final organ-specific representation $z(V, o)$ .
A global representation is also defined by concatenating all 11 organ-specific representations, enabling whole-volume tasks.

Key Contributions

OrganLens Framework: A shared encoder architecture that learns organ-specific features while retaining surrounding anatomical context, producing 11 distinct representations from a single CT scan without external segmentation at inference.
Self-Distillation Strategy: A novel training pipeline combining organ-identity conditioning, organ-guided cropping, and anatomy-mask supervision to shape features for anatomy-weighted pooling.
Clinical Utility Demonstration: Empirical evidence showing that anatomically matched representations provide finer-grained, task-relevant features for organ-focused downstream applications, while the global aggregation retains broad utility for whole-volume tasks.

Experimental Results

The authors evaluated OrganLens on four diverse cohorts: CT-RATE, RAD-ChestCT, INSPECT, and the National Lung Screening Trial (NLST). The evaluation covered four task families: abnormality detection, prognostic prediction, longitudinal analysis, and image–text retrieval.

Abnormality Detection:
- On CT-RATE (16 abnormalities), the anatomically matched OrganLens representation achieved a macro-AUROC of 0.856, outperforming CT-pretrained DINOv2 (0.788), GigaHeart (0.792), Merlin (0.780), and SPECTRE (0.836).
- Specific gains were notable: Heart representations improved cardiomegaly detection from 0.910 to 0.953 AUROC compared to DINOv2. Lung representations improved mosaic attenuation pattern detection to 0.906.
- On the external RAD-ChestCT dataset, OrganLens achieved a macro-AUROC of 0.767, significantly outperforming all baselines (e.g., 0.684 for DINOv2).
Prognostic Prediction:
- On INSPECT and NLST, anatomically matched representations consistently ranked highest. For 16 of 17 endpoints, the matched organ representation outperformed other OrganLens variants.
- In NLST, the lung representation improved the Harrell C-index for lung-cancer mortality by 14.2% relative to CT-pretrained DINOv2 (0.780 vs. 0.686).
- Kaplan–Meier curves demonstrated that high-risk quartiles identified by OrganLens showed consistently lower event-free survival, validating the prognostic stratification.
Longitudinal Analysis:
- OrganLens achieved the strongest overall C-index performance across T0, T1, and T2 screening visits in NLST.
- Cox log-hazard scores for participants who later experienced mortality events increased monotonically from T0 to T2, demonstrating the model's ability to capture longitudinal risk progression.
Image–Text Retrieval:
- The global OrganLens representation (concatenation of all 11 organ features) achieved the best performance in both text-to-image and image-to-text retrieval on CT-RATE and INSPECT.
- On INSPECT, OrganLens achieved a Recall@10 of 33.09% for text-to-image and 32.04% for image-to-text, outperforming all baselines.
Ablation Studies:
- Removing anatomy-mask supervision or organ-guided cropping resulted in significant performance drops, confirming that both components are essential for learning localized, organ-specific signals.
- Anatomy-weighted pooling (using the predicted mask) outperformed uniform patch pooling and mean CLS pooling.

Significance and Claims

The paper claims that OrganLens offers a scalable approach to organ-specific CT representation learning. By conditioning a shared encoder on an organ identity, it resolves the mismatch between volume-level representations and organ-focused biomedical questions.

The authors assert that:

Anatomical Matching is Critical: Predictive signals for specific diseases are concentrated in anatomically matched organs rather than distributed uniformly across the volume.
Context Preservation: Unlike methods that crop out surrounding tissue, OrganLens retains useful anatomical context, which is vital for abnormality detection.
Inference Efficiency: The model produces organ-specific representations without requiring external segmentation masks at inference, relying instead on a predicted spatial mask from the trained decoder.
Dual Utility: The framework supports both granular, organ-specific analyses (via matched representations) and broad, whole-volume tasks (via global aggregation).

The authors conclude that OrganLens provides a reusable framework for the medical research community to study organ-specific disease across cohorts and clinical endpoints, offering a foundation for future applications in risk stratification and longitudinal monitoring. They note limitations, including the retrospective nature of the evaluation, reliance on pseudo-labels for pretraining, and the need for prospective validation before clinical deployment.