Retrieval-Augmented Anatomical Guidance for Text-to-CT Generation

This paper proposes a retrieval-augmented framework for text-to-CT generation that leverages a 3D vision-language encoder to retrieve semantically related clinical cases and their anatomical annotations as structural proxies, thereby enhancing image fidelity and spatial controllability in a realistic inference setting without requiring ground-truth annotations.

The Gist

Imagine you are an architect trying to build a 3D model of a human body (specifically a CT scan) based only on a written description from a doctor.

The Problem:
If you just listen to the doctor's notes (e.g., "The patient has a small tumor in the left lung"), you have a great idea of what to build, but you have no idea where to put the walls, the floor, or the other organs.

• Text-only AI is like a painter who hears "a red house" but paints the house floating in the sky or upside down because they don't know the rules of gravity or anatomy.
• Structure-only AI is like a robot that builds a perfect house, but it needs a blueprint (a detailed map of every wall) before it starts. In medicine, we often don't have these blueprints for the new patient we are trying to simulate.

The Solution: The "Retrieval-Augmented" Approach
The authors of this paper came up with a clever trick to solve this. Instead of asking the AI to guess the anatomy from scratch, they gave it a library of past cases.

Here is how their system works, using a simple analogy:

1. The Detective (The Search Engine)

Imagine the AI is a detective. When a new doctor's report comes in, the detective doesn't just stare at the paper. Instead, they run to a massive library of thousands of previous patient cases.

• They look for a past case that sounds very similar to the new one.
• Example: If the new report says "fluid in the lungs," the detective finds an old file about a patient who had "fluid in the lungs."

2. The Ghost Blueprint (The Structural Proxy)

Here is the magic part: The detective doesn't just read the old report; they pull out the anatomical map (the segmentation mask) from that old patient's file.

• They don't copy the old patient's body exactly.
• Instead, they use the old patient's body layout as a "Ghost Blueprint" or a scaffolding.
• Think of it like putting a transparent, slightly blurry outline of a human skeleton over your drawing paper. It tells you, "The heart goes here, the lungs go there," but it doesn't force the final drawing to look exactly like that specific skeleton.

3. The Artist (The Generator)

Now, the AI acts as an artist. It has two things on its desk:

1. The New Doctor's Report (The instructions: "Make it look sick").
2. The Ghost Blueprint (The rules: "Keep the organs in the right places").

The AI uses the report to decide what to paint (the texture, the disease), but it uses the Ghost Blueprint to make sure the painting doesn't look like a monster with a heart in its foot. It fills in the details inside the safe boundaries provided by the retrieved example.

Why This is a Big Deal

• No Magic Required: Usually, to get a perfect anatomical structure, you need a perfect map (which you don't have for new patients). This method "borrows" the map from a similar, existing patient.
• Better than Guessing: Previous AI models that only read text often produced weird, impossible anatomy (like a liver growing out of a head). This method keeps the anatomy realistic.
• Better than Copying: Previous methods that used maps required you to have the exact map for the new patient first. This method finds a similar map on the fly, making it usable for real-world scenarios where data is missing.

The Result

When they tested this on a dataset of chest CT scans:

• The images looked more realistic (higher fidelity).
• Doctors could tell the difference between healthy and sick tissue more easily (better clinical consistency).
• The organs stayed in the right places (better spatial control).

In a nutshell: The paper teaches AI how to build a realistic human body by saying, "Hey, remember that guy we saw last week who had a similar problem? Let's use his body layout as a guide, but fill in the details based on this new patient's notes." It bridges the gap between "what the doctor says" and "how the body is actually built."

Technical Summary

Here is a detailed technical summary of the paper "Retrieval-Augmented Anatomical Guidance for Text-to-CT Generation."

1. Problem Statement

The paper addresses a critical gap in Text-to-CT generation (synthesizing 3D Computed Tomography volumes from radiology reports). Existing methods face a dichotomy:

• Text-Conditioned Methods: These offer semantic flexibility but often produce outputs that are spatially ambiguous or anatomically inconsistent because radiology reports describe pathologies without encoding explicit spatial constraints or normal anatomy.
• Structure-Driven Methods: These use segmentation masks to ensure anatomical precision but require ground-truth annotations at inference time. Since the goal is to synthesize the volume, such annotations are unavailable, making these methods impractical for realistic generation scenarios.

The core challenge is to generate anatomically coherent 3D CT volumes conditioned on text reports without access to the target volume's ground-truth segmentation or structural annotations during inference.

2. Methodology

The authors propose a Retrieval-Augmented Generation (RAG) framework that treats anatomical structure as a "latent, retrievable proxy" rather than a direct input. The pipeline consists of three main components:

A. Problem Formulation

The generation task is modeled as $\hat{x} \sim p_\theta(x | r, m)$, where:

• $r$: The input radiology report (semantic condition).
• $m$: A retrieved structural proxy (coarse anatomical guidance).
• $\hat{x}$: The generated CT volume.
  Crucially, $m$ is not assumed to match the target anatomy perfectly but acts as a spatial scaffold to bias the generation toward anatomical coherence.

B. Retrieval-Augmented Structural Proxy

1. Encoder: A pretrained 3D vision-language encoder (based on CLIP paradigms) maps the input report $r$ into a semantic embedding space.
2. Retrieval: The system retrieves the semantically nearest neighbor from a reference corpus of clinical cases (training set) based on cosine similarity.
3. Proxy Extraction: The anatomical annotation (e.g., segmentation mask) associated with the retrieved case is extracted to serve as the structural proxy $m$.
   • Hypothesis: Semantic similarity in the embedding space correlates with coarse pathological and anatomical patterns, making the retrieved mask a suitable, albeit noisy, structural guide.

C. Anatomical Guidance via ControlNet

The proxy $m$ is integrated into a Latent Diffusion Model (operating in a compressed VAE latent space) using a ControlNet architecture:

• Architecture: A trainable ControlNet branch ($\epsilon_\psi$) runs parallel to the frozen text-conditioned diffusion backbone ($\epsilon_\theta$).
• Injection Mechanism: The ControlNet processes the noisy latent, text embedding, and the structural proxy $m$. It generates residual corrections ($\Delta s, \Delta b$) via zero-initialized convolutional layers.
• Modulation: These residuals are added to the skip connections and bottleneck features of the frozen backbone.
• Training Strategy: The diffusion backbone and vision-language encoder remain frozen. Only the ControlNet branch and projection layers are trained using a rectified flow objective. At inference, the proxy is derived exclusively via retrieval, ensuring no leakage of ground-truth data.

3. Key Contributions

1. RAG Framework for 3D Medical Synthesis: A novel formulation that reinterprets anatomical structure as a retrievable latent proxy, bridging the gap between semantic text conditioning and structural control without requiring inference-time annotations.
2. Multimodal Integration Strategy: The introduction of a ControlNet-based mechanism that injects retrieved anatomical proxies into a text-conditioned latent diffusion model, enabling explicit spatial controllability while preserving semantic variability.
3. Comprehensive Evaluation: Extensive quantitative and qualitative analysis demonstrating that retrieval quality directly impacts generation performance, with semantically aligned proxies yielding superior results across fidelity, clinical consistency, and spatial controllability.

4. Experimental Results

Experiments were conducted on the CT-RATE dataset (paired 3D chest CTs and reports). The proposed method (RAG-Nearest) was compared against text-only baselines (GenerateCT, MedSyn, Text-to-CT) and a structure-conditioned upper bound (MAISI).

• Image Fidelity (FID):
  • RAG-Nearest achieved the lowest FID scores (e.g., 0.303 Avg 2.5D, 0.004 3D), outperforming both text-only models and MAISI.
  • Insight: MAISI performed worse in FID because, while anatomically consistent, it lacked the semantic conditioning required to match the distribution of the test reports.
• Clinical Consistency (CT-Net):
  • RAG-Nearest achieved the highest AUC (0.787) and weighted F1 (0.606), significantly outperforming text-only baselines.
  • Insight: This confirms that retrieving semantically related cases improves the preservation of clinically meaningful patterns.
• Spatial Controllability (Dice & HD95):
  • RAG-Nearest achieved a Dice score of 0.772 and HD95 of 3.072, approaching the performance of MAISI (which uses ground-truth masks, Dice 0.792).
  • Insight: This demonstrates that the retrieved proxy effectively constrains the global thoracic layout, reducing spatial ambiguity without forcing a rigid copy of the anatomy.
• Ablation Study:
  • Retrieving the nearest semantic neighbor yielded the best results.
  • Retrieving the farthest or random cases degraded performance, confirming that the quality of semantic alignment between the report and the retrieved proxy is critical.

5. Significance

This work introduces a principled and scalable mechanism to solve the "anatomical plausibility vs. semantic flexibility" trade-off in medical image synthesis.

• Practical Impact: It enables the generation of realistic, anatomically consistent 3D CT volumes from text reports alone, which is vital for data augmentation, privacy-preserving learning, and simulation where ground-truth annotations are unavailable.
• Methodological Advance: It demonstrates that anatomical structure can be effectively modeled as a latent, retrievable source of information, moving beyond the need for explicit segmentation masks during the generation phase.
• Future Direction: The authors suggest extending this approach to pathology-specific evaluations and longitudinal scenarios to model disease progression.