⚡ electrical engineering

Multimodal pseudo-CT synthesis for PET attenuation correction using separate modality encoding and topogram conditioning

This paper presents a multimodal 3D patch-based U-Net that generates pseudo-CT images for PET attenuation correction by integrating separate PET and MRI encoders with FiLM-based topogram conditioning to effectively fuse cross-modal information while minimizing reliance on precise voxel-wise alignment.

Original authors: Rory Bell, Artemis Bouzaki, Jiaming Cao, Jasmine Morrison, Chelsea Sargeant

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rory Bell, Artemis Bouzaki, Jiaming Cao, Jasmine Morrison, Chelsea Sargeant

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of medical imaging, doctors often rely on two powerful tools to see inside the human body: Positron Emission Tomography, or PET, which reveals how organs are functioning by tracking radioactive tracers, and Computed Tomography, or CT, which uses X-rays to create detailed maps of bone and tissue density. To make a PET scan useful, the computer needs to know exactly how much the body's tissues will block or absorb the radiation as it travels through; this process is called attenuation correction. Traditionally, doctors have used a CT scan to provide this map. However, a CT scan adds a significant dose of ionizing radiation to the patient, which is a concern, especially for those who need frequent monitoring. The challenge, then, is to create a digital version of that density map—a "pseudo-CT"—using only the information already present in the PET scan and a magnetic resonance image, or MRI, without exposing the patient to any extra X-rays.

A team of researchers from the University of Manchester tackled this problem by entering a competition designed to test exactly this capability. They built a sophisticated computer system capable of looking at three different types of medical images simultaneously: the raw PET scan, an MRI, and a 2D projection image known as a topogram, which is essentially a quick, low-dose X-ray snapshot of the body's overall shape. Their goal was to teach a machine to combine these distinct sources of information to generate a high-quality 3D map of tissue density that could replace a full CT scan. The researchers found that by treating each type of image differently and then carefully merging their insights, they could create a model that produces accurate density maps, which in turn allows for clearer and more precise PET images without the need for a separate CT scan.

The core of their approach was a digital architecture designed to respect the unique nature of each image type. Unlike previous methods that might have simply stacked the different images on top of each other as if they were layers of a single cake, this team recognized that the PET scan, the MRI, and the 2D topogram do not always line up perfectly in space. Instead, they built separate pathways for the 3D PET and MRI data, allowing the computer to learn the specific features of each modality independently. As these separate streams of information moved deeper into the system, they were brought together at multiple levels of detail. At each stage, the system used a mechanism to weigh the importance of the combined information, ensuring that the most useful details from both the PET and MRI were preserved while less relevant noise was filtered out.

The 2D topogram was handled in a unique way because it represents the entire body in a flat projection rather than a 3D volume. Since it cannot be matched point-for-point with the 3D slices of the PET or MRI, the researchers used it as a guide for the overall shape and anatomy of the patient. They fed this 2D image into a pre-trained system that extracted a summary of the body's form. This summary was then used to gently nudge and adjust the main 3D processing pathway, effectively telling the system, "Remember, this patient has this specific body shape," without forcing the 2D image to fit into the 3D grid. This method allowed the system to use the topogram's global view to refine the local details of the generated map, a technique that proved more effective than trying to force a rigid spatial alignment between the different image types.

When the researchers tested their final system on four subjects from an internal validation set, the results were promising. The computer-generated maps were remarkably close to the real CT scans, with very small average differences in the density values. More importantly, when these generated maps were used to correct the PET scans, the resulting images showed a high degree of accuracy. The average error in the standardized uptake values—a key measure of how much tracer the body absorbs—was extremely low, and the bias in specific organs was minimal. The system also performed well in the brain, a critical area for detecting abnormalities, with very few outliers. These findings suggest that the method successfully integrated the complementary strengths of the different imaging modalities to create a reliable substitute for a CT scan.

The team also explored whether adding extra variations to the training data, such as shifting or rotating the images to mimic real-world imperfections, would improve the results. They found that these common techniques did not help; in fact, the model trained on the original, unaltered data performed better than any version that had been subjected to these augmentations. This indicates that the specific architecture they designed, with its separate encoding paths and the unique way it handled the 2D topogram, was robust enough to handle the natural variations in the data without needing artificial tricks to force it to learn. The study concludes that this multimodal approach offers a viable path toward high-quality PET attenuation correction that avoids the extra radiation dose of a CT scan, relying instead on a clever synthesis of existing imaging data.

Technical Summary: Multimodal Pseudo-CT Synthesis for PET Attenuation Correction

Problem Statement
In Positron Emission Tomography (PET) imaging, Computed Tomography (CT) is traditionally required for attenuation correction (AC). However, the use of CT introduces additional ionizing radiation exposure to patients. The BIC-MAC Challenge addresses this issue by exploring the generation of pseudo-CTs from non-attenuation-corrected PET (NAC-PET), MRI, and 2D topograms. The objective is to synthesize high-quality pseudo-CTs that enable accurate PET reconstruction without the need for a dedicated CT scan, utilizing a training set of 75 subjects.

Methodology
The authors propose a multimodal, 3D patch-based U-Net architecture designed to handle misalignment between input modalities and integrate complementary information effectively.

Modality-Specific Encoding: Recognizing that NAC-PET and MRI (in-phase and out-phase) are not perfectly aligned, the architecture employs separate 3D encoders for PET and MRI rather than early channel-wise fusion. This allows each modality to learn distinct feature representations before integration.
Multi-Scale Feature Fusion: At each of the four encoder scales, feature maps from the PET and MRI encoders are concatenated channel-wise. These are then projected back to a single-modality channel width using a $1\times1\times1$ convolution. Crucially, a Squeeze-and-Excitation (SE) gate reweights these fused representations. The SE gate utilizes global average pooling, a channel-reduction bottleneck (ratio 4), and a sigmoid activation to adaptively reweight channels, ensuring complementary information is prioritized.
Topogram Conditioning via FiLM: The 2D topogram, which provides global anatomical and body-shape information but lacks direct spatial correspondence with 3D patches, is processed separately. A pretrained ResNet-18 (adapted for single-channel input) extracts a 256-dimensional feature vector. Instead of spatial fusion, these features condition the shared U-Net bottleneck using Feature-wise Linear Modulation (FiLM). The topogram features learn scaling ( $\gamma$ ) and shifting ( $\beta$ ) parameters applied as $b' = b(1 + \gamma) + \beta$ , where $b$ is the PET-MR bottleneck feature. The FiLM layer is zero-initialized, ensuring the topogram influence is learned progressively from an identity mapping.
Decoder and Reconstruction: The conditioned features are reconstructed through a shared decoder. The architecture uses resize convolutions (trilinear upsampling followed by convolution) instead of transposed convolutions to mitigate checkerboard artifacts. The final output is generated via a $1\times1\times1$ convolution followed by a sigmoid activation, consistent with input normalization to $[0,1]$ .

Implementation and Training Details

Architecture: Base channel widths of 32, scaling to 64, 128, 256, and a 512-channel bottleneck. Blocks utilize two $3\times3\times3$ convolutions with instance normalization and LeakyReLU.
Data Preprocessing: Volumetric inputs are intensity-normalized. NAC-PET and MRIs are scaled from 0.5–99.5 percentiles to $[0,1]$ . CT values are clipped to $[-1000, 2000]$ HU and mapped to $[0,1]$ . Topograms are resized to $224\times224$ .
Training Strategy: The model was trained on 71 subjects with 4 used for internal validation. Patches of $64\times64\times64$ were sampled using positive-only foreground sampling. The model was trained for 250 epochs using an AdamW optimizer with L1 loss evaluated within the prediction mask. Notably, extensive ablation studies (including data augmentation, attention-based fusion, and registration networks) showed that the unaugmented model described above performed best on whole-body CT Mean Absolute Error (MAE) and organ bias. Consequently, the final submission used the model trained without data augmentation.

Results
The final model demonstrated strong performance on the internal validation set:

CT Mu-map MAE: 0.005454
Whole-body PET SUV MAE: 0.0379
Region-specific PET Organ Bias: 2.45%
Brain Outlier Score: 0.024

Visual inspection of the generated pseudo-CTs for four validation subjects confirmed the model's ability to produce anatomically consistent outputs when masked to the body region.

Significance and Claims
The paper claims that the proposed architecture successfully integrates complementary cross-modal information while reducing reliance on precise voxel-wise correspondence between modalities. By utilizing separate encoders for volumetric data and FiLM-based conditioning for the 2D topogram, the model effectively leverages global anatomical context without forcing spatial alignment that does not exist. The authors conclude that their multimodal approach yields promising performance for downstream PET attenuation correction, offering a viable pathway to reduce radiation exposure in clinical PET imaging.

Technical Summary: Multimodal Pseudo-CT Synthesis for PET Attenuation Correction

More like this