🔬 physics

Dual-Domain Equivariant Generative Adversarial Network for Multimodal CT-PET Synthesis

The paper proposes a Dual-Domain Equivariant Generative Adversarial Network (DDE-GAN) that integrates spatial and frequency domain learning with rotational equivariance constraints to achieve superior anatomical fidelity and robustness in multimodal CT-PET image synthesis.

Original authors: Gabriel Steele, Alzahra Altalib, Alessandro Perelli

Published 2026-06-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Gabriel Steele, Alzahra Altalib, Alessandro Perelli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to recreate a detailed, glowing map of a city's energy usage (the PET scan) just by looking at a standard, black-and-white blueprint of the city's buildings (the CT scan).

In the medical world, doctors usually need both maps to diagnose diseases like cancer. But getting both is expensive, takes a long time, and exposes patients to extra radiation. Sometimes, a patient only has the blueprint (CT) and the energy map (PET) is missing or damaged. The goal of this paper is to teach a computer to "imagine" the missing energy map perfectly based on the blueprint.

Here is how the authors, Gabriel Steele and his team, built a new tool called DDE-GAN to do this, explained simply:

1. The Problem: The Old Way Was "Blind"

Previous computer programs tried to do this translation, but they had two main flaws:

They only looked at the picture: They analyzed the image like a human eye does (spatial domain), but they ignored the "music" or "vibrations" hidden inside the image (frequency domain). It's like trying to understand a song by only looking at the sheet music notes, without listening to the rhythm or tone.
They got confused by rotation: If you turned the blueprint sideways, the computer sometimes got confused and drew the energy map upside down or in the wrong shape. In medicine, the body can be scanned from different angles, so the computer needs to understand that a rotated head is still the same head.

2. The Solution: A "Dual-Domain" Detective

The authors created a new system called DDE-GAN that acts like a super-detective with two different sets of eyes:

Eye #1 (The Spatial Domain): This looks at the image normally, seeing the shapes of organs and tumors.
Eye #2 (The Frequency Domain): This looks at the image as a collection of waves and patterns (like turning the image into a sound wave). This helps the computer catch fine details and textures that the first eye might miss.

By using both eyes at the same time, the computer gets a much clearer, more complete picture of what the missing PET scan should look like.

3. The Secret Sauce: "Rotational Equivariance"

This is the most unique part of their invention. The authors realized that the physics of how CT and PET scanners work are naturally "rotationally equivariant."

The Analogy: Imagine you have a spinning top. If you spin the top, the top itself doesn't change; it just rotates. If you take a photo of it, the photo rotates too. The relationship between the object and the photo stays consistent.

The authors taught their computer this rule: "If I rotate the input blueprint, the generated energy map must rotate in the exact same way."

They built a special "rulebook" (a mathematical loss function) into the computer's training. If the computer tries to draw a tumor in the wrong spot just because the image was turned, the rulebook punishes it. This ensures that no matter how the patient is positioned, the computer always draws the anatomy correctly.

4. The Training Process: Three Stages

The computer didn't learn everything at once. It went through three stages of training, like a student progressing from elementary school to college:

Stage 1: It learned to translate within each "eye" (Spatial to Spatial, and Frequency to Frequency).
Stage 2: It learned to make sure the two "eyes" agreed with each other.
Stage 3 (The Final Exam): It applied the Rotation Rule. It practiced translating images that were turned sideways, ensuring the output rotated perfectly with the input.

5. The Results: A Sharper Picture

The team tested their new system on a real-world dataset of head and neck tumors (called HECKTOR 2022).

The Comparison: They compared their new system against older methods (like CycleGAN).
The Outcome: Their new system produced much clearer and more accurate images.
- The "sharpness" score (PSNR) went up by about 2.7 points, which is a huge jump in image quality.
- The "structural similarity" score (SSIM) improved significantly, meaning the shapes of the organs looked much more realistic.
- The images were less "jittery" (more consistent) than the old methods.

Summary

In short, the authors built a smarter computer program that doesn't just "guess" what a missing medical scan looks like. Instead, it:

Listens to the image in two different ways (visual and wave-based) to get all the details.
Respects the physics of rotation, so it never gets confused if the patient is turned sideways.

The result is a highly accurate tool that can generate a missing PET scan from a CT scan, which could help doctors in situations where they only have one type of image available.

Technical Summary: Dual-Domain Equivariant Generative Adversarial Network for Multimodal CT-PET Synthesis

Problem Statement
Multimodal imaging, specifically the combination of Computed Tomography (CT) and Positron Emission Tomography (PET), is critical for oncology, neurology, and cardiology. However, acquiring both modalities increases cost, scan time, and radiation exposure. In many scenarios, one modality may be missing, corrupted, or misaligned, necessitating cross-modality image synthesis (e.g., generating PET from CT). Existing Generative Adversarial Network (GAN) approaches for this task face two primary limitations:

Domain Limitation: Most models operate solely in the spatial (image) domain, failing to leverage complementary spectral information available in the frequency domain.
Geometric Inconsistency: Standard deep networks lack inherent equivariance to spatial transformations (rotation, translation). This can lead to anatomically implausible hallucinations and reduced generalizability when patient orientation varies, despite the fact that CT and PET acquisition physics are inherently rotationally equivariant.

Methodology: DDE-GAN
The authors propose the Dual-Domain Equivariant Generative Adversarial Network (DDE-GAN), a framework designed to synthesize PET images from CT inputs by jointly learning in spatial and frequency domains while enforcing geometric equivariance.

Dual-Domain Architecture: The framework utilizes a Generalised Dual-Domain Generative approach involving four networks: two operating in the image domain ( $G^I_t, G^I_s$ ) and two in the measurement (frequency) domain ( $G^A_t, G^A_s$ ). Here, $F$ represents the linear operator for ray-tracing discretization (forward projection), and $F^{-1}$ represents back-projection.
Hierarchical Training Strategy: Training occurs in three stages to enforce consistency:
- Stage S1 (Intra-domain): Networks are trained individually to reconstruct target images from source data and vice versa within their respective domains (image and measurement) using $L_1$ norm loss functions.
- Stage S2 (Inter-domain): The networks are paired to compare reconstructions across domains. This stage introduces cross-domain consistency constraints, ensuring that the forward projection of a generated image matches the measurement domain representation, and vice versa.
- Stage S3,eq (Equivariant Consistency): This stage integrates the rotational equivariance constraint directly into the loss function. By defining a rotational operator $T_g$ , the model enforces that rotating the input source image and then generating the target yields the same result as generating the target and then rotating it. This is mathematically enforced via specific loss terms comparing rotated inputs and outputs across both image and frequency domains.
Network Architecture: The implementation utilizes a Residual U-Net architecture for the generator and discriminator networks.

Key Contributions

Dual-Domain Learning: The paper introduces a method that processes data in both spatial and Fourier domains simultaneously, allowing the model to capture high-frequency anatomical details and low-frequency contextual patterns.
Equivariant Design: The authors embed rotational equivariance constraints into the loss functions of both the generator and discriminator. This ensures that geometric transformations in the input are consistently reflected in the output, preserving anatomical plausibility.
Novel Framework: To the authors' knowledge, this is the first application of a Dual-Domain Equivariant GAN specifically for CT-PET modality synthesis.

Results and Validation
The method was evaluated on the HECKTOR 2022 dataset, comprising CT-PET acquisitions from 10 different centers. The dataset included 341 acquisitions (approx. 125,852 images for training and 74,400 for testing). The images were preprocessed to a resolution of 256×256 pixels.

Qualitative Analysis: Visual comparisons showed that the DDE-GAN (Stage S3) produced PET images with noticeably improved details compared to the baseline DD-GAN (without equivariance) and CycleGAN.
Quantitative Analysis: The DDE-GAN outperformed baseline models on the test set:
- PSNR: Achieved 28.12 ± 1.92 dB, an improvement of approximately 2.7 dB over the non-equivariant DD-GAN (25.42 dB) and a significant increase over the Stage 2 variant.
- SSIM: Achieved 0.92 ± 0.04, compared to 0.75 for the baseline DD-GAN.
- The results demonstrated reduced variability and higher structural fidelity.

Significance and Claims
The paper claims that combining dual-domain learning with geometric equivariance substantially enhances the accuracy and robustness of multimodal image synthesis. The authors state that the DDE-GAN algorithm improves the quality of synthetic PET images from CT by enforcing the rotational equivariance inherent in the acquisition physics. They conclude that this approach is a "potentially promising method" for clinical applications, specifically for PET completion (generating missing PET data from CT) and data augmentation for multimodal training pipelines. The work suggests that enforcing geometric consistency through equivariant loss constraints is critical for improving structural fidelity and generalization in medical image synthesis.