💻 computer science

Supervise-assisted Multi-modality Fusion Diffusion Model for PET Restoration

This paper proposes a supervise-assisted multi-modality fusion diffusion model (MFdiff) that leverages anatomical MR images and a two-stage learning strategy to effectively restore high-quality standard-dose PET images from low-dose inputs while addressing multi-modality fusion inconsistencies and out-of-distribution data challenges.

Original authors: Yingkai Zhang, Shuang Chen, Ye Tian, Yunyi Gao, Jianyong Jiang, Ying Fu

Published 2026-02-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yingkai Zhang, Shuang Chen, Ye Tian, Yunyi Gao, Jianyong Jiang, Ying Fu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Blurry Photo" Dilemma

Imagine you are trying to take a photo of a busy city street at night.

The Goal: You want a crystal-clear picture (Standard-Dose PET) to see every detail of the cars and people.
The Problem: To get a clear picture, you need a lot of light. But in medical imaging, "light" is radiation. Too much radiation is bad for the patient.
The Compromise: Doctors use less radiation (Low-Dose PET) to keep patients safe. The result? The photo comes out grainy, blurry, and full of "static" (noise). It's like looking at the city through a foggy window.

Doctors also have a second camera: an MRI. This camera doesn't use radiation; it uses magnets to take incredibly sharp, detailed pictures of the body's structure (bones, organs, tissues).

The Challenge: Can we use the sharp MRI picture to fix the blurry PET picture without accidentally copying the wrong things?

The Risk: If you just mash the two pictures together, the MRI might "hallucinate" details into the PET scan. For example, the MRI might show a healthy bone structure, but the PET scan (which tracks metabolism) might show a tumor there. If you aren't careful, your fix might erase the tumor because the MRI says "everything looks normal here."

The Solution: The "Master Chef" (MFdiff)

The authors propose a new AI system called MFdiff. Think of it as a Master Chef who is trying to restore a ruined, blurry soup (the Low-Dose PET) into a gourmet dish (the Standard-Dose PET), using a perfect recipe book (the MRI) as a guide.

Here is how the Chef works, broken down into three main steps:

1. The Smart Tasting Spoon (Multi-modality Feature Fusion)

Before cooking, the Chef needs to understand the ingredients.

The Old Way: Previous chefs just dumped the MRI and PET ingredients into the same bowl. This caused a mess where the MRI's "flavor" (anatomical details) overpowered the PET's "flavor" (metabolic activity), leading to errors.
The MFdiff Way: The Chef uses a Dual-Branch Tasting Spoon.
- Branch A (Global): Looks at the big picture. "Okay, the MRI says this is a brain. The PET says there is activity here. Let's align the general shape."
- Branch B (Detail): Looks at the tiny grains. "The MRI shows a specific bone edge. The PET shows a fuzzy spot. I need to keep the PET's fuzzy spot because it might be a tumor, even if the MRI doesn't show it."
- The Result: The Chef creates a "Perfect Fusion Ingredient" that respects the MRI's structure but protects the unique, vital details of the PET scan.

2. The Sculptor's Clay (Diffusion Model)

Now that the Chef has the perfect ingredients, how do they shape the dish?

The Old Way: Traditional AI tries to guess the final image in one giant leap. It's like trying to sculpt a statue by throwing clay at a wall and hoping it sticks. It often results in blurry or weird artifacts.
The MFdiff Way: The system uses a Diffusion Model. Imagine a sculptor starting with a block of noisy, chaotic clay (pure static).
- The AI doesn't guess the whole image at once. Instead, it takes tiny, iterative steps to remove the noise, refining the image over and over again.
- At every single step, it asks the "Perfect Fusion Ingredient" (from Step 1): "Does this look right? Does it match the MRI's structure but keep the PET's unique details?"
- Slowly, the chaos turns into a high-definition, crystal-clear image.

3. The Two-Stage Training (Supervise-assisted Learning)

This is the secret sauce that makes the Chef adaptable to real-world messiness.

Stage 1: The Simulation School (External Stage).
- Real medical data is hard to get (privacy issues, radiation risks). So, the Chef first trains in a virtual simulation lab. They practice on thousands of fake, computer-generated brains.
- Goal: Learn the general rules of anatomy and how noise looks. This gives the Chef "Generalized Priors" (common sense).
Stage 2: The Real Kitchen Internship (Internal Stage).
- The Chef then moves to a real kitchen but with a small team. They practice on a tiny amount of real patient data (which might be weird or different from the simulation).
- Goal: Fine-tune their skills to handle the specific quirks of real patients (different scanners, different injection doses). This gives them "Specific Priors."
- Why this matters: Most AI fails when it meets a new type of scanner or a different hospital. This two-step method ensures the Chef is smart enough to handle any kitchen, not just the one they trained in.

The Results: Why It Matters

The paper tested this "Master Chef" against other top methods.

Quantitatively: The numbers (PSNR, SSIM) show MFdiff produces significantly clearer images with less error than anyone else.
Qualitatively (The Visuals):
- In the simulation tests, other methods accidentally erased tumors because they trusted the MRI too much. MFdiff kept the tumors visible.
- In real patient tests (where the data was "Out of Distribution"—meaning the scanner settings were different than what the AI was trained on), MFdiff still produced sharp, usable images.

The Bottom Line

This paper introduces a smart, two-step AI system that uses Diffusion Models (iterative refinement) and Multi-modality Fusion (careful blending of MRI and PET) to turn low-radiation, blurry medical scans into high-quality, diagnostic-ready images.

It solves the problem of "too much radiation" by using AI to "fill in the blanks" intelligently, ensuring that doctors get the clarity they need without exposing patients to unnecessary danger. It's like having a super-powered photo editor that knows exactly which details to sharpen and which to ignore, even when the original photo is terrible.

1. Problem Statement

Positron Emission Tomography (PET) is a vital functional imaging modality for diagnosing cancer and neurological diseases. However, it involves significant radiation exposure. To adhere to the "As Low As Reasonably Achievable" (ALARA) principle, clinicians often reduce the radiotracer dose or scan time. This reduction leads to Low-Dose PET (LPET) images with low signal-to-noise ratios (SNR), compromising diagnostic quality.

Restoring standard-dose PET (SPET) images from LPET is challenging due to two main factors:

Multi-modality Inconsistency: While Magnetic Resonance (MR) images provide clear anatomical guidance, they often contain structural details that do not correspond to metabolic activity in PET. Naive fusion (e.g., simple concatenation) can introduce "hallucinated" anatomical artifacts into the PET image, obscuring true lesions or creating false positives.
Out-of-Distribution (OOD) Data: Medical datasets are often small due to privacy and ethical constraints. Furthermore, variations in scanners, radiotracers, and protocols create OOD data. Models trained on specific datasets often fail to generalize to these real-world clinical variations.

2. Methodology: MFdiff

The authors propose MFdiff, a Supervise-assisted Multi-modality Fusion Diffusion Model. The framework consists of three core components:

A. Multi-modality Feature Fusion Module

This module addresses the challenge of integrating MR and PET data without introducing extraneous artifacts. It comprises two sub-modules:

Intra-Modality Learning (IML):
- Modality Encoder: Uses two independent Transformer encoders (shared architecture, different parameters) to extract initial features from LPET and MR, preserving modality-specific characteristics.
- Global/Detailed Encoder: A dual-branch extractor.
  - Global Branch: Uses Multi-Head Attention (Transformer) to capture high-level global anatomical consistency between PET and MR.
  - Detail Branch: Uses Invertible Residual Blocks (IRB) based on invertible neural networks. This ensures lossless extraction of fine details from the original LPET, preventing the high-frequency textures of MR from overwriting the lower-resolution PET signals.
Cross-Modality Aggregation (CMA):
- Employs a dual-fusion strategy with a Global Fusion Layer and a Detail Fusion Layer.
- This module learns an optimized fusion representation that selectively integrates shared information while filtering out inconsistent anatomical structures (pseudo-structures) from the MR that do not match PET metabolic activity.

B. Conditional Diffusion Restoration Module

Instead of a direct mapping (like a standard GAN or CNN), MFdiff uses a Denoising Diffusion Probabilistic Model (DDPM) to iteratively generate high-quality SPET images.

Conditioning: The fused feature vector ( $x_{fusion}$ ) derived from the CMA module, along with the original LPET image ( $x_P$ ), is injected as a conditional constraint into the reverse diffusion process.
Process: The model learns to predict noise at each step $t$ given the noisy state $x_t$ , time $t$ , the LPET input, and the fusion feature. This allows the model to generate complex details and textures more accurately than deterministic models.

C. Two-Stage Supervise-Assisted Learning Strategy

To overcome data scarcity and OOD challenges, the authors introduce a two-stage training pipeline:

External Stage (Generalized Priors): The model is trained on a large-scale simulated phantom dataset (BrainWeb). This allows the model to learn generalized priors regarding noise distribution and basic anatomical fusion without the cost of acquiring massive clinical paired data.
Internal Stage (Specific Priors): The pre-trained model is fine-tuned on a small cohort of real in-vivo clinical data (OOD data). This adapts the model to specific clinical variations (scanner types, tracer protocols) without requiring large-scale paired clinical datasets.

3. Key Contributions

Novel Architecture: Introduction of MFdiff, the first conditional diffusion model specifically designed for multi-modality PET restoration that effectively handles the structural mismatch between PET and MR.
Advanced Fusion Mechanism: Development of a dual-branch fusion module (Global/Detail) using Invertible Residual Blocks to extract features losslessly, ensuring MR guidance enhances PET without introducing anatomical hallucinations.
Robust Learning Strategy: Proposal of a two-stage supervise-assisted learning approach that combines simulated data (for generalization) and limited real data (for domain adaptation), effectively solving the OOD problem in medical imaging.
State-of-the-Art Performance: Demonstrated superior performance over existing methods (GANs, CNNs, and other diffusion models) in both quantitative metrics and qualitative visual fidelity.

4. Experimental Results

The method was evaluated on a simulated phantom dataset and three distinct Out-of-Distribution (OOD) clinical datasets involving variations in:

Imaging time (2-min vs. 30-min scans).
Radiotracer injection dose (Low-dose vs. Ultra-low-dose).
Administration protocols (Bolus, Infusion, Hybrid).

Quantitative Results:

Phantom Dataset: MFdiff achieved a PSNR of 37.93 dB and SSIM of 0.991, outperforming the second-best method (CSRD) by a significant margin.
OOD Datasets: MFdiff consistently outperformed all baselines (M-UNet, FB-SEM, EA-GAN, Hi-Net, CNCL, CSRD) across all three OOD scenarios. For example, in the 2-minute scan scenario, MFdiff improved PSNR by 0.92 dB over the next best method.

Qualitative Results:

Artifact Suppression: Unlike methods that simply concatenate inputs (which often blur lesions or hallucinate healthy tissue from MR into PET), MFdiff successfully preserved the shape and intensity of metabolic lesions while suppressing MR-induced artifacts.
Detail Preservation: The restored images exhibited sharper edges and richer texture details compared to GAN-based and CNN-based approaches.

Ablation Studies:

Confirmed that the Multi-modality Feature Fusion Module is superior to simple concatenation.
Showed that the Diffusion Module (CDDPM) significantly outperforms standard UNet architectures for this task.
Validated that the Two-Stage Learning Strategy yields better generalization than single-stage mixed training or training on simulation/internal data alone.

5. Significance

This research addresses a critical bottleneck in clinical PET imaging: the trade-off between radiation safety and image quality.

Clinical Impact: By enabling high-quality SPET restoration from low-dose inputs, MFdiff could allow for reduced radiation exposure for patients (especially in pediatric or longitudinal studies) without sacrificing diagnostic accuracy.
Generalizability: The two-stage learning strategy offers a practical blueprint for deploying deep learning in medical imaging where large, paired, multi-center datasets are unavailable. It demonstrates how to leverage simulated data for pre-training and fine-tune on small real-world datasets to handle domain shifts.
Technical Advancement: The work highlights the superiority of diffusion models over GANs in medical image restoration, particularly in terms of stability, mode coverage, and the ability to preserve fine-grained structural details without generating mode-collapse artifacts.