⚡ electrical engineering

Leveraging Image Editing Foundation Models for Data-Efficient CT Metal Artifact Reduction

This paper proposes a data-efficient CT metal artifact reduction method that reframes the task as an in-context reasoning problem by adapting a vision-language diffusion foundation model with Low-Rank Adaptation (LoRA) and multi-reference conditioning, achieving state-of-the-art results with only 16 to 128 training examples while mitigating hallucinations through domain adaptation.

Original authors: Ahmet Rasim Emirdagi, Süleyman Aslan, Mısra Yavuz, Görkay Aydemir, Yunus Bilge Kurt, Nasrin Rahimi, Burak Can Biner, M. Akın Yılmaz

Published 2026-04-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ahmet Rasim Emirdagi, Süleyman Aslan, Mısra Yavuz, Görkay Aydemir, Yunus Bilge Kurt, Nasrin Rahimi, Burak Can Biner, M. Akın Yılmaz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a beautiful, detailed photograph of a city skyline. Now, imagine someone has spilled a bucket of thick, black ink over the center of the photo. The ink isn't just a blob; it's shooting out long, jagged streaks that ruin the view of the buildings behind it.

In the medical world, this is exactly what happens in a CT scan when a patient has metal implants (like a hip replacement or dental fillings). The metal is so dense that it blocks X-rays, creating "streak artifacts" that look like lightning bolts or spiderwebs. These streaks hide the important organs and tissues doctors need to see, making the scan useless for diagnosis.

For years, fixing this was like trying to clean a muddy window by scrubbing harder and harder. It required massive amounts of data (thousands of clean vs. dirty photos) and complex math that often just made the image blurry or created new, fake shapes.

This paper introduces a clever new way to solve the problem using AI "Foundation Models"—the same kind of super-smart AI that can edit photos, write stories, or generate art. Here is how they did it, explained simply:

1. The Problem: The AI Got Confused

The researchers tried using a powerful AI model (trained on millions of internet photos) to fix the CT scans without teaching it anything new first.

The Result: The AI got totally confused. Because the metal streaks looked a bit like the patterns on a waffle or the grid of a petri dish, the AI tried to "fix" the image by drawing a waffle or a petri dish right over the patient's liver!
The Lesson: A general AI knows what a waffle looks like, but it doesn't know what a human liver looks like when it's covered in metal streaks. It needs a little guidance.

2. The Solution: The "Smart Tutor" (LoRA)

Instead of retraining the entire giant AI brain (which would take forever and require a massive library of data), the researchers used a technique called LoRA (Low-Rank Adaptation).

The Analogy: Imagine the AI is a brilliant chef who knows how to cook every dish in the world. But this chef has never cooked a specific regional dish (CT scans with metal).
The Fix: Instead of sending the chef back to culinary school for 4 years, you just give them a small, specialized recipe card (the LoRA adapter). This card teaches the chef the specific rules for this one dish.
The Magic: With this tiny recipe card, the chef can fix the CT scans using only 16 to 128 examples. That is like learning a whole new language by reading just a few pages of a book, instead of a whole dictionary.

3. The Secret Weapon: "Reference Photos"

Even with the recipe card, the AI still needed help figuring out what the hidden organs should look like.

The Strategy: The researchers gave the AI a "cheat sheet." Alongside the damaged scan, they showed it five clean photos of other people's healthy bodies (but of the same body part).
The Analogy: It's like asking a painter to restore a damaged portrait. You don't just say, "Fix this." You say, "Here is the damaged face, and here are five photos of healthy faces of the same age and gender. Use those to guess what the missing nose and eyes should look like."
The Result: The AI used these "reference photos" to infer the missing anatomy, filling in the gaps with realistic tissue instead of hallucinating waffles.

4. The Outcome: A New Era of Efficiency

The results were impressive:

Data Efficiency: They achieved top-tier results with 100 times less data than previous methods.
Quality: The cleaned-up images were so clear that doctors could see the organs perfectly, and the "waffle" hallucinations disappeared completely.
Realism: While older methods made the image look mathematically "correct" but blurry (like a smoothed-out painting), this method made the image look realistic and sharp, preserving the fine details of the anatomy.

Why This Matters

Think of this as a shift from building a custom factory for every single problem to hiring a versatile expert who can learn any new task with a quick tutorial.

Previously, hospitals needed massive supercomputers and thousands of labeled scans to fix metal artifacts. Now, this method shows that with a smart foundation model and a tiny bit of guidance, we can fix these images quickly, cheaply, and with high accuracy. It turns a difficult, data-hungry medical problem into a manageable task that could be deployed in hospitals anywhere in the world.

1. Problem Statement

Metal Artifact Reduction (MAR) in Computed Tomography (CT) is a critical challenge. High-density implants (e.g., dental fillings, orthopedic hardware) cause severe image degradations such as streaking and shadowing due to beam hardening and photon starvation. These artifacts obscure adjacent soft tissues, rendering them non-diagnostic.

Current Limitations:

Data Dependency: State-of-the-art deep learning methods typically require tens of thousands of paired training examples (corrupted/clean pairs), which are difficult to acquire and annotate.
Domain Constraints: Dual-domain methods (using raw sinogram data) are often impractical because hospitals rarely store raw projection data, only reconstructed volumes.
Hallucinations: Generative models (like GANs or Diffusion models) trained from scratch often suffer from anatomical hallucinations (inventing structures that don't exist) or training instability.
Generalization: Existing MAR networks are purpose-built and lack the rich structural priors (lighting, texture, geometry) learned by large-scale foundation models.

2. Methodology

The authors propose a paradigm shift: reframing MAR as an in-context reasoning task using a general-purpose Vision-Language Diffusion Foundation Model adapted via Parameter-Efficient Fine-Tuning (PEFT).

Core Architecture

Base Model: The authors utilize Qwen-Image-Edit, a 20-billion parameter vision-language diffusion transformer (DiT) pre-trained on internet-scale natural images.
Adaptation Strategy: Instead of full fine-tuning, they employ Low-Rank Adaptation (LoRA). This freezes the massive pre-trained backbone and injects trainable low-rank matrices ( $\Delta W = BA$ ) into specific layers (attention projections and MLPs). This reduces trainable parameters by orders of magnitude, allowing adaptation with minimal data.

Input Formulation

The restoration task is defined as a conditional generation problem:
$\hat{I}_{trg} = F_\theta(I_{src}, R, T)$

$I_{src}$ : The metal-corrupted CT slice.
$R$ : A set of $K$ clean reference slices from unrelated patients (serving as anatomical exemplars).
$T$ : A textual instruction (e.g., "Remove metal artifacts").

Conditioning Strategies

The paper investigates two ways to integrate reference slices:

Multi-Image Conditioning (Proposed): The model natively accepts a sequence of images. The corrupted slice and 5 clean references are passed as a single list. The model uses cross-image attention to draw structural cues directly from the references at full resolution.
Grid-Concatenation: A baseline where images are tiled into a grid and passed as a single composite image. This was found to be inferior due to resolution loss and spatial ambiguity.

Training & Inference

Data Efficiency: The model is trained on as few as 16 to 128 paired examples.
Loss Function: Flow-matching supervised fine-tuning (FlowMatchSFT).
Self-Ensembling: To reduce stochastic variance inherent in diffusion models, the inference is run $M=10$ times with different seeds, and the outputs are averaged.

3. Key Contributions

First Application of Image Editing Foundation Models to CT MAR: Demonstrates that general-purpose vision-language models, when adapted via LoRA, can effectively transfer rich visual priors to the medical domain without domain-specific pretraining.
Extreme Data Efficiency: Achieves state-of-the-art perceptual performance using only 16–128 training pairs, reducing data requirements by two orders of magnitude compared to traditional supervised methods.
Reference-Conditioned Restoration: Introduces a strategy using clean anatomical exemplars from other patients to guide the restoration, enabling the model to infer uncorrupted anatomy without access to raw sinogram data.
Mitigation of Hallucinations: Proves that domain adaptation (LoRA) is critical; without it, foundation models interpret metal streaks as natural objects (e.g., waffles or petri dishes), leading to dangerous diagnostic errors.

4. Experimental Results

The method was evaluated on the AAPM CT-MAR Grand Challenge benchmark (1,000 test slices).

Quantitative Performance:
- Perceptual Metrics: The proposed method (LoRA + Multi-Ref + Ensembling) achieved State-of-the-Art (SOTA) on perceptual metrics:
  - FID-Rad: 0.0037 (vs. 0.0241 for OSCNet+, a ~6x improvement).
  - LPIPS-Rad: 0.0391 (vs. 0.0931 for OSCNet+, a ~2.5x improvement).
- Pixel Metrics: While OSCNet+ led in PSNR (37.74 dB) due to its sinogram-consistency prior, the proposed method achieved a competitive 35.51 dB (with ensembling) and the highest SSIM (0.8966) among trained models.
- Data Efficiency: Even with only 16 training pairs, the model outperformed baselines trained on massive datasets on radiological feature metrics (LPIPS-Rad).
Qualitative Findings:
- Zero-Shot Failure: Without LoRA adaptation, the base model hallucinated natural textures (waffles, petri dishes) into the CT scans, confirming the necessity of domain alignment.
- Artifact Suppression: The method successfully eliminated streak artifacts while preserving soft-tissue boundaries, whereas baselines left residual streaks or over-smoothed tissues.
- Reference Importance: Multi-image conditioning significantly outperformed grid-concatenation, proving that maintaining full resolution and explicit token labeling is crucial for leveraging reference anatomy.

5. Significance and Conclusion

This work establishes that foundation models offer a scalable, interpretable, and data-efficient alternative for medical image reconstruction. By leveraging the "reasoning" capabilities of vision-language models and grounding them with clean anatomical references, the authors overcome the data scarcity bottleneck that plagues traditional deep learning MAR.

Implications:

Clinical Viability: The approach works with standard PACS data (reconstructed images only), bypassing the need for raw sinogram access.
Deployment: The compact LoRA adapters allow for deploying specialized models for different scanner protocols or implant types without retraining massive networks.
Future Direction: While currently limited to synthetic benchmarks, the framework suggests a path toward low-data, high-fidelity medical image restoration that prioritizes perceptual and diagnostic realism over simple pixel-wise regression.

Limitations: The evaluation is currently restricted to synthetic benchmarks (sim-to-real gap remains), and the self-ensembling technique increases inference latency, which may be a constraint in time-sensitive clinical workflows.