🤖 machine learning

Local Label-Informed Feature Transfer for Generating Ground-Truth Medical Images: A Comparison of GAN- and Diffusion-Based Approaches

This paper introduces Local Label-Informed Feature Transfer (LLIFT), a framework that generates realistic semi-synthetic brain MRI images with spatially controlled lesions using either GAN or diffusion-based models trained without pixel-level annotations, thereby creating reliable ground-truth datasets for validating Explainable AI methods in medical imaging.

Original authors: Rick Wilming, Irem Ozseker, Luca Matteo Cornils, Ahcène Boubekki, Benedict Clark, Danny Panknin, Stefan Haufe

Published 2026-07-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rick Wilming, Irem Ozseker, Luca Matteo Cornils, Ahcène Boubekki, Benedict Clark, Danny Panknin, Stefan Haufe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to spot a fake painting. You show it thousands of real masterpieces and thousands of forgeries, and you ask, "Which one is the fake?" To make sure the robot is actually looking at the paint and not just guessing based on the frame or the lighting, you need a special kind of test. You need a "ground truth"—a perfect map showing exactly where the fake brushstrokes are. In the world of medical imaging, this is a huge problem. Doctors and scientists want to know if their AI is looking at a tumor or just a weird shadow, but the "maps" of where tumors are often have mistakes, or they are made by humans who might miss something.

To solve this, scientists have tried making fake tumors by drawing simple shapes (like circles or squares) on top of healthy brain scans. But a brain isn't a canvas; a real tumor looks messy and organic, not like a perfect circle drawn by a computer. So, we are stuck: we don't have perfect maps to test our AI, and the fake maps we do have look too fake to be useful. This paper steps into that messy middle ground to see if we can build a better kind of "fake" brain scan—one that looks real enough to fool the eye, but has a secret, perfect map hidden inside that tells us exactly where the "fake" disease is.

The Magic Trick: LLIFT

The authors of this paper introduce a new framework called LLIFT (Local Label-Informed Feature Transfer). Think of it as a digital "magic wand" that can take a healthy brain scan and, with a little help from a user, insert a realistic-looking lesion (a spot of disease) exactly where you point your finger.

The coolest part? The AI doesn't need to be taught by a doctor who spends hours drawing the exact outline of a tumor. Instead, it learns from much simpler clues. The researchers tested two different "magic wands" to see which one works better:

The GAN Wand (LLIFT-GAN): Imagine a forger and a detective playing a game of cat and mouse. The forger tries to draw a fake tumor on a healthy brain, and the detective tries to spot it. The forger only knows if the whole picture looks "sick" or "healthy" (a simple yes/no label), not exactly where the sickness is. Over time, the forger gets so good at making the "sick" parts look real that the detective can't tell the difference. The result is a brain scan with a realistic tumor hidden inside a user-drawn box.
The Diffusion Wand (LLIFT-DM): This one is more like a master painter who has seen millions of pictures. This AI starts with a healthy brain and a "mask" (a box telling it where to paint). It then uses a powerful, pre-trained brain to "inpaint" or fill in that box with a realistic lesion. It's like asking a famous artist to add a storm cloud to a sunny landscape; they know exactly how clouds look, so they just fill in the box you gave them.

The Results: How Real is Real?

The team tested these wands on brain scans from the Human Connectome Project. They wanted to see two things:

Do the fake tumors look real? (Qualitative check)
Do the fake brains look statistically like real sick brains? (Quantitative check)

The Visuals:
When they looked at the pictures, both wands did a pretty good job. The tumors didn't look like perfect circles or geometric blobs. They looked organic, with the right brightness and texture, blending smoothly into the brain tissue. However, sometimes the GAN wand got a little wobbly, creating artifacts that looked a bit off, while the Diffusion wand sometimes created lesions that looked like "holes" in the brain rather than growths.

The Numbers:
To measure how close the fake brains were to real sick brains, the authors used a score called the Fréchet Inception Distance (FID). Think of this as a "distance meter."

The distance between two batches of healthy brains was 27.87.
The distance between healthy brains and real sick brains was 41.75 (for the GAN dataset) and 5.84 (for the Diffusion dataset).
The LLIFT-GAN fake brains had a score of 41.69 when compared to real sick brains. This is almost identical to the real sick brains! It suggests the GAN created a batch of images that is just as "different" from healthy brains as real disease is.
The LLIFT-DM blended images (where only the lesion was changed) had a score of 7.61 against real sick brains. This is very close to the real sick reference of 5.84, suggesting the diffusion model also created very realistic lesions.

The Catch and the Goal

The authors are careful not to call this a "cure" or a "perfect solution." They admit that the GAN training can be unstable and that the Diffusion model sometimes needs a little more tweaking. They also note that these are still "semi-synthetic" images—meaning the background is real, but the disease is generated.

However, the main victory here isn't just making pretty pictures. It's about trust. By creating these images, the researchers have built a perfect testing ground. Because they know exactly where they told the AI to put the lesion (the "ground truth"), they can now test other AI tools to see if those tools are actually looking at the disease or just guessing. It's like giving a student a math test where you know the answer key perfectly, so you can finally see if they are actually doing the math or just copying the answer.

In short, LLIFT suggests that we can generate realistic medical images with known "truths" without needing perfect, pixel-by-pixel maps from doctors. This could be a huge step forward in making sure the AI doctors of the future are actually looking at the right things.

Technical Summary: Local Label-Informed Feature Transfer (LLIFT)

Problem Statement

Validating Explainable Artificial Intelligence (XAI) methods in medical imaging requires ground-truth datasets where the precise locations of informative features (e.g., lesions) are known. Current validation approaches suffer from two primary limitations:

Expert Annotations: Relying on human-labeled datasets introduces labeling errors, inter-rater variability, and unquantifiable biases, potentially missing lesions or mislabeling healthy tissue.
Hand-Crafted Perturbations: Existing synthetic benchmarks often superimpose artificial geometric shapes onto healthy images. These lack clinical realism and fail to represent the complex visual signatures of actual pathology.

There is a need for a framework that generates semi-synthetic medical images with realistic lesions placed in user-controlled regions, without requiring pixel-level lesion annotations during the training phase.

Methodology: The LLIFT Framework

The authors propose Local Label-Informed Feature Transfer (LLIFT), a framework designed to generate semi-synthetic brain MRI images. The core principle is to learn the visual signature of pathology from weak supervision (class labels or coarse spatial masks) rather than pixel-level segmentations. The framework ensures that the location of the discriminative feature is known by construction while keeping the surrounding healthy anatomy identical to the input.

The paper implements and compares two distinct generative paradigms:

1. LLIFT-GAN (Generative Adversarial Network)

Supervision: Trains using only binary class labels (healthy vs. pathological). No bounding boxes or spatial masks are provided during training.
Architecture: Extends the DCGAN family with a dual-view discriminator and an encoder-decoder generator.
- Generator: Takes a healthy image, a random binary mask, and noise as input. The output is architecturally constrained to modify only the masked region, defined as $G_\theta(x_h, m, z) = (1-m) \odot x_h + m \odot \tilde{G}_\theta(x_h, m, z)$ . This eliminates the need for post-hoc blending.
- Discriminator: Uses a dual-view approach to overcome the limitations of single-scale critics. It processes the full image and a version where the bounding box is masked out. This forces the discriminator to learn that the unmasked context must remain healthy while the masked region must contain pathological features to be classified as "real."
Training: Optimized via a smoothed adversarial objective using Adam.

2. LLIFT-DM (Diffusion Model)

Supervision: Conditions on bounding-box masks rather than binary labels. Crucially, during training, lesion masks are shuffled across images. This decouples the conditioning signal (the mask location) from the actual lesion location, forcing the model to learn the visual signature of pathology rather than memorizing specific lesion positions.
Architecture: Fine-tunes a large-scale pretrained latent diffusion model (Stable Diffusion) using ControlNet.
- Conditioning: The ControlNet receives a mask where the lesion area is replaced by a constant placeholder, while the surrounding context is preserved. A fixed text prompt ("A brain MRI with a lesion") is used as a secondary signal.
- Training: Only the UNet and ControlNet components are fine-tuned; the VAE, CLIP encoder, and scheduler remain frozen.
Inference & Blending: Generates content for the masked region. Unlike the GAN, the diffusion process may introduce sub-pixel changes outside the mask. Therefore, a post-hoc blending step is applied: $\hat{x}_{blended} = (1-m) \odot x_h + m \odot \hat{x}_u$ , ensuring unmasked pixels remain identical to the healthy input.

Key Contributions

LLIFT Framework: A novel method for generating semi-synthetic brain MRIs with controllable lesion placement that does not require pixel-level annotations during training.
Dual Paradigm Comparison: The first comparative study of a GAN-based approach (LLIFT-GAN) and a diffusion-based approach (LLIFT-DM) specifically for generating XAI benchmark data.
Weak Supervision Strategy: Demonstrates that pathological features can be learned from binary class labels (GAN) or shuffled bounding boxes (Diffusion), relaxing the requirement for precise segmentation masks.
Benchmark Generation: Produces datasets where the ground-truth location of the informative feature is known by construction, enabling rigorous evaluation of feature attribution methods.

Results

The models were evaluated on data derived from the Human Connectome Project (HCP), using the Fréchet Inception Distance (FID) as the primary quantitative metric, alongside qualitative inspection.

Quantitative Performance (FID):
- LLIFT-GAN: Achieved an FID of 41.69 between generated and real pathological batches. This is comparable to the inter-class reference (Healthy vs. Real Pathological) of 41.75. The authors note that while the overall image FID is dominated by the unchanged healthy background, visual inspection confirms the generated lesion patches are structurally distinct and pathological.
- LLIFT-DM: The blended generated images achieved an FID of 7.61 against real pathological data, slightly above the inter-class reference of 5.84. The low FID against healthy data (4.78) confirms the effectiveness of the blending step in preserving the background.
Qualitative Performance:
- LLIFT-GAN: Generated lesions exhibited organic shape variations and blended smoothly with surrounding tissue, learning that lesions are brighter, irregular regions rather than geometric blobs.
- LLIFT-DM: Successful generations integrated seamlessly with background textures. Less successful cases occasionally produced artifacts resembling the absence of matter rather than lesions.

Significance and Claims

The paper claims that LLIFT provides a solution to the scarcity of ground-truth data for validating XAI methods. By generating datasets where the discriminative region is precisely defined by the input mask, the framework allows researchers to directly test the assumption that attribution methods highlight informative features.

The authors emphasize that:

The framework addresses the reliability of feature attribution assessments by removing the ambiguity of expert annotations and the artificiality of geometric perturbations.
Both LLIFT-GAN and LLIFT-DM successfully transfer pathological content into user-specified regions without ever seeing pixel-level lesion annotations.
The resulting benchmark datasets enable standardized, reproducible testing of whether attribution methods correctly identify pathology-relevant regions.

The authors remain modest regarding clinical deployment, noting that while the structural fidelity is high, expert reader studies would be essential if the approach were extended to clinical data augmentation. They also identify limitations, such as the instability of GAN training and the need for future metrics that operate at the scale of the edited region rather than the whole image.