⚡ electrical engineering

LesionDiffusion: Towards Text-controlled General Lesion Synthesis

The paper proposes LesionDiffusion, a text-controllable framework for 3D CT imaging that synthesizes lesions and their corresponding masks using a structured report template, thereby addressing data scarcity and improving segmentation performance across diverse lesion types and organs.

Original authors: Wenhui Lei, Henrui Tian, Linrui Dai, Hanyu Chen, Xiaofan Zhang

Published 2026-02-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wenhui Lei, Henrui Tian, Linrui Dai, Hanyu Chen, Xiaofan Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef trying to teach a robot how to cook a perfect steak. The problem? You only have a few real steaks to show the robot, and they are expensive, rare, and you can't just go buy more because of privacy rules (you can't just show the robot real patients' medical scans).

If you try to teach the robot with just those few real steaks, it might learn to cook them perfectly but fail miserably when asked to cook a chicken or a fish.

This is the exact problem doctors and AI researchers face with medical imaging. They need huge amounts of data (like CT scans of tumors) to train AI to spot diseases, but real data is hard to get.

Enter LesionDiffusion, the new "AI Chef" from the paper. Here is how it works, explained simply:

1. The Problem: The "One-Size-Fits-None" Approach

Previous AI models were like robots that only knew how to cook one specific type of steak. If you wanted a chicken, they couldn't do it. They also couldn't listen to specific instructions like, "Make the steak rare," or "Make the crust extra crispy." They just guessed.

2. The Solution: A Text-Controlled "Magic Cookbook"

The researchers created LesionDiffusion, which is like a super-smart robot chef that listens to a text recipe to create fake medical images.

Instead of just showing the robot a picture, you can tell it exactly what you want using a structured report (like a fill-in-the-blank form).

You say: "I need a tumor in the liver. It should be round, about the size of a grape, and look like a cyst."
The AI says: "Got it!" and generates a brand new, realistic 3D CT scan of that exact liver with that exact tumor, along with a "mask" (a digital stencil showing exactly where the tumor is).

3. How It Works: The Two-Step Dance

The system works in two main stages, like building a house:

Step 1: Drawing the Blueprint (LMNet)
First, the AI draws the shape and location of the tumor. It looks at your text instructions ("Round," "Inside the liver") and draws a digital outline. It's like an architect drawing the floor plan based on your description.
Step 2: Painting the Walls (LINet)
Once the blueprint is ready, the AI fills in the details. It takes a healthy liver scan and "paints over" the area where the tumor should be, making it look exactly like the texture and density you described ("Heterogeneous," "Hypodense"). It's like a painter taking a photo of a healthy wall and digitally adding a realistic-looking crack or stain exactly where the blueprint said it should go.

4. The Secret Sauce: The "Structured Report"

What makes this special is the Structured Report.
Think of previous AI models as a child who just says, "Draw a monster." The result is random.
LesionDiffusion uses a checklist with 10 specific categories:

Shape: Round? Irregular?
Location: Inside the organ or on the edge?
Texture: Smooth or spiky?
Size: 8mm or 32mm?

By forcing the AI to fill out this checklist, the researchers can control the output with incredible precision. They can even ask for "weird" tumors they've never seen before, and the AI can guess how to draw them based on the text description.

5. Why Does This Matter? (The "Training Gym")

The real magic isn't just making fake pictures; it's using them to train better doctors (or AI doctors).

The Gym Analogy: Imagine a boxer training for a fight. If they only train against one specific opponent, they will lose when they face a new style.
The Result: The researchers used LesionDiffusion to generate thousands of "fake" tumors of all different shapes and sizes. They used these to train a new AI to spot real tumors.
The Win: This new AI became a champion. It performed better than models trained only on real data, and it was so good at generalizing that it could even spot brain hemorrhages (which it had never seen in its training data) just because it learned the concept of a lesion from the text descriptions.

Summary

LesionDiffusion is a tool that turns text descriptions into realistic 3D medical images.

Old way: "Here is a picture of a tumor. Learn from it." (Hard to get enough pictures).
New way: "Write a description of a tumor, and I will generate 1,000 unique examples for you to learn from."

It solves the shortage of medical data, allows for precise control over what is generated, and helps train AI to be much better at saving lives, even for diseases it has never seen before.

1. Problem Statement

Medical imaging analysis, particularly lesion detection and diagnosis, relies heavily on fully supervised deep learning methods. These methods face two critical bottlenecks:

Data Scarcity & Privacy: Collecting large-scale, annotated datasets is expensive, time-consuming, and hindered by patient privacy concerns.
Limitations of Existing Synthesis: Current synthetic lesion generation methods (e.g., GANs, standard Diffusion models) suffer from:
- Poor Scalability: They often train models independently for each lesion type, limiting generalizability.
- Lack of Control: They offer insufficient fine-grained control over lesion attributes (shape, density, texture).
- Structural Simplicity: They primarily simulate solid lesions, failing to generate complex structures like hollow lesions found in colorectal or gastric cancers.

The authors aim to develop a general, text-controllable framework capable of synthesizing 3D CT lesions and their corresponding masks across diverse organs and lesion types, addressing the need for scalable, high-quality data augmentation.

2. Methodology: LesionDiffusion

The proposed framework, LesionDiffusion, is a two-stage pipeline designed to generate both the lesion mask and the corresponding 3D CT image content based on structured text reports.

A. Dataset Construction

The authors curated a new dataset of 1,505 annotated CT scans covering 14 lesion types across 8 organs (e.g., lung, liver, kidney, stomach, colon).

Annotation: Each scan includes a lesion mask and a structured lesion report.
Structured Report: This is a key innovation. It contains 10 specific attributes categorized into:
- Mask Attributes: Shape, location, size, organ type (hollow vs. parenchymal).
- Image Attributes: Density, homogeneity, surface characteristics, enhancement status.

B. Stage I: Lesion Mask Synthesis (LMNet)

Goal: Generate a precise binary lesion mask guided by text attributes.
Mechanism: A diffusion model (LMNet) is trained to denoise a mask from Gaussian noise.
Conditioning: The model is guided by:
- Bounding Box (B): Derived from text attributes (size, location).
- Organ Semantics (M): The target organ context.
- Mask Attributes ( $A_{mask}$ ): Encoded via a pretrained text encoder (BiomedCLIP) and fused via cross-attention.
Loss Function: A bbox-weighted loss is introduced to force the model to focus on the constrained area defined by the bounding box, ensuring the generated mask adheres to the specified spatial constraints.

C. Stage II: Lesion Image Inpainting (LINet)

Goal: Fill the masked region in the 3D CT image with realistic lesion textures based on image attributes.
Mechanism: To manage the high memory cost of 3D CTs, the authors use Latent Diffusion.
- Autoencoder: A VQ-GAN (trained with an adversarial module) compresses 3D CTs into a latent space and reconstructs them.
- Diffusion Process: The diffusion process occurs only within the lesion region (masked by $m$ ) in the latent space.
Conditioning: Guided by Image Attributes ( $A_{image}$ ) (e.g., "Hypodense," "Heterogeneous") encoded by BiomedCLIP.
Training: Uses an L1 loss within the bounding box to minimize the difference between predicted and actual noise, ensuring high-fidelity texture generation.

D. Inference Pipeline

Attribute Generation: The user provides a structured report (either manually selected or generated by an LLM).
Bounding Box Sampling: Based on text attributes (e.g., "inside hollow organ"), a 3D bounding box is sampled.
Mask Generation: LMNet generates the lesion mask using DDIM sampling.
Image Inpainting: The mask is used to mask the input CT in latent space; LINet performs reverse diffusion to inpaint the lesion, which is then decoded back to image space.

3. Key Contributions

Generalizable Framework: Unlike previous models trained per-lesion-type, LesionDiffusion is a unified model capable of synthesizing 14 different lesion types across 8 organs.
Text-Controlled Granularity: The use of structured reports allows for precise control over both geometric properties (shape, size, location) and radiological properties (density, texture), enabling the generation of complex hollow lesions.
New Dataset: Introduction of a large-scale, multi-organ dataset with paired masks and structured reports, filling a gap in available medical data.
Dual-Stage Architecture: A novel separation of mask generation (LMNet) and image inpainting (LINet), optimizing the specific challenges of 3D medical image synthesis.

4. Experimental Results

The model was evaluated on downstream lesion segmentation tasks using nnUNet-v2.

Segmentation Performance:
- Augmentation: Models pre-trained on LesionDiffusion-synthesized data outperformed those trained on DiffTumor (a state-of-the-art baseline) by an average of 6.9% DSC (Dice Similarity Coefficient).
- Fine-tuning: After fine-tuning with real data, LesionDiffusion-augmented models achieved a 2.4% higher DSC than models trained only on real data, and 3.5% higher than DiffTumor-augmented models.
Generalization (Zero-Shot):
- The model was tested on hematencephalon (brain) lesions, a domain not present in the training set.
- LesionDiffusion maintained strong performance, whereas DiffTumor suffered a 46% drop in DSC due to the domain gap.
- After fine-tuning, the LesionDiffusion model achieved 15% better performance than models trained solely on real brain data.
Alignment with Text:
- Quantitative metrics (convexity and sphericity) confirmed that "round-like" prompts generated more spherical masks than "irregular" prompts.
- Qualitative results showed accurate control over density and texture variations.
Ablation Study: Removing text conditions caused a 15.4% average drop in DSC, proving the critical role of explicit textual guidance, especially for organs with multiple lesion types (e.g., liver, kidney).

5. Significance

Scalable Data Augmentation: LesionDiffusion offers a scalable solution to the data scarcity problem in medical AI, reducing the reliance on expensive manual annotation.
Controllable Synthesis: It bridges the gap between generative AI and clinical requirements by allowing radiologists to specify exact lesion characteristics, making the synthetic data more relevant for training robust diagnostic models.
Foundation for General Models: The work lays the groundwork for a "controllable lesion generation foundation model," capable of adapting to new diseases and organs with minimal retraining, provided the structured attribute schema is maintained.

In summary, LesionDiffusion represents a significant leap forward in medical image synthesis, moving from rigid, single-task generators to flexible, text-driven systems that can generate high-fidelity, diverse, and controllable 3D lesion data for improving downstream diagnostic AI.