💻 computer science

Few-Shot Synthetic Data Generation with Diffusion Models for Downstream Vision Tasks

This paper proposes a lightweight pipeline that fine-tunes a LoRA adapter on a small set of real images to leverage pretrained diffusion models for generating synthetic data, demonstrating that this approach effectively improves rare-class recall and F1 scores in vision tasks across medical and industrial domains without requiring additional real-world data.

Original authors: Daniil Dushenev, Nazariy Karpov, Daniil Zinovjev, Alexander Gorin, Konstantin Kulikov

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daniil Dushenev, Nazariy Karpov, Daniil Zinovjev, Alexander Gorin, Konstantin Kulikov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to spot a very specific, rare problem—like finding a tiny crack in a factory tile or spotting a rare disease in a chest X-ray. The problem is, you only have a handful of examples (maybe 20 to 50) to show the robot. It's like trying to teach someone what a "golden retriever" looks like by showing them only two photos. The robot gets confused and misses the rare cases most of the time.

This paper proposes a clever trick to solve this problem using a type of AI called a Diffusion Model. Think of a diffusion model as a very talented artist who has seen millions of pictures and knows how to paint anything. However, this artist usually needs a huge library of reference photos to learn a new subject.

Here is how the authors' method works, broken down into simple steps:

1. The "Quick Study" Artist (LoRA Fine-Tuning)

Instead of retraining the whole artist (which takes forever and needs massive computers), the authors use a technique called LoRA.

The Analogy: Imagine the artist has a giant, blank sketchbook. Instead of rewriting the whole book, the authors give the artist a small, sticky-note pad (the LoRA adapter) with just 20–50 photos of the rare object (like a specific crack or disease).
The Result: The artist quickly learns the essence of that specific object from those few notes. They don't need thousands of photos; they just need a few to understand the "vibe" of the rare class.

2. The "Imagination Machine" (Synthetic Generation)

Once the artist has learned from those few photos, they are asked to draw hundreds of new pictures of that same rare object.

The Analogy: The artist isn't just copying the original photos; they are using their imagination to create new, slightly different versions. One drawing might show the crack at a different angle, another might have different lighting, but they all look like the real thing.
The Output: The computer generates about 1,000 of these "fake" but realistic images. These are called synthetic data.

3. The "Student" Learner (Downstream Training)

Now, the robot (the classifier) gets to study.

The Setup: The robot is shown the original 20–50 real photos plus the 1,000 new "fake" photos generated by the artist.
The Goal: The robot learns to recognize the rare object much better because it has seen many more variations of it.

What Did They Find?

The researchers tested this on two very different worlds:

Medical: Finding rare diseases in chest X-rays.
Industrial: Finding cracks on magnetic tiles in a factory.

The Results:

It Works: In both cases, giving the robot the "fake" photos made it much better at spotting the rare problems.
The "Goldilocks" Zone: There is a sweet spot.
- If you add a little bit of fake data (like 4 times as many fake photos as real ones), the robot gets significantly smarter.
- If you add too much fake data (like 20 times as many), the robot starts to get confused. The fake pictures start to overwhelm the real ones, and performance drops slightly. It's like eating too much candy; a little helps, but too much makes you sick.
Fake vs. Real: The robot still needs the real photos to learn best. If you train it only on the fake photos, it does okay, but not as well as when you mix real and fake. The fake photos are a great supplement, not a total replacement.

The Bottom Line

The paper shows that you don't need a massive database of rare examples to teach an AI. By using a "quick study" technique (LoRA) to teach a generative AI how to draw new examples from just a few real ones, you can create a training set that helps computers spot rare, dangerous, or expensive-to-find events much more effectively.

Key Takeaway: It's a lightweight, scalable way to turn a few real examples into a rich library of training data, helping AI learn from rare events in fields like medicine and manufacturing.

Technical Summary: Few-Shot Synthetic Data Generation with Diffusion Models for Downstream Vision Tasks

Problem Statement

Class imbalance remains a critical barrier in computer vision, particularly within safety-critical and industrial domains where positive examples of rare events (e.g., medical pathologies or manufacturing defects) are scarce, expensive to collect, and inherently underrepresented. Traditional data augmentation strategies, limited to geometric and photometric transformations, fail to introduce the fundamental visual variations necessary to improve model recall for these rare classes. While generative models like diffusion models offer a pathway to synthesize realistic, diverse data, their application has historically been constrained by the need for large datasets and substantial computational resources, rendering them impractical for few-shot scenarios.

Methodology

The authors propose a lightweight synthetic data augmentation pipeline designed to operate with as few as 20–50 real images of a rare class. The approach consists of three primary stages:

LoRA Fine-Tuning: A pretrained text-to-image diffusion model (FLUX.2-dev) is adapted using Low-Rank Adaptation (LoRA). The adaptation targets the DiT transformer backbone while keeping text encoders frozen. The process utilizes DreamBooth-style training with a fixed caption format ("a photo of ") on a small subset of real positive images (approx. 50). Training is conducted with rank $r=64$ , $\alpha=8$ , and 8-bit AdamW optimization, utilizing NF4 quantization to fit within a single A100 GPU.
Synthetic Data Generation: The fine-tuned adapter generates approximately 1,000 synthetic images of the target rare class at 512×512 resolution. Diversity is achieved solely through stochastic sampling with varying random seeds, without prompt augmentation or multiple caption templates.
Downstream Evaluation: A ResNet-18 classifier, pretrained on ImageNet and equipped with a binary head, is trained on datasets augmented with varying ratios of synthetic-to-real data (0.5× to 20×). The positive class weight is automatically adjusted to account for imbalance. Crucially, all evaluations are performed on held-out test sets consisting only of real images to ensure the metrics reflect genuine generalization.

Key Contributions

The paper presents the following contributions:

Lightweight Pipeline: A synthetic data augmentation framework based on LoRA-adapted diffusion models that functions effectively with extremely limited data (20–50 real images).
Systematic Scaling Analysis: A comprehensive investigation into the impact of synthetic dataset size by varying the synthetic-to-real ratio from 0.5× to 20×.
Cross-Domain Validation: Demonstration of consistent performance improvements in rare-class detection across two structurally distinct domains: medical chest X-ray pathology classification (NIH ChestX-ray14) and industrial surface crack detection (Magnetic Tile Defect dataset).
Scalability Evidence: Results indicating that diffusion-based synthetic augmentation serves as a scalable strategy for learning from rare events in data-scarce visual domains.

Experimental Results

The study evaluates the approach on two datasets with severe class imbalance:

Magnetic Tile Defect Dataset: The baseline model trained on real data alone achieved an F1-score of 0.051. Synthetic augmentation significantly improved performance, peaking at an F1-score of 0.296 with a 4× synthetic ratio (a nearly six-fold improvement). Recall increased substantially with higher ratios, reaching 0.658 at 20×.
NIH Chest X-ray Dataset: The real-only baseline achieved an F1-score of 0.193. Synthetic augmentation improved this to a peak of 0.686 at a 4× ratio.

Key Observations:

Diminishing Returns: Performance improves with moderate synthetic augmentation but shows diminishing returns or slight degradation as the synthetic ratio increases beyond optimal points (e.g., 10× or 20×), suggesting that excessive synthetic data may distort the training distribution.
Augmentation vs. Replacement: Training exclusively on synthetic data (synth-only) yielded lower overall performance than mixed real+synthetic training, confirming that synthetic data is most effective as an augmentation tool rather than a complete replacement for real data.
Diversity Analysis: Analysis using LPIPS and PSNR distributions indicated that the generated samples preserve the low-level structural properties of real data (similar PSNR) while introducing semantic variation (LPIPS shift), demonstrating that the model avoids mode collapse.

Significance and Claims

The paper claims that LoRA-adapted diffusion models provide a simple, scalable mechanism for augmenting rare classes, enabling effective learning in data-scarce scenarios across heterogeneous visual domains. The authors posit that this approach addresses the specific challenge of collecting positive examples in safety-critical fields by leveraging parameter-efficient tuning to generate useful synthetic data from extremely small datasets. The results suggest that this method offers a practical solution for improving rare-class recall and F1 scores without requiring large-scale labeled datasets or extensive computational resources for generative model training.