Technical Summary: Representation-Conditioned Diffusion Models for Guided Training Data Generation

Problem Statement

Data availability remains a critical bottleneck in deep learning, where large-scale datasets are often expensive to collect, curate, and annotate. This limits the scalability of supervised learning. While generative models, particularly diffusion models, offer a potential solution for synthetic data generation, existing approaches face significant challenges in the most demanding scenarios:

1. No external guidance: Only the target dataset is available, preventing the use of pre-trained text-to-image models that rely on text prompts.
2. Exclusive synthetic training: The downstream model is trained solely on generated data without access to real images.
3. No additional annotations: No text prompts or extra labels are available to guide generation beyond class labels.

In this scenario, the generative model must capture the full diversity of the real data distribution without the benefit of external semantic guidance, a task where conventional class-conditioned generation often struggles with sample quality and mode coverage.

Methodology

The authors propose a pipeline utilizing Representation-Conditioned Diffusion Models (RCDMs) to generate synthetic training data. The methodology consists of two primary stages:

1. Synthetic Data Generation (RCDM)

The generation process involves training a Latent Diffusion Model (LDM) conditioned on high-level visual representations rather than class labels or text prompts.

• Encoder Selection: Input images are projected into a lower-dimensional representation space using pre-trained vision encoders. The study evaluates three specific encoders:
  • DINOv2 (ViT-B/14): Produces 768-dimensional CLS token embeddings.
  • DINOv3 (ViT-B/16): Produces 768-dimensional CLS token embeddings.
  • CLIP (ViT-B/32): Produces 512-dimensional embeddings (referred to as unCLIP).
• Training: The LDM is trained on the ImageNet-100 dataset (130k images, 100 classes, 256×256 resolution) for 125 epochs with a batch size of 16.
• Sampling: Images are generated using a Denoising Diffusion Probabilistic Model (DDPM) with 100 inference steps. To ensure diversity beyond the original dataset size, distinct random seeds are used for latent noise initialization while maintaining semantic consistency through representation conditioning.

2. Evaluation

The quality of the synthetic data is assessed by training a ResNet-50 classifier exclusively on the generated datasets.

• Baseline Comparison: Performance is compared against a classifier trained on the real ImageNet-100 dataset and a class-conditioned LDM (Cond-LDM) baseline.
• Augmentation: Synthetic images are also used to augment real data, compared against classical augmentation techniques (AutoAug, RandAug, MixUp, ME-ADA).
• Filtering: A sample filtering strategy is tested where generated images are mapped back to the representation space, and extreme samples (nearest and farthest 25% from class centroids) are removed to retain "middle" samples.

Key Contributions

The paper presents the following contributions:

• Evaluation of RCDMs for Training: The authors evaluate the classification performance of models trained on synthetic datasets generated by RCDMs, investigating the impact of dataset size and the specific conditioning representation (DINOv2, DINOv3, CLIP).
• Scaling Synthetic Data: The study demonstrates that by scaling the size of the synthetic dataset, RCDMs can surpass the performance of models trained solely on real data.
• Augmentation and Filtering: The paper shows that RCDM-generated images are effective for data augmentation, outperforming classical techniques, and that the conditioning representation space enables simple, effective sample filtering strategies.

Results

Experiments were conducted on ImageNet-100, a challenging dataset due to its diversity and limited size.

1. Scaling Synthetic Datasets

• Performance vs. Real Data: When the synthetic dataset size was scaled to four times the size of the real dataset, classifiers trained on RCDM-generated data outperformed the baseline trained on real data.
  • DINOv2-conditioned: Achieved 79.26% Top-1 accuracy (vs. 77.26% real baseline).
  • DINOv3-conditioned: Achieved 78.8% Top-1 accuracy.
  • CLIP-conditioned: Achieved 76.3% Top-1 accuracy at 4x scale, showing a competitive trend.
• Comparison to Class-Conditioned: The class-conditioned baseline (Cond-LDM) achieved a maximum of 68.5% Top-1 accuracy, significantly underperforming the RCDM approaches. This highlights the necessity of rich semantic conditioning via pre-trained vision encoders to capture the underlying data distribution structure.

2. Representation Space Filtering

Filtering synthetic data by removing the 25% nearest and 25% farthest representations from class centroids resulted in slight improvements over unfiltered synthetic data (e.g., DINOv3 filtered reached 71.10% vs. 70.02% unfiltered at 1x scale), though a gap to the real-data baseline remained.

3. Data Augmentation

When used to augment real data (increasing dataset size to 2x or 3x):

• RCDM + Real Data: Adding synthetic data generated with DINOv2 representations (3x total size) achieved 82.22% Top-1 accuracy.
• Comparison: This outperformed all classical augmentation methods tested, including MixUp (81.58%), AutoAug (81.28%), and ME-ADA (80.14%).
• Cond-LDM: Augmentation with class-conditioned synthetic data (80.68%) performed worse than RCDM-based augmentation.

Significance and Claims

The authors claim that Representation-Conditioned Diffusion Models provide a promising approach for augmenting, complementing, or potentially replacing real-world datasets in large-scale visual learning tasks.

• Surpassing Real Data: The ability to outperform a classifier trained on real images by scaling a purely synthetic dataset is presented as "remarkable evidence" for the potential of generative deep learning in training data synthesis.
• Overcoming Limitations: Unlike text-to-image models that require prompts, RCDMs utilize self-supervised visual representations to guide generation, effectively reducing the domain gap between real and synthetic data.
• Future Directions: The paper suggests that future work could focus on improving sampling via the representation space (e.g., interpolation, hard example mining), refining outlier removal mechanisms, and verifying performance across different datasets and data availability scenarios.

The work positions RCDMs not just as a data augmentation tool, but as a viable engine for generating high-value training data in scenarios where real data is scarce, restricted (e.g., privacy concerns in medical imaging), or insufficient for scaling.