🤖 AI

Post-Generation Curation of Synthetic Images via Homogeneous-Heterogeneous Splitting

This paper introduces a generator-agnostic post-generation curation method that improves the utility of synthetic images by splitting real classes into canonical and non-redundant subsets to select a diverse, high-fidelity subset that counteracts the structural bias of modern generators, achieving performance comparable to real data with up to 40% fewer samples.

Original authors: Disheng Liu, Tuo Liang, Chaoda Song, Yu Yin

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Disheng Liu, Tuo Liang, Chaoda Song, Yu Yin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a student (an AI model) how to recognize different animals. You have a textbook full of real photos (Real Data), but you also have a massive library of photos created by a robot artist (Synthetic Data).

The robot artist is getting very good at drawing. However, it has a bad habit: it loves to draw the "perfect" version of a horse over and over again. It draws the same pose, the same lighting, and the same angle thousands of times. It rarely draws the weird, awkward, or unique horses that exist in real life.

If you just throw all the robot's drawings at the student, the student gets bored and confused. They learn to recognize only the "perfect" horse and fail when they see a real horse standing sideways or in the rain.

The Problem:
Current methods to fix this usually involve either:

Retraining the Robot: Teaching the robot to draw better (which is expensive and slow).
Giving the Robot Better Prompts: Telling the robot, "Draw a horse, but make it weird!" (which requires an expert and doesn't always work).

The Paper's Solution: The "Smart Sorter"
This paper proposes a third way: Don't fix the robot; just be a smarter librarian.

Instead of using all the robot's drawings, the authors created a system to pick only the best ones to give to the student. They call this "Post-Generation Curation."

Here is how their "Smart Sorter" works, using a simple analogy:

1. Splitting the Real World into "The Standard" and "The Weirdos"

First, the researchers look at the real photos (the textbook). They split every category (like "Horses") into two piles:

The HO Pile (Homogeneous): These are the "Standard" photos. They are the most common, typical, and similar to each other. Think of these as the "Cover Art" of the class.
The HE Pile (Heterogeneous): These are the "Weirdos." They are the unique, diverse, and slightly unusual photos that don't look exactly like the others. They represent the variety in the real world.

Why do this? The robot artist naturally loves to copy the "Standard" photos (HO) because they are easy to recognize. It ignores the "Weirdos" (HE). If you only give the student the robot's "Standard" copies, they miss out on the "Weirdos."

2. The Scoring System: "Fidelity" vs. "Diversity"

Now, the researchers take the robot's pile of drawings and score them based on two rules:

Fidelity (Is it real?): Does this drawing look like a real horse? (We want this to be high).
Diversity (Is it unique?): Does this drawing look like one of the "Weirdos" from the real world, or is it just another copy of the "Standard" horse? (We want this to be high too).

The system gives a high score to a drawing that looks like a real horse AND adds something new to the collection (like a horse with a unique pose). It gives a low score to a drawing that is just a carbon copy of the "Standard" horse, even if it looks perfect.

3. The Result

By using this scoring system, the researchers selected a smaller, smarter group of robot drawings.

The Magic: They found that they could train the student AI using 40% fewer robot drawings and still get the same (or better) results as if they had used the whole pile.
The Benefit: The student learned to recognize horses in all their forms, not just the "perfect" ones. This made the student better at recognizing horses in tricky situations (like bad lighting or strange angles), a skill called "Out-of-Distribution" robustness.

Key Takeaways

It's Generator-Agnostic: You don't need to know how the robot artist works or retrain it. You just take its output and sort it.
It's a Complement, Not a Replacement: This doesn't mean we don't need better robots. It means that even with a good robot, we need a smart librarian to pick the right books.
The Trade-off: As the robots get better at drawing perfect images, the most important thing becomes finding the unique images. The system automatically balances this: if the robot is great, the system focuses more on finding the "Weirdos" to fill the gaps.

In short, the paper teaches us that quality isn't just about how good the fake images look; it's about how well they cover the full range of real-life variety. By sorting out the duplicates and keeping the unique ones, we can train smarter AI with less data.

Technical Summary: Post-Generation Curation of Synthetic Images via Homogeneous-Heterogeneous Splitting

1. Problem Statement

Recent generative models (GMs) offer a scalable solution for data scarcity by producing high-quality synthetic images. However, simply increasing the volume of synthetic data does not guarantee improved downstream performance due to three primary challenges:

Noise: Synthetic datasets often contain low-fidelity or mislabeled samples.
Bias: Generators tend to over-represent canonical modes of a class while under-representing intra-class variation, leading to inherent biases in downstream models.
Efficiency: Scaling data to mitigate these issues incurs significant computational overhead and longer training times.

Existing approaches to improve synthetic data utility typically involve generator-side interventions, such as fine-tuning generators or using lightweight strategies like prompt engineering and inference-time guidance. These methods are often generator-specific, require significant domain expertise, or incur substantial computational costs for retraining.

This paper addresses a complementary question: Given a fixed pool of generated images, can downstream utility be improved purely by selecting an informative subset? The authors argue that effective selection must counter the structural bias of modern generators by balancing semantic fidelity with representational diversity.

2. Methodology

The proposed framework is a generator-agnostic, post-generation curation pipeline that does not require retraining the generative model. It consists of two main stages:

A. Homogeneous-Heterogeneous (HO-HE) Splitting of Real Data

Instead of treating the real reference dataset as a monolithic distribution, the method partitions each real class into two distinct subsets based on a directed 1-nearest-neighbor (1-NN) graph in feature space:

Homogeneous (HO) Set: Contains "canonical" samples that serve as local representatives for other samples in the class. Formally, $I_{HO}$ is the set of nodes with positive in-degree in the 1-NN graph. These samples exhibit high intra-class similarity and represent the core semantics.
Heterogeneous (HE) Set: Contains samples with zero in-degree in the 1-NN graph. These represent non-redundant variations and less typical instances that are not needed as nearest neighbors for other examples.

Key Insight: Generators tend to over-produce samples resembling the HO set while neglecting the HE set. Furthermore, discriminators often struggle more with HE samples, indicating they contain harder, more diverse variations.

B. Partition-Conditioned Fidelity-Diversity Scoring

Once the real data is split, synthetic images are scored and selected based on a dual-criterion mechanism that references both the HO and HE partitions separately:

Fidelity Score ( $S^p_{fid}$ ): Measures the semantic alignment between a synthetic sample and the real partition $p$ (where $p \in \{HO, HE\}$ ) using cosine similarity between feature vectors.
Diversity Score ( $S^p_{div}$ ): Encourages deviation from canonical patterns.
- For the HO partition, the reference anchor is the class centroid.
- For the HE partition, the anchor is the nearest matching feature in the HO set.
- The diversity score is calculated as the negative cosine similarity between the vector from the real instance to its anchor and the vector from the real instance to the synthetic sample. This rewards synthetic samples that diverge from the dominant canonical direction.

Total Score: The final selection score is a weighted sum:
$S^p = \alpha \cdot S^p_{div} + (1 - \alpha) \cdot S^p_{fid}$
Where $\alpha$ is a hyperparameter controlling the trade-off between diversity and fidelity. The method selects the top- $k$ samples from each partition to form the final curated pool.

3. Key Contributions

HO-HE Partitioning: A nearest-neighbor-cover partition of real data with a minimality guarantee (Proposition 1), providing an explicit definition of canonical versus non-redundant samples without requiring global axis assumptions (unlike k-means).
Principled Selection Strategy: A post-generation framework that jointly quantifies and balances fidelity and diversity. It is generator-agnostic, requiring only a synthetic data pool and avoiding costly retraining or fine-tuning of generators.
Comprehensive Validation: Extensive experiments across classification and segmentation benchmarks (CIFAR-10, ImageNet, SVHN, Tiny-ImageNet) and multiple architectures (ResNet, ViT, EfficientNet) and generators (EDM, EDM2, StyleGAN).

4. Experimental Results

The method was evaluated under two settings: general selection from a fixed pool and "plug-in" selection after generator-side interventions.

In-Domain Performance: The method consistently outperformed state-of-the-art baselines (including Random Selection, CLIP-Align, RealScore, and SBSim).
- On SVHN, the method matched real-data baselines with ~100k synthetic samples, whereas baselines required >200k.
- On CIFAR-10, it achieved 95% accuracy with 300k samples, while baselines required ~500k.
- On ImageNet-1K, it achieved the best Top-1 accuracy across both training-from-scratch and fine-tuning settings using ViT-B/16 and ResNet-50.
Out-of-Distribution (OOD) Robustness: Models trained on curated data showed superior robustness across various OOD benchmarks (e.g., ImageNet-C, ImageNet-Sketch, CIFAR-10-Warehouse) compared to models trained on uncurated or randomly selected synthetic data.
Plug-in Efficacy: When applied as a post-processing step after task-specific generator interventions (e.g., inference-time intervention or fine-tuned JoDiffusion), the curation method provided additional gains.
- For classification on ImageNet-100, it achieved comparable or better accuracy with 40–50% fewer synthetic samples compared to the intervention-only baseline.
- For segmentation on VOCaug, it improved mIoU over the fine-tuned generator baseline.
Trade-off Analysis: The optimal $\alpha$ (diversity weight) shifts based on generator quality and data volume. As generator fidelity increases (e.g., EDM2) and training budgets grow, the optimal strategy shifts toward prioritizing diversity ( $\alpha \to 1$ ), as canonical high-fidelity samples provide diminishing returns.

5. Significance and Claims

The paper posits that post-generation selection is not a substitute for better generators but a complementary mechanism to improve the utility of synthetic data.

Addressing Generator Bias: The study highlights that modern generators have a structural bias toward canonical modes (HO). The proposed HO-HE splitting explicitly addresses this by ensuring the curated dataset includes non-redundant variations (HE) that generators naturally under-produce.
Efficiency: The method offers a practical, scalable alternative to generation-time interventions. The computational overhead of the curation process (feature extraction and scoring) is negligible compared to the savings in downstream training time and data volume required to achieve equivalent performance.
Generalizability: The framework is effective across different generative models, feature extractors (SigLIP, DINOv3, ViT), and downstream tasks (classification, segmentation), suggesting that balancing fidelity and diversity is a fundamental principle for leveraging synthetic data effectively.