Label-Consistent Dataset Distillation with Detector-Guided Refinement

This paper proposes a detector-guided dataset distillation framework that leverages a pre-trained detector to identify and refine anomalous synthetic samples generated by diffusion models, thereby ensuring label consistency and enhancing image quality to achieve state-of-the-art performance.

The Gist

Imagine you are a chef trying to teach a new apprentice how to cook a massive, complex banquet. You have a library of 1,000,000 recipes and ingredients (the Original Dataset). But the apprentice's kitchen is tiny, and they can only hold a few ingredients at a time.

Dataset Distillation is the art of shrinking that massive library down to a tiny, perfect "survival kit" of just 10 or 50 recipes that still teaches the apprentice everything they need to know.

However, there's a problem. When previous chefs tried to shrink these libraries using AI (specifically Diffusion Models, which are like AI artists that paint pictures from scratch), they often made mistakes. They might paint a picture of a "dog" that looks like a blurry blob, or accidentally paint a "cat" but label it "dog." If you teach your apprentice with these bad examples, they will fail the final exam.

This paper introduces a new method called "Detector-Guided Refinement" to fix this. Here is how it works, using simple analogies:

1. The Problem: The "Blurry Photo" Factory

Imagine you have a robot artist (the Diffusion Model) tasked with painting 10 pictures of a "vacuum cleaner" to teach the apprentice.

• The Old Way: The robot paints 10 pictures. Some look great, but 2 of them are weird—they look like a pile of dust, or they are labeled "chair" by mistake. The robot didn't realize its own mistakes.
• The Result: The apprentice studies these bad pictures and gets confused.

2. The Solution: The "Strict Art Critic"

The authors added a second robot, a Detector, which acts like a strict Art Critic. This critic has studied the original 1,000,000 pictures and knows exactly what a real "vacuum cleaner" looks like.

Here is the new process, step-by-step:

Step A: The First Draft (Prototype-Guided Synthesis)

The robot artist starts by looking at a "blueprint" (a prototype) of a vacuum cleaner and paints a batch of new images.

Step B: The Inspection (Anomaly Detection)

The Art Critic (the Detector) immediately looks at every new painting.

• "Is this actually a vacuum cleaner?"
• "Does it look like a vacuum cleaner, or just a random shape?"
• "Is the label correct?"

If the Critic says, "No, this is garbage," the painting is flagged as defective.

Step C: The "Do-Over" (Refinement)

Instead of throwing the bad painting away and giving up, the system gives the robot artist a second chance.

• The Critic says: "You tried to paint a vacuum cleaner, but you failed. Try again!"
• The robot paints 20 new versions of that specific vacuum cleaner, using the same blueprint.

Step D: The Final Selection (The "Unique & Confident" Rule)

Now, the Critic has 20 new options. It doesn't just pick the first one that looks okay. It uses two rules to pick the winner:

1. Confidence: "Which one do I (the Critic) feel 100% sure is a vacuum cleaner?"
2. Uniqueness: "Which one looks different from the other good vacuum cleaners we already have?"

Why Uniqueness? If we already have a perfect red vacuum cleaner, we don't want another perfect red one. We want a blue one, or a different angle, so the apprentice learns that vacuum cleaners come in all shapes and sizes.

The system picks the most confident AND most unique image to replace the bad one.

Why This Matters

Think of it like curating a museum exhibit.

• Old Method: You grab 50 random paintings from a hat. Some are masterpieces, some are scribbles. The visitor (the AI student) gets confused.
• New Method: You grab 50 paintings, but you have a Security Guard (the Detector) checking every single one. If a painting is a scribble, the guard sends the artist back to the studio to paint 20 more versions until they get one that is both perfect and unique.

The Results

The paper tested this on famous image datasets (like CIFAR-10 and ImageNette).

• Before: The AI student learned from messy data and got confused.
• After: The AI student learned from a "super-charged" mini-dataset where every single image is high-quality and clearly labeled.
• The Outcome: The student scored significantly higher on tests, even when the dataset was tiny.

In a Nutshell

This paper is about quality control. It admits that AI artists make mistakes when creating fake data. So, instead of trusting the artist blindly, it adds a smart supervisor who catches the mistakes, forces a re-do, and ensures the final collection of images is not only correct but also diverse enough to teach the AI everything it needs to know.

Technical Summary

1. Problem Statement

Dataset Distillation (DD) aims to synthesize a small, compact surrogate dataset that allows models to achieve performance comparable to training on the original large-scale dataset. While recent generative approaches (using GANs and Diffusion Models) have improved scalability and image resolution compared to traditional pixel-space optimization methods, they suffer from two critical limitations:

1. Label Inconsistency: Generated images often contain incorrect labels or semantic mismatches (e.g., a "vacuum cleaner" image containing only background textures).
2. Structural Deficiencies: Synthetic samples frequently lack sufficient class-discriminative details or structural coherence, leading to poor feature extraction in downstream tasks.

Quantitative analysis in the paper reveals that existing diffusion-based methods (like D4M) can produce datasets where up to 12% of labels are incorrect and 5% of samples have low confidence scores (<0.7), significantly degrading model reliability.

2. Methodology

The authors propose a Detector-Guided Dataset Distillation Framework that integrates a pre-trained detector to identify and refine anomalous synthetic samples. The framework consists of two primary modules:

A. Prototype-Guided Image Synthesis

• Prototype Extraction: The original dataset is processed to extract latent features. K-means clustering is applied to partition each class into $C$ clusters (based on the Images-Per-Class, IPC, setting). The cluster centers serve as image prototypes.
• Generation: A Latent Diffusion Model (LDM), specifically Stable Diffusion, is used to synthesize images. The generation is conditioned on:
  1. The extracted image prototype (latent representation).
  2. The corresponding class label text (encoded via a text encoder like CLIP).
• This approach allows for the generation of diverse images while maintaining semantic alignment with the class prototype.

B. Anomaly Detection and Iterative Refinement

This is the core novelty of the work. Instead of accepting all generated images, the system employs a quality control loop:

1. Detection: A detector model (trained on the original dataset using CutMix augmentation) evaluates the synthetic dataset. A sample is flagged as defective if:
   • The predicted label differs from the intended label.
   • The softmax confidence score for the target class falls below a threshold $\beta$.
2. Candidate Generation: For every defective sample, the system re-extracts its prototype and label to generate multiple candidate images (e.g., 20 variants) using the same diffusion conditioning.
3. Selection Strategy: The optimal candidate is selected based on a dual criterion:
   • Confidence: The candidate must rank in the top-$k$ based on the detector's confidence score and exceed the threshold $\beta$.
   • Diversity: Among the high-confidence candidates, the one with the maximum feature-level dissimilarity (minimum cumulative cosine similarity) to existing qualified samples of the same class is chosen.
   • Fallback: If no candidate meets the confidence criteria, the one with the highest confidence is selected to ensure the prototype is represented.

3. Key Contributions

1. Detector-Guided Framework: Introduces a novel pipeline that explicitly leverages a pre-trained detector to filter and refine synthetic data, directly addressing label noise and structural flaws in generative distillation.
2. Targeted Refinement Strategy: Proposes a mechanism to regenerate multiple variations for defective samples and selects the most diverse yet confident candidate. This enhances intra-class diversity while ensuring label accuracy.
3. State-of-the-Art Performance: Demonstrates that the method generates high-quality, structurally coherent images that significantly outperform existing baselines in downstream classification tasks.

4. Experimental Results

The method was evaluated on CIFAR-10, ImageNette, and ImageWoof across various Images-Per-Class (IPC) settings and architectures (ResNet, ConvNet).

• Performance Gains:
  • ImageWoof: Outperformed the D4M baseline by an average of 1.7% in Top-1 accuracy. The gap widened with higher IPC (e.g., +3.1% at IPC=100).
  • ImageNette: Consistently outperformed D4M and other baselines (DM, Minimax, DiT). At IPC=10, it achieved a 2.4% improvement over D4M.
  • CIFAR-10: Achieved 39.8% accuracy at IPC=10, surpassing D4M by 3.7% and RDED by 2.7%.
• Quality Metrics:
  • Label Consistency: Reduced incorrect labels from ~10.2% (D4M) to 0.2% and eliminated samples with confidence <0.7.
  • Generation Quality: Improved FID scores (lower is better) and increased Precision, Density, and Coverage metrics on both ImageNette and ImageWoof.
• Visualization:
  • Grad-CAM Analysis: Models trained on the distilled dataset showed accurate attention maps focused on target objects, whereas D4M-trained models often focused on backgrounds or irrelevant regions.
  • Qualitative: Generated images exhibited more complete semantic structures and clearer class-discriminative features compared to baselines.

5. Significance and Conclusion

This work addresses a critical bottleneck in dataset distillation: the trade-off between generative efficiency and data quality. By integrating anomaly detection into the distillation loop, the authors ensure that the compact surrogate dataset is not only small but also semantically reliable.

• Impact: The method enables more efficient training in resource-constrained settings (e.g., privacy-preserving learning, continual learning) without sacrificing model performance due to noisy synthetic data.
• Limitations & Future Work: The authors note that K-means prototypes may not fully capture the complexity of the original data distribution. Future work will explore more advanced prototype construction techniques to further enhance representativeness.

In summary, the paper presents a robust, detector-guided approach that significantly advances the state-of-the-art in dataset distillation by ensuring label consistency and structural integrity in synthetic datasets.