💻 computer science

Rectified Decoupled Dataset Distillation: A Closer Look for Fair and Comprehensive Evaluation

This paper introduces Rectified Decoupled Dataset Distillation (RD $^3$ ), a framework that systematically analyzes and standardizes inconsistent post-evaluation protocols in existing decoupled methods to reveal that reported performance variations often stem from procedural discrepancies rather than intrinsic method quality, thereby establishing a fair and reproducible benchmark for future research.

Original authors: Xinhao Zhong, Shuoyang Sun, Xulin Gu, Chenyang Zhu, Bin Chen, Yaowei Wang

Published 2026-02-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinhao Zhong, Shuoyang Sun, Xulin Gu, Chenyang Zhu, Bin Chen, Yaowei Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a student how to recognize animals. Usually, you'd give them a massive library of photos (the "real dataset") to study. But what if you could shrink that entire library down to just a few perfect, tiny flashcards (the "synthetic dataset") that still teach the student everything they need to know? This is the goal of Dataset Distillation.

However, the paper argues that the current way researchers are testing these "flashcards" is like judging a race where everyone starts at different lines, runs on different surfaces, and uses different shoes. It's not a fair race, and the results are misleading.

Here is a breakdown of the paper's findings using simple analogies:

1. The Problem: The "Unfair Race"

For a long time, researchers have been creating these tiny synthetic datasets and claiming, "My method is the best!" They show off huge score improvements (like going from 20% to 45% accuracy).

But the authors of this paper realized something suspicious: The scores weren't actually about the quality of the flashcards. They were about how the researchers set up the final exam.

Some researchers gave their student models extra study time.
Some used a different type of teacher to grade the answers.
Some used special "training wheels" (data augmentation) that others didn't use.

It was like comparing a runner who had a tailwind, a smooth track, and a personal coach against a runner who had to run uphill in the rain. The first runner looked amazing, but it wasn't because they were faster; it was because the conditions were rigged.

2. The Solution: RD3 (The "Standardized Track")

To fix this, the authors created RD3 (Rectified Decoupled Dataset Distillation). Think of this as building a perfectly flat, standardized track where every runner must wear the same shoes and run the same distance.

They took all the popular methods (Optimization-based, Selection-based, and Generation-based) and forced them to compete under one single, strict set of rules:

Same amount of training time.
Same type of teacher model to generate "soft labels" (hints).
Same data augmentation (no special tricks).

3. The Shocking Results: The Gap Shrinks

When they ran this fair race, the results changed dramatically.

The Big Lie: Previously, the gap between the "best" and "worst" methods looked huge (over 27% difference).
The Reality: Under the fair rules, that gap shrank to less than 7%.

The Analogy: Imagine a group of chefs claiming their soup is vastly superior. When you taste them all using the exact same salt, water, and temperature, you realize they all taste almost the same. The "superiority" was just because one chef used a fancy stove and another used a different pot.

4. What Actually Matters? (Efficiency & Generalization)

Since the accuracy scores are now so close, the paper says we should stop obsessing over tiny percentage points and look at other things:

Time (Efficiency): Some methods take 100 hours to create their flashcards, while others take 1 hour. If the flashcards are almost equally good, the one that takes 1 hour is the clear winner.
Generalization: Can the flashcards teach a student who uses a different brain architecture? Some methods work great on simple students but fail on complex ones.

5. The "Magic Trick" That Wasn't Magic

The paper discovered two "secret weapons" that many researchers were accidentally using to boost their scores, which made their methods look better than they really were:

Better Starting Points: Some methods started with "random noise" while others started with "random real images." Starting with real images gave a huge unfair advantage.
Hybrid Grading: Some methods used a committee of teachers to grade the answers, while others used just one. Using a committee made the scores look higher, but it wasn't part of the core method.

6. The Biggest Surprise: Randomness is Powerful

The most mind-blowing finding is this: If you just grab random photos from the original library and give them to the student with "soft labels" (hints from a teacher), it often beats the complex, fancy methods.

On simple topics (like "Cat" vs. "Dog"): A random pile of photos works surprisingly well because the variety is enough to teach the student.
On hard topics (like "100 different breeds of dogs"): The fancy methods that carefully select specific parts of images still win because random photos are too confusing.

Summary

The paper is a "reality check" for the field of Dataset Distillation. It says:

Stop comparing apples to oranges. We need a standard way to test these methods.
Many "breakthroughs" were just better settings. Once we fix the settings, the methods are much more similar than we thought.
Don't ignore the basics. Sometimes, a simple random selection of images is just as good as a complex algorithm, especially if you use the right way to grade the student.

The authors hope that by cleaning up the rules, future research will focus on making truly better synthetic datasets rather than just tweaking the exam conditions to get a higher score.

Technical Summary: Rectified Decoupled Dataset Distillation (RD3)

Problem Statement

Dataset distillation (DD) aims to generate compact synthetic datasets that enable models trained on them to match the performance of models trained on full real datasets, thereby reducing storage and computational costs. While early bi-level optimization methods (e.g., MTT) showed promise on small-scale datasets, they suffer from prohibitive computational overhead when scaling to large datasets like ImageNet-1K.

To address scalability, recent decoupled dataset distillation methods (e.g., SRe2L, CDA, RDED) separate teacher model pre-training from synthetic data generation. These methods often employ post-evaluation techniques such as random data augmentation and epoch-wise soft labels to improve performance. However, the field faces a critical challenge: inconsistent post-evaluation protocols. Existing studies utilize varying settings for batch sizes, learning rate schedulers, data augmentations, and label types (hard vs. soft). This inconsistency creates confounded performance gains, making it difficult to determine whether reported improvements stem from genuine methodological advances in data synthesis or merely from favorable evaluation hyperparameters. As illustrated in the paper, reported performance gaps between methods can exceed 27%, yet under unified settings, this gap shrinks to less than 7%.

Methodology: Rectified Decoupled Dataset Distillation (RD3)

The authors propose RD3, a unified and comprehensive baseline framework designed to ensure fair and reproducible comparisons across decoupled dataset distillation methods. The methodology involves:

Standardized Evaluation Protocol: The authors establish a consistent set of post-evaluation settings to eliminate confounding variables. Key standardizations include:
- Training Epochs: Fixed at 400 epochs to align convergence iterations across methods.
- Batch Size: Uniformly set to 50 for ResNet-18 evaluation (and 100 for generalization tasks) to balance performance and efficiency.
- Learning Rate Scheduler: Implementation of a smoothing learning rate scheduler with a universal smoothing factor ( $\zeta$ ) tailored to the evaluation architecture.
- Data Augmentation: Adoption of a unified augmentation strategy (including CutMix, Random Resized Cropping, and patch exchange) across all methods.
- Soft Labels: Exclusive use of a single pre-trained ResNet-18 to generate soft labels via KL divergence, ensuring fairness by avoiding ensemble-based hybrid labels unless explicitly tested as a variable.
Systematic Re-evaluation: The framework re-evaluates representative methods across three paradigms:
- Optimization-based: SRe2L, CDA, G-VBSM, DWA, EDC.
- Generation-based: Minimax, D4M.
- Selection-based: RDED.
  These are tested on six datasets (CIFAR-10/100, TinyImageNet, ImageNette, ImageWoof, ImageNet-1K) with varying Images Per Class (IPC) and across multiple network architectures (ResNet series, EfficientNet, MobileNet, Swin-T, ViT-B).
Ablation of Subtle Factors: The study investigates factors often overlooked in prior comparisons, including:
- Initialization: The impact of different initialization strategies (e.g., Gaussian noise vs. random sampling) on optimization-based methods.
- Hybrid Soft Labels: The effect of using multiple teacher models to generate soft labels.
- Loss Functions: Comparing KL divergence against MSE + Ground Truth alignment.
- Baselines: Testing random image sampling as a baseline under soft label paradigms.

Key Results

1. Evaluation Inconsistencies Drive Reported Gaps

Under the unified RD3 framework, the performance differences between methods are significantly reduced.

ImageNet-1K (IPC=10): The reported performance gap between methods was previously ~27.3%. After rectification, the gap drops to 6.7%.
Method Ranking: No single method consistently outperforms all others across all datasets and IPC settings. For instance, EDC (Optimization-based) dominates on CIFAR and low-IPC settings, while D4M (Generation-based) excels in high-IPC settings due to image diversity. RDED (Selection-based) shows degradation as IPC increases.

2. Efficiency and Generalization are Critical Metrics

Efficiency: While accuracy differences are marginal, computational costs vary by orders of magnitude (up to 100x). Optimization-based methods require classifier training, making their deployment costs significantly higher than reported generation times.
Generalization: Synthetic datasets show varying generalization capabilities. Methods with higher image diversity (e.g., D4M) transfer better to Transformer-based architectures (ViT, Swin), whereas methods with lower diversity struggle on these models.

3. Impact of "Hidden" Variables

Initialization: The choice of initialization significantly affects optimization-based methods. For example, DWA and EDC degrade with Gaussian noise, while G-VBSM improves with random sampling.
Hybrid Soft Labels: Applying hybrid soft labels (from multiple teachers) to methods not originally designed for them yields substantial performance gains (e.g., SRe2L doubles its performance at IPC=1).
Loss Functions: Switching from KL divergence to MSE + GT alignment generally improves performance, suggesting loss function selection is a critical, often uncontrolled variable.

4. The Random Sampling Baseline

A surprising finding is that randomly sampled images, when evaluated with soft labels, outperform all existing decoupled distillation methods on coarse-grained datasets (CIFAR-10/100, ImageNette, ImageNet-1K). This suggests that for these datasets, the diversity of random samples combined with soft label alignment is more effective than complex distillation mechanisms. Conversely, on fine-grained datasets (TinyImageNet, ImageWoof), distillation methods remain superior to random sampling.

Significance and Claims

The paper claims that the field of dataset distillation has been hindered by inconsistent evaluation protocols that conflate algorithmic innovation with hyperparameter tuning and auxiliary enhancements.

Fair Benchmarking: RD3 provides the first exhaustive evaluation of representative decoupled methods under fully standardized conditions, offering a foundation for fair comparison.
Re-evaluation of Progress: The authors argue that many reported performance gains are not due to intrinsic improvements in synthetic data quality but rather to inconsistent evaluation settings (e.g., batch size, schedulers, augmentation).
New Baselines: The work identifies random sampling with soft labels as a powerful, previously overlooked baseline that future research must surpass, particularly for coarse-grained tasks.
Holistic Evaluation: The paper advocates for moving beyond test accuracy to include computational efficiency and generalization across diverse architectures as primary evaluation criteria.

In conclusion, the authors posit that future progress in dataset distillation requires rigorous adherence to standardized protocols to distinguish true methodological advances from artifacts of evaluation design.