Technical Summary: Rethinking Dataset Distillation for Classification

Problem Statement

Dataset Distillation (DD) aims to synthesize compact, synthetic training sets that preserve the generalization performance of models trained on large-scale original datasets. Despite its promise in data-centric machine learning, the field faces two critical challenges:

1. Inconsistent Evaluation Protocols: DD methods are often evaluated using varying pipelines (e.g., standard Empirical Risk Minimization vs. single/multi-teacher soft-label supervision). This makes it difficult to isolate whether performance gains stem from the quality of the distilled data or specific evaluation design choices.
2. Unsubstantiated Superiority over Coresets: Many DD methods claim to outperform Coreset Selection (CS)—which selects subsets of real data—based on the assumption that restricting condensed data to real samples limits expressiveness. However, these claims often lack systematic comparison against strong CS baselines, particularly on large-scale datasets and under standardized protocols.

Furthermore, many DD approaches rely on pretrained generative models trained on the full dataset, limiting their scalability and applicability to domains where such models are unavailable.

Methodology

The authors conduct a large-scale, systematic benchmarking study to disentangle the intrinsic effectiveness of DD from evaluation artifacts and to rigorously compare DD against CS.

Experimental Design

• Datasets: Experiments are performed on ImageNet-1K (1,000 classes), ImageNet-100 (100 classes), and ImageNette (10 classes).
• Methods Compared:
  • DD Methods (7 SOTA): The study covers three categories:
    • Generative Training-based: VLCP, D3HR, Minimax (require fine-tuning generative models).
    • Generative Training-free: DiT-Distillation, MGD, ManifoldGD (use pretrained generative models with guided sampling).
    • Non-Generative/Decoupled: FADRM+ (optimizes samples via data residual matching).
  • CS Methods (3 Strategies):
    • Random Sampling: A simple class-wise baseline.
    • AUM (Accumulated Margin): Scores samples based on training dynamics (margin) of a model trained on the full dataset.
    • Concept-guided CS: Uses LLM-based concept bottlenecks to estimate sample importance without full dataset training.
• Evaluation Protocols: To test sensitivity to supervision, three protocols are used:
  1. Hard Label: Models trained directly on condensed sets with original labels (no teacher).
  2. Single-Teacher Soft Label (RDED): Uses one pretrained teacher to generate soft labels for knowledge distillation.
  3. Multi-Teacher Soft Label (EDC): Uses multiple teachers and hybrid supervision (soft + hard labels).
• Metrics: Beyond accuracy, the study evaluates Representativeness (coverage of original distribution), Diversity (redundancy within the condensed set), and Construction Cost (time and computational resources).

Key Contributions

1. Systematic Benchmarking: The paper provides the first comprehensive comparison of seven SOTA DD methods against three CS strategies under uniform datasets and evaluation protocols.
2. Protocol Sensitivity Analysis: It demonstrates that DD performance is highly entangled with the evaluation pipeline, with no single DD method consistently outperforming others across all protocols and datasets.
3. Re-evaluation of DD vs. CS: The study challenges the narrative that DD inherently outperforms CS, showing that CS methods often match or exceed DD performance, particularly on large-scale datasets.
4. Multi-Dimensional Quality Assessment: The authors introduce a broader evaluation framework including representativeness, diversity, and construction cost, revealing that CS methods often provide better distributional coverage and efficiency.

Results

Accuracy Performance

• Large-Scale Datasets (ImageNet-1K/100): Under the Hard Label protocol, CS methods (specifically AUM and Concepts) consistently match or significantly outperform SOTA DD methods. For example, on ImageNet-1K with 10 IPC (Images Per Class), AUM outperforms the best DD method by 3.3% (ResNet-18) and 4.8% (ResNet-50).
• Soft Label Protocols: Even when using single or multi-teacher soft labels (where DD typically gains an advantage), CS methods remain highly competitive, often matching or surpassing DD performance.
• Small-Scale Datasets (ImageNette): DD methods perform well on ImageNette, sometimes outperforming CS. However, the authors note that the "best" DD method varies by protocol, highlighting the instability of DD rankings compared to the robustness of CS.
• Random Baselines: Several DD methods fail to outperform simple random subsets of real data, particularly under hard-label evaluation.

Efficiency and Quality

• Construction Cost: CS methods exhibit constant time complexity relative to IPC (Images Per Class), whereas DD methods (especially generative ones) scale linearly with IPC. Training-based DD methods incur substantial costs due to generative model training/fine-tuning.
• Representativeness and Diversity: CS methods (particularly AUM) achieve a superior trade-off, occupying the high-representativeness and high-diversity region of the metric space. DD methods tend to concentrate on narrower visual modes, resulting in lower diversity and higher redundancy.
• Visual Quality: Qualitative analysis shows that CS subsets (AUM) capture diverse poses, viewpoints, and backgrounds. In contrast, DD sets often exhibit repeated canonical views, artifacts, and unnatural textures.

Significance and Claims

The paper argues that the current success of Dataset Distillation requires reevaluation. The authors claim that:

1. CS is a Necessary Baseline: Future DD research must include strong CS baselines (like AUM) to avoid misinterpreting evaluation-specific gains as intrinsic data efficiency.
2. Standardized Evaluation is Critical: The field must adopt standardized protocols, including hard-label evaluation, to fairly assess the true utility of distilled data.
3. Practical Efficiency: For large-scale data-centric learning, Coreset Selection remains a more computationally efficient, scalable, and robust alternative to current DD methods, offering comparable or superior accuracy with significantly lower construction costs and better distributional coverage.

The authors conclude that while DD shows promise on small, separable datasets, its practical advantages diminish on large-scale benchmarks where CS provides a more efficient and reliable solution.