Rethinking Dataset Distillation: Hard Truths about Soft Labels

1. Problem Statement

Dataset Distillation (DD) aims to synthesize a small, compact dataset that retains the essential information of a large original dataset for downstream model training. While recent large-scale DD methods (e.g., SRe2L) have achieved state-of-the-art (SOTA) results, they rely heavily on Soft Labels (SL) and Knowledge Distillation (KD) (the SL+KD regime), where a teacher model provides soft labels for augmented images during student training.

Recent evidence suggests a critical discrepancy:

• In the SL+KD regime, simple random subsets of real images often perform on par with sophisticated DD methods.
• In contrast, in the Hard Label (HL) regime (standard coreset selection), high-quality subsets consistently outperform random baselines.

The paper investigates why high-quality data selection fails to provide benefits in soft-label regimes and whether current DD methods are genuinely distilling information or merely exploiting the supervision signal of soft labels. It further seeks to identify scalable, effective DD methods for the Hard Label setting.

2. Methodology and Analysis

The authors conduct a systematic analysis across three label regimes: SL+KD (multiple soft labels per image), SL (fixed soft label per image), and HL (hard labels).

A. Scalability Analysis of Label Regimes

The authors analyze how dataset quality, size, and compute interact in different regimes using ImageNet-1K and TinyImageNet.

• SL+KD Regime: They find that performance is dominated by compute budget rather than data quality or size. High-quality coresets (e.g., EL2N) and random subsets converge to near-optimal performance (close to the full dataset) as long as the compute budget is sufficient. The "Pareto frontier" (optimal trade-off between size and difficulty) vanishes; subset quality has negligible influence.
• SL Regime: While dataset size matters, subset quality offers limited benefits beyond a minimum IPC (Images Per Class). The authors attribute this to the homogenization of sample difficulty: under fixed soft labels, the per-sample influence scores (derived from a modified EL2N metric) cluster tightly, making all samples contribute similarly to learning.
• Conclusion: The widespread use of SL+KD for evaluation masks the true efficacy of DD methods, as it allows random subsets to succeed.

B. Evaluation of Small-Scale DD Methods (DCS)

To understand why methods like Trajectory Matching (TM) and DATM work on small datasets but fail to scale, the authors propose the Distillation Correlation Score (DCS).

• DCS Metric: It measures the Spearman rank correlation between a distillation loss objective and the downstream generalization error across diverse subsets.
• Finding: DCS reveals that TM-based objectives fail to scale to larger architectures (e.g., ResNet-18). For larger models, the TM loss remains constant (~0.999) regardless of the subset quality, failing to capture meaningful variations needed for effective synthesis.

C. Proposed Method: CAD-Prune and CA2D

Focusing on the Hard Label (HL) setting where data quality matters, the authors critique the current SOTA large-scale method, RDED, for relying on a simple heuristic (selecting only the most confident patches).

• CAD-Prune (Compute-Aware Difficulty Pruning): A pruning metric that identifies samples of "optimal difficulty" for a specific compute budget.
  • Instead of using static scores, it estimates per-sample uncertainty by training a model on the full dataset with a compute budget matching the downstream task.
  • It averages uncertainty scores over a small window near the end of the training period to capture samples that are actively being learned at that budget level.
• CA2D (Compute-Aware Dataset Distillation): A DD method built on RDED but using CAD-Prune to select patches. It constructs a synthetic set using confident patches from examples of suitable difficulty, aligned with the available compute.

3. Key Contributions

1. Revelation of Soft Label Saturation: The paper demonstrates that in SL+KD and SL regimes, dataset quality and size have negligible impact on performance once a minimum compute budget is met. This challenges the validity of using SL+KD as the primary evaluation metric for DD.
2. DCS Metric: Introduction of a zero-shot, efficient metric (DCS) to evaluate distillation objectives without expensive synthesis. It successfully identifies that trajectory-matching objectives do not scale to larger models.
3. CAD-Prune & CA2D:
   • CAD-Prune: A compute-aware pruning strategy that outperforms the computationally expensive "sliding window" approach used in previous coreset methods (e.g., EL2N-Best) while being significantly more efficient.
   • CA2D: A new DD method that achieves SOTA results on ImageNet-1K in the Hard Label setting, outperforming both RDED and the best coreset baselines.

4. Key Results

• SL+KD Regime: Random subsets achieve accuracy comparable to high-quality coresets (e.g., EL2N) and SOTA DD methods (SRe2L, DWA) on ImageNet-1K. Performance saturates near the full-dataset baseline regardless of subset quality.
• SL Regime: Similar saturation occurs; subset quality differences vanish, and performance is driven by compute.
• HL Regime (ImageNet-1K):
  • RDED is the only large-scale DD method that reliably beats random baselines but still lags behind optimal coreset methods.
  • CA2D (using CAD-Prune) outperforms RDED and EL2N-Best across IPC 10, 50, and 100 settings.
  • CAD-Prune matches the performance of the exhaustive sliding-window selection (EL2N-Best) but requires only a single compute-matched training run, offering 2–4x compute savings.
• Scalability: TM-based methods (TM, DATM) show strong performance on TinyImageNet with ConvNet-D4 but fail to scale to ResNet-18, confirmed by DCS showing a lack of correlation between loss and generalization.

5. Significance and Implications

• Paradigm Shift in Evaluation: The paper argues that the community should move away from evaluating DD methods solely in the SL+KD regime, as it renders data quality irrelevant. Evaluation should prioritize the Hard Label (HL) setting to truly measure the ability to distill information.
• Compute-Aware Learning: The work establishes that "optimal" data subsets are not static; they depend on the available compute budget. A sample that is "easy" for a high-budget model might be "hard" (and thus more informative) for a low-budget model.
• Efficiency: By introducing CAD-Prune, the authors provide a practical tool to select high-quality data subsets without the prohibitive cost of sliding-window searches, making high-quality data selection feasible for large-scale datasets.
• Future Directions: The findings suggest that in "model-stealing" scenarios (where a student learns from a teacher's soft labels), a small dataset with repeated supervision can achieve near-optimal performance, reducing the need for massive data collection in specific transfer learning contexts.

In summary, the paper exposes a fundamental flaw in current DD evaluation practices (reliance on soft labels) and provides a rigorous framework and new methods (CAD-Prune/CA2D) to advance data-efficient learning in the more challenging and meaningful Hard Label regime.