Learnability-Guided Diffusion for Dataset Distillation

1. Problem Statement

Dataset Distillation aims to synthesize a small synthetic dataset ($D_S$) from a large real dataset ($D_T$) such that models trained on $D_S$ achieve performance comparable to those trained on $D_T$.

The Core Issue: Redundancy in Existing Methods
Current state-of-the-art methods (e.g., Influence-Guided Diffusion, Minimax Diffusion) typically optimize for visual diversity or average training trajectories (matching gradients across the whole dataset). The authors identify a critical flaw: these approaches produce redundant training signals.

• Empirical Finding: When existing distilled datasets are partitioned into disjoint subsets, any single subset captures 80–90% of the training signal found in other subsets.
• Consequence: Models trained on these datasets learn overlapping information rather than complementary knowledge. The optimization process converges to "medium-strength" gradients that are useful at no specific training stage, failing to leverage the natural progression of learning (from coarse features to fine-grained details).

2. Methodology: Learnability-Guided Diffusion (LGD)

The authors propose a paradigm shift from static, one-shot synthesis to an incremental, learnability-driven framework. The core idea is to construct the distilled dataset stage-by-stage, conditioning each new batch of samples on the current state of the model to ensure they provide new learning signals.

A. Incremental Distillation Formulation

The distilled dataset $D_S$ is partitioned into $K$ disjoint increments $\{I_1, I_2, ..., I_K\}$.

1. Initialization: Start with a small seed dataset (e.g., 10 Images Per Class - IPC).
2. Iterative Loop:
   • Train a model $\theta_{i-1}$ on the cumulative dataset $D_{i-1} = \bigcup_{k=1}^{i-1} I_k$.
   • Synthesize the next increment $I_i$ specifically to maximize the learning gain for $\theta_{i-1}$.
   • Update the dataset and repeat.

B. Learnability Score & Objective

To determine which samples are most useful for the current model, the authors define a Learnability Score $S(x, y)$:
$$S(x, y) = \mathcal{L}(\theta_{i-1}, x, y) - \omega \cdot \mathcal{L}(\theta^*, x, y)$$

• $\mathcal{L}(\theta_{i-1}, x, y)$: Loss on the current learner. High loss indicates the sample is currently difficult/hard for the model.
• $\mathcal{L}(\theta^*, x, y)$: Loss on a reference model (trained on the full dataset). This acts as a regularizer to ensure the sample is semantically valid and learnable (not out-of-distribution).
• Goal: Generate samples that are hard for the current model (high $\mathcal{L}(\theta_{i-1})$) but easy/valid for the reference model (low $\mathcal{L}(\theta^*)$). This targets the "learnable gap."

C. Learnability-Guided Diffusion Sampling

The synthesis process uses a diffusion model (DiT) guided by the learnability score:

1. Guidance Signal: During the denoising process, the predicted noise is adjusted by the gradient of the learnability score:
   $$\tilde{\epsilon}_\phi = \epsilon_\phi + \lambda \cdot \rho_t \cdot \nabla_{x_t} S(x_t, y)$$
   This steers the generation toward regions of the data distribution that the current model struggles with.
2. Diversity Mechanism: To prevent the new samples from simply copying existing ones, a deviation guidance term is added, repelling the generated sample from the memory buffer of previously synthesized samples.
3. Selection: For each target sample, $\kappa$ candidates are generated. The one with the highest learnability score is selected, ensuring high-quality, non-redundant additions.

3. Key Contributions

1. Learnability-Driven Framework: A novel incremental approach that reframes dataset distillation as a sequential curriculum learning problem, generating complementary rather than redundant data.
2. Learnability-Guided Diffusion (LGD): A specific synthesis algorithm that conditions diffusion generation on the current model's learning frontier, balancing informativeness with semantic validity.
3. Redundancy Analysis: A diagnostic framework demonstrating that existing methods suffer from 80–90% signal overlap, which LGD reduces by 39.1%.
4. State-of-the-Art Performance: Achieving new records on large-scale datasets where previous methods struggled.

4. Experimental Results

The method was evaluated on ImageNette, ImageWoof, and ImageNet-1K across various architectures (ConvNet-6, ResNet-AP-10, ResNet-18) and IPC budgets (20–100).

• Performance Gains:
  • ImageNet-1K (50 IPC): Achieved 60.1% accuracy, matching the SOTA (MGD3 at 60.2%) and significantly outperforming the base DiT (52.9%).
  • ImageNette (100 IPC): Achieved 87.2%, outperforming IGD (84.5%) and DiT (78.2%).
  • ImageWoof (100 IPC): Achieved 72.9%, the best reported result.
• Redundancy Reduction: Cross-increment validation showed that models trained on one LGD increment had significantly lower accuracy on subsequent increments (avg. 57.65% vs. 94.7% for DiT), proving the data is complementary.
• Training Dynamics:
  • Loss Spikes: LGD showed large loss spikes when new increments were added, indicating the new data was genuinely challenging and informative.
  • Curriculum Effect: The method naturally created a curriculum where early increments captured easy structures and later increments introduced complex, hard-to-learn features.
• Generalization: The distilled datasets showed strong cross-architecture transferability, maintaining high accuracy when tested on architectures different from the one used for distillation.

5. Significance

This paper addresses a fundamental bottleneck in dataset distillation: sample redundancy. By shifting the focus from "matching average gradients" to "targeting the current model's learning frontier," the authors demonstrate that:

1. Efficiency: Smaller, non-redundant datasets can achieve performance equal to or better than larger, redundant ones.
2. Scalability: The incremental, curriculum-based approach allows distillation to scale effectively to high-resolution, large-scale datasets like ImageNet-1K, which were previously difficult to distill.
3. Theoretical Insight: It establishes a connection between dataset distillation and curriculum learning, suggesting that the order and difficulty of synthetic data matter as much as the data itself.

The proposed Learnability-Guided Diffusion provides a principled way to generate synthetic data that evolves with the learner, offering a new standard for efficient and effective dataset distillation.