Technical Summary: DIVER - Diving Deeper into Distilled Data via Expressive Semantic Recovery

1. Problem Statement

Dataset Distillation (DD) aims to compress large-scale, semantically rich datasets into compact, human-unreadable proxy datasets to facilitate efficient learning and privacy protection. However, classical DD frameworks typically employ a single-stage bi-level optimization paradigm. In this process, the optimization occurs directly in pixel space, often using a specific prior architecture (e.g., ConvNet) for matching strategies (gradient, distribution, or trajectory matching).

This approach leads to two critical issues:

1. Overfitting to Architecture-Specific Patterns: The distilled images tend to learn specific high-frequency patterns and noise tailored to the prior architecture used during distillation, rather than capturing generalizable high-level semantics.
2. Cross-Architecture Generalization Failure: Consequently, these distilled images exhibit abstract, noisy, and unrealistic characteristics that suppress the expression of visual semantics. This results in significant performance degradation when the distilled dataset is evaluated on heterogeneous architectures (e.g., Transformers, MobileNets) different from the prior architecture.

Existing attempts to address this, such as generative prior methods (using GANs) or dual-time matching, either remain constrained by expensive inner-loop mechanisms, fail to produce realistic images, or represent a complete departure from the classical DD framework, discarding the potential of existing distilled datasets.

2. Methodology: The DIVER Framework

The authors propose DIVER (Diving Deeper into Distilled Data via Expressive semantic Recovery), a novel dual-stage distillation framework. Unlike traditional approaches that stop after generating a distilled dataset, DIVER treats the output of classical DD as an intermediate input for a second stage of refinement.

Stage I: Classical Dataset Distillation

This stage is identical to standard DD. It extracts a compact distilled dataset ($D^*$) from the original dataset ($O$) using existing techniques (e.g., MTT, DM, SRe2L). The authors note that if a distilled dataset already exists, Stage I can be skipped.

Stage II: Diving into Distilled Data (DDD)

The core innovation lies in Stage II, termed Diving into Distilled Data (DDD). This stage employs a pre-trained, frozen diffusion model (specifically a Diffusion Transformer, DiT) and a VAE to refine $D^*$ into a Synthetic Dataset ($S$) without requiring access to the original dataset or additional training. The process involves three semantic recovery strategies:

1. Semantic Inheritance (SI):

   • Mechanism: The distilled image $x_0$ is encoded into a latent code $z_0$ using a pre-trained VAE encoder.
   • Purpose: The VAE acts as a filter, projecting the image into a deep latent space. This process inherently filters out architecture-specific "noise" (high-frequency patterns overfit to the prior ConvNet) while retaining high-level semantic representations.
   • Initialization: Instead of starting the reverse diffusion process with random Gaussian noise, the process is initialized with a noisy version of the inherited latent ($z_{tf}$), created by adding $t_f$ steps of forward noise to $z_0$. This ensures the sampling trajectory remains proximate to the original semantic structure.

2. Semantic Guidance (SG):

   • Mechanism: A guidance function $G_t = ( \hat{z}_t - z_0 )^2 / 2\sigma_t$ is introduced during the reverse process.
   • Purpose: To prevent the semantic information inherited from the distilled dataset from degrading or drifting too far during the generative process. It steers the sampling to produce realistic outputs that remain faithful to the original distilled semantics.

3. Semantic Fusion (SF):

   • Mechanism: The authors leverage the observation that diffusion sampling occurs in distinct phases: Chaotic Phase (CP), Semantic Phase (SP), and Refinement Phase (RP). Semantic Fusion integrates conditional labels (class information) with the inherited and guided latents exclusively during the Semantic Phase (SP).
   • Purpose: Full-phase guidance can lead to semantic ambiguity and artifacts. By fusing category information only during the critical SP, the method ensures clear, category-specific semantics are injected without compromising the structural integrity or causing overfitting to the label guidance.

The entire Stage II process is training-free and raw-data-free, relying solely on the pre-trained generative models and the distilled dataset.

3. Key Contributions

• Dual-Stage Framework: The paper decouples the dataset distillation problem into DD (extraction) and DDD (refinement), proposing a new paradigm to enhance cross-architecture generalization.
• DDD Task Definition: The authors formally introduce "Diving into Distilled Data," the first work to distill knowledge from an existing distilled dataset into a synthetic dataset without accessing the original data.
• Semantic Recovery Strategies: The integration of Semantic Inheritance, Semantic Guidance, and Semantic Fusion into a pre-trained guided diffusion model. These strategies recover expressive semantics suppressed by architecture-specific patterns without requiring additional model training.
• Efficiency: The method acts as a plugin for traditional DD, achieving significant performance gains with minimal computational overhead (comparable to raw DiT inference) and low GPU memory usage (4 GB).

4. Experimental Results

Extensive experiments were conducted on ImageNet-1K and its subsets (e.g., ImageNette, ImageFruit) with resolutions of 128×128 and 256×256.

• Cross-Architecture Generalization: DIVER consistently improves the performance of various classical DD methods (DM, MTT, EDF, NCFM, SRe2L, G-VBSM) when evaluated on unseen architectures (ResNet, ShuffleNet, MobileNet, EfficientNet, ViT).
  • For example, on ImageNet subsets with IPC=1, DIVER improved MTT performance from ~15% to ~22% on average across architectures.
  • On ImageNet-1K (IPC=10), DIVER enhanced MTT from 21.3% to 24.5% on ResNet-18 and from 28.4% to 31.2% on ResNet-50.
• Comparison with Generative Priors: DIVER outperforms GLaD (a generative prior method) in generalization while being more efficient, as it does not require gradient computation or storing generator gradients.
• Ablation Studies:
  • Module Impact: Both Semantic Inheritance and Semantic Guidance individually provide significant gains. Their combination yields the best results, though with a saturation effect.
  • Guidance Factor: An optimal guidance factor ($\gamma=0.1$) is critical; too low leads to insufficient guidance, while too high causes over-reliance on the inherited latent, diminishing label conditioning.
  • Forward Steps: An intermediate number of forward steps ($t_f=25$) balances Gaussian alignment and feature preservation.
  • Robustness: The method demonstrates consistent improvements across different diffusion backbones (DiT, SiT, Stable Diffusion V1.5), though gains are most pronounced with DiT and SiT due to architectural alignment.
• Trade-off on Prior Architecture: The paper acknowledges a measurable decline in performance when the synthetic dataset is evaluated on the same prior architecture (ConvNet) used for the initial distillation. However, this is presented as an acceptable trade-off for the substantial gains in cross-architecture generalization.

5. Significance and Claims

The paper claims that DIVER effectively addresses the "cross-architecture generalization dilemma" inherent in classical dataset distillation. By treating the distilled dataset not as a final product but as a source of suppressed semantic information, DIVER leverages the powerful synthesis capabilities of pre-trained diffusion models to "dive deeper" and recover expressive semantics.

The significance of the work lies in:

1. Methodological Advancement: It bridges the gap between classical DD and generative models, showing that classical distilled datasets retain valuable semantic information that can be unlocked via a second-stage refinement.
2. Practical Efficiency: The approach requires no access to the original large-scale dataset and no additional training, making it highly suitable for privacy-sensitive scenarios and resource-constrained environments (e.g., edge computing).
3. Generalization: It provides a robust solution for deploying distilled datasets across diverse neural network architectures, a critical requirement for real-world applications where model heterogeneity is common.

The authors conclude that DIVER serves as an efficient, plug-and-play enhancement for traditional DD, significantly improving generalization with minimal storage and computational overhead.