Technical Summary: Spectral Gradient Surgery for Domain-Generalizable Dataset Distillation

1. Problem Formulation

The paper addresses a critical limitation in Dataset Distillation (DD): the assumption that training and test data are independent and identically distributed (i.i.d.). In practice, models trained on distilled datasets often suffer significant performance degradation when faced with domain shifts (Out-of-Distribution or OOD scenarios).

The authors identify that existing DD methods, particularly Distribution Matching (DM), inadvertently entangle class-discriminative information with domain-specific information during the compression process. Because standard DM methods aim to match the feature distribution of the entire source dataset without distinguishing between these two types of information, the resulting synthetic dataset lacks robustness to unseen domains.

Furthermore, the paper argues that simply applying post-hoc Domain Generalization (DG) techniques to distilled datasets is insufficient for two reasons:

1. Data Mismatch: Existing DG methods rely on the natural diversity of large-scale real datasets, an assumption violated by compact synthetic sets.
2. Efficiency Conflict: DG methods often employ heavy augmentation strategies that increase downstream training time, contradicting the core efficiency objective of DD.

Consequently, the authors introduce a new problem setting: Domain Generalizable Dataset Distillation (DGDD). This formulation explicitly targets the synthesis of a compact dataset that ensures robust generalization to unseen target domains, rather than merely preserving in-distribution (ID) performance.

2. Methodology: Spectral Gradient Surgery (SGS)

To solve the DGDD problem, the authors propose Spectral Gradient Surgery (SGS), a plug-and-play extension to existing DM methods. The core insight is that cross-domain agreement among domain-wise gradients in the spectral domain can effectively disentangle class-discriminative signals from domain-specific noise.

The methodology proceeds through the following steps:

A. Domain-Wise Gradient Computation

Given $S$ source domains, the method computes a specific gradient $g_i^s$ for each domain $D_s$ with respect to the synthetic sample $\hat{x}_i$. This gradient represents the direction to update $\hat{x}_i$ to better match the feature distribution of domain $s$.

B. Spectral Decomposition

Instead of operating in pixel space, SGS transforms these gradients into the spectral domain using the 2D Fast Fourier Transform (FFT). This is crucial because:

• Pixel-wise agreement is susceptible to spatial misalignment.
• Frequency components represent global spatial patterns, making them semantically meaningful for measuring cross-domain consensus.

C. Signal Separation via Resultant Length

The method computes the complex mean of gradients across all domains and measures the element-wise resultant length ($r_i$) at each frequency component.

• High Agreement ($r_i \to 1$): Indicates that gradient directions are aligned across domains, signifying class-discriminative information.
• Low Agreement ($r_i \to 0$): Indicates conflicting directions, signifying domain-specific noise.

Using this metric, the gradients are decomposed into two complementary signals:

1. Class-Discriminative Signal ($g_i^{class}$): Obtained by weighting the mean gradient by $r_i$. This reinforces patterns shared across all source domains.
2. Domain-Specific Signal ($g_i^{domain, s}$): Obtained by calculating the deviation of a specific domain's gradient from the consensus. This preserves unique characteristics of specific domains to encourage diversity.

D. Adaptive Optimization

The synthetic dataset is updated using a combination of three gradient signals:
$$\hat{x}_i \leftarrow \hat{x}_i - \eta \cdot (g_i + \lambda_c \cdot g_i^{class} + \lambda_d \cdot g_i^{domain, s(i)})$$
Where:

• $g_i$ is the standard DM gradient (serving as the baseline).
• $g_i^{class}$ reinforces domain-invariant patterns.
• $g_i^{domain, s(i)}$ preserves domain-specific variations (assigned uniformly to synthetic samples to ensure diversity).
• $\lambda_c$ and $\lambda_d$ control the strength of these additional signals.

3. Key Contributions

The paper outlines three primary contributions:

1. Problem Definition: The introduction of Domain Generalizable Dataset Distillation (DGDD), a novel setting that explicitly targets OOD generalization, distinguishing it from conventional DD which focuses on ID performance.
2. Algorithm Proposal: The development of Spectral Gradient Surgery (SGS), a method that disentangles class and domain information via spectral decomposition of gradients. It provides a class-discriminative signal for robustness and a domain-specific signal for diversity.
3. Plug-and-Play Compatibility: The demonstration that SGS can be integrated into existing DM frameworks (e.g., DM, HDD-DM) without modifying their core objectives, requiring no changes to the underlying optimization architecture.

4. Experimental Results

The authors evaluated SGS on three benchmarks: Digits-DG, PACS, and CORe50-Hard, under both Multi-Domain Generalization (MDG) and Single-Domain Generalization (SDG) settings.

• Performance Gains: SGS consistently improved OOD generalization across all datasets and Image Per Class (IPC) settings (10 and 20).
  • On PACS (IPC=10), SGS improved the average OOD accuracy of the baseline DM by +3.0% and HDD-DM by +2.2%.
  • On CORe50-Hard (IPC=10), it improved DM by +1.5% and HDD-DM by +3.4%.
• Comparison with Post-hoc DG: Unlike post-hoc DG methods (e.g., FACT, MixStyle, AdvFreq) which often failed to improve performance or drastically increased training time (up to 6.8x overhead), SGS achieved superior OOD accuracy with zero additional computational overhead during downstream training.
• Ablation Studies:
  • Using only the class-discriminative signal ($g^{class}$) preserved semantics but reduced visual diversity.
  • Using only the domain-specific signal ($g^{domain}$) failed to encode fine-grained class features.
  • The combination of all three signals yielded the best results, confirming their complementary nature.
• In-Distribution Performance: SGS maintained ID performance comparable to the baseline methods, proving that improving OOD robustness does not come at the cost of ID accuracy.

5. Significance and Claims

The paper claims to provide the first effective approach to domain-generalizable dataset distillation. Its significance lies in:

• Addressing a Critical Gap: It tackles the vulnerability of distilled datasets to domain shifts, a frequent challenge in real-world deployment that has been largely unexplored in DD literature.
• Efficiency: By integrating generalization directly into the distillation process via spectral surgery, it avoids the computational penalties associated with post-hoc augmentation strategies.
• Theoretical Insight: It establishes that cross-domain gradient agreement in the spectral domain is a reliable proxy for identifying class-invariant features, offering a new perspective on how to decompose information in compressed datasets.

The authors note that while SGS is currently validated within the Distribution Matching framework, the underlying principle of spectral decomposition via cross-domain agreement is conceptually applicable to other DD paradigms, such as Gradient Matching and Trajectory Matching, which they leave for future work.