🤖 machine learning

Understanding Dataset Distillation via Spectral Filtering

This paper introduces UniDD, a spectral filtering framework that unifies diverse dataset distillation methods by analyzing their effects on frequency components, and proposes Curriculum Frequency Matching (CFM) to dynamically balance low- and high-frequency information, thereby achieving superior performance across various datasets.

Original authors: Deyu Bo, Songhua Liu, Xinchao Wang

Published 2026-03-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Deyu Bo, Songhua Liu, Xinchao Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library containing millions of books (the Real Dataset). You want to teach a student (the AI Model) everything they need to know, but you don't have time to read every single book to them. So, you decide to write a tiny, perfect "cheat sheet" (the Synthetic Dataset) that contains just a few pages but somehow holds all the essential knowledge.

This process is called Dataset Distillation.

The problem is, different researchers have been writing these cheat sheets using different rules. Some focus on the main ideas, others on the tiny details, and they often don't agree on which rule is best.

This paper introduces a new way of looking at the problem called UniDD (Universal Dataset Distillation) and a new method called CFM (Curriculum Frequency Matching). Here is how it works, explained simply:

1. The Big Idea: The "Sound Mixing Board" Analogy

The authors realized that all these different methods for making cheat sheets are actually doing the same thing, just with different settings on a sound mixing board.

The Data as Sound: Imagine your dataset is a complex song. It has deep, rumbling bass notes (low-frequency information like the general shape of a cat) and high-pitched, sharp treble notes (high-frequency information like the texture of fur or whiskers).
The Filter: Every method tries to match the "song" of the real books to the "song" of the cheat sheet.
- Low-Pass Filters (The Bass Players): Methods like DM and DC act like a filter that only lets the bass through. They are great at capturing the big picture (e.g., "this is a cat"), but the result looks blurry and lacks detail.
- High-Pass Filters (The Treble Players): Methods like MTT and FrePo act like a filter that only lets the high-pitched sounds through. They capture amazing details and textures, but the result might look chaotic or noisy without a clear shape.

The Discovery: The paper proves that every existing method is just a different way of turning the knobs on this mixing board to pick specific frequencies.

2. The Problem: The "One-Note" Limitation

The old methods had a major flaw: they were stuck on one setting.

If you used a "Bass" method, your cheat sheet was clear but blurry.
If you used a "Treble" method, your cheat sheet was sharp but messy.
None of them could do both at the same time because their "knobs" were glued in place.

3. The Solution: The "Curriculum" Approach (CFM)

The authors propose a new method called Curriculum Frequency Matching (CFM). Think of this as a smart teacher who knows how to teach a student effectively over time.

Instead of trying to teach the student everything at once, the teacher follows a schedule (Curriculum):

Early Days (The Basics): At the start of training, the teacher uses a Low-Pass Filter. They focus on the big picture. "Okay, student, first learn that this is a bird, not a car." This establishes the basic structure and consistency.
Middle Days (Adding Detail): As training progresses, the teacher slowly turns the knob to let in more High-Frequency information. "Now that you know it's a bird, let's look at the feathers and the beak."
Final Days (Polishing): By the end, the teacher has balanced both. The student has learned the general shape and the fine details.

This dynamic approach allows the AI to build a cheat sheet that is both structurally sound (not blurry) and rich in detail (not chaotic).

4. The Results: A Better Cheat Sheet

The authors tested this on various "libraries" (datasets like CIFAR and ImageNet).

The Old Way: The cheat sheets were either too blurry or too noisy.
The New Way (CFM): The cheat sheets were clear, sharp, and contained all the necessary information. When they used these cheat sheets to train new AI models, the models performed significantly better than those trained with old methods.

Summary

In short, this paper says:

"We figured out that all these different ways of compressing data are just different ways of filtering sound frequencies. The old methods were stuck listening to only the bass or only the treble. Our new method, CFM, acts like a smart DJ who starts with the bass and gradually mixes in the treble, creating the perfect playlist for the AI to learn from."

This makes the AI training process faster, more efficient, and much smarter.

1. Problem Statement

Dataset Distillation (DD) aims to compress large-scale real datasets into compact, synthetic datasets such that models trained on the synthetic data achieve performance comparable to those trained on the original data. While various DD methods exist (e.g., Gradient Matching, Trajectory Matching, Statistical Matching), they operate under disparate optimization objectives with little theoretical connection.

Key Gap: There is no unified framework explaining the relationship between these diverse objectives. Existing methods typically rely on fixed optimization strategies that capture only a single type of frequency information (either low-frequency global structures or high-frequency textures), limiting their adaptability and generalization capabilities.

2. Methodology

The paper proposes a two-part methodology: a theoretical unification framework (UniDD) and a novel algorithmic solution (Curriculum Frequency Matching - CFM).

A. UniDD: A Spectral Filtering Framework

The authors theoretically analyze representative DD methods (DM, DC, MTT, FrePo) and unify them under a spectral filtering framework.

Core Concept: DD objectives are interpreted as applying specific filter functions to the eigenvalues of the Feature-Feature Correlation (FFC) matrix ( $X^\top X$ ) to extract frequency components of the Feature-Label Correlation (FLC) matrix ( $X^\top Y$ ).
Mathematical Formulation: The general objective is formulated as:
$\min_{X_s} \| f(X^\top X)g(X^\top Y) - f(X_s^\top X_s)g(X_s^\top Y_s) \|_F^2$
Where $f(\cdot)$ is the filter function and $g(\cdot)$ is a binary function ( $I$ or $X^\top Y$ ).
Classification of Methods:
- Low-Frequency Matching (LFM): Methods like DM and DC act as low-pass filters. They capture principal components (large eigenvalues), resulting in synthetic images with coarse-grained colors and blurred textures but high intra-class consistency.
- High-Frequency Matching (HFM): Methods like MTT and FrePo act as high-pass filters. They emphasize small eigenvalues, capturing fine-grained textures and improving diversity, often at the cost of consistency.

B. CFM: Curriculum Frequency Matching

Recognizing that existing methods use fixed filters (learning only one frequency band), the authors propose CFM to dynamically adjust the filter during distillation.

Dynamic Filter Design: CFM uses a filter function $f(\lambda) = (\lambda + \beta)^{-1}$ $f (λ) = (λ + β)^{- 1}$ . The parameter $\beta$ $β$ controls the frequency band:
- Large $\beta$ $\rightarrow$ Low-pass behavior (Consistency).
- Small $\beta$ $\rightarrow$ High-pass behavior (Diversity).
Curriculum Strategy: Instead of fixing $\beta$ , CFM employs a scheduler to gradually decrease $\beta$ over training batches ( $b$ ):
$\beta_b = \beta^* \frac{1 + \cos(\pi b / B)}{2}$
This allows the model to first learn low-frequency global structures (ensuring stability) and progressively incorporate high-frequency details (enhancing diversity).
Implementation Details:
- FFC/FLC Calculation: Computed from feature maps of convolutional layers. To avoid numerical overflow from reshaping large feature maps, the authors use covariance matrices and mean representations as substitutes.
- Exponential Moving Updating (EMU): To handle mini-batch training (vs. full-batch theoretical assumption), the FFC and FLC statistics are updated using a moving average across batches.
- Loss Function: Combines a filter-matching loss, a signal-matching loss, and a standard classification loss.

3. Key Contributions

UniDD Framework: The first theoretical framework unifying diverse DD objectives as spectral filtering problems on FFC and FLC matrices, revealing that DD is fundamentally a frequency-matching problem.
Filter Characterization: A rigorous classification of existing methods into Low-Frequency (global/color) and High-Frequency (texture/diversity) matchers, explaining their respective strengths and weaknesses (e.g., why DM blurs images while MTT preserves texture).
CFM Algorithm: A novel DD method that overcomes the limitation of fixed filters by using a curriculum strategy to dynamically balance low- and high-frequency information, leading to superior synthetic data quality.
Comprehensive Validation: Extensive experiments demonstrating that CFM outperforms state-of-the-art baselines across small-scale (CIFAR-10/100) and large-scale (ImageNet-1K) datasets, with improved cross-architecture generalization.

4. Experimental Results

The authors evaluated CFM on CIFAR-10/100, Tiny-ImageNet, and ImageNet-1K using various backbones (ConvNet, ResNet-18/101).

Performance on CIFAR-10/100: CFM consistently outperformed advanced baselines (DM, DC, MTT, FrePo, SRe2L, G-VBSM).
- Example: On CIFAR-10 with 50 IPC and ResNet-18, CFM achieved 82.3% accuracy, significantly surpassing the next best (MTT at 71.7%).
Performance on Large-Scale Datasets:
- Tiny-ImageNet: CFM outperformed DWA by ~3% on average.
- ImageNet-1K: CFM achieved the best results among training-based methods for IPC=50 and IPC=100, surpassing SRe2L, CDA, and RDED.
Cross-Architecture Generalization: When trained on ResNet-18 and tested on diverse architectures (ResNet-50/101, DenseNet, RegNet, ConvNeXt), CFM maintained the highest accuracy, proving it encodes generalizable knowledge rather than overfitting to a specific architecture.
Efficiency: CFM demonstrated lower space overhead compared to G-VBSM and competitive time overhead compared to SRe2L.
Ablation Studies:
- Removing the filter-matching loss caused significant performance drops, confirming the importance of FFC matrix matching.
- Fixed filters (pure low-pass or pure high-pass) performed worse than the dynamic CFM strategy, validating the curriculum approach.

5. Significance

Theoretical Insight: The paper shifts the paradigm of DD from heuristic objective design to spectral analysis. It provides a principled explanation for why different methods behave differently (e.g., why some produce blurry images while others produce noisy ones).
Practical Impact: By introducing CFM, the paper offers a robust, generalizable solution for dataset distillation that works effectively even on large-scale, complex datasets like ImageNet-1K, where previous methods often struggle.
Future Direction: The framework opens avenues for applying spectral filtering concepts to unsupervised and multi-modal dataset distillation, moving beyond the current focus on supervised image classification.