💻 computer science

Condensing Large-Scale Datasets Directly with Minimal Information Loss

The paper introduces CIM, a novel metric-driven framework that eliminates the information-loss-inducing dual-compression paradigm of existing dataset distillation methods to achieve state-of-the-art performance on large-scale datasets like ImageNet-1K with minimal computational overhead.

Original authors: Xinyi Shang, Peng Sun, Bei Shi, Zixuan Wang, Tao Lin

Published 2026-07-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinyi Shang, Peng Sun, Bei Shi, Zixuan Wang, Tao Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library containing millions of books (a huge dataset like ImageNet). You want to teach a student (an AI model) everything in that library, but you don't have the time, space, or money to let them read every single book.

The Old Way: The "Summarize, Rewrite, and Guess" Problem

Previous methods tried to solve this by using a three-step process that the authors call "Squeeze, Recover, and Relabel."

Squeeze: They tried to cram all the knowledge from the millions of books into a single, tiny, compressed summary (a pre-trained model).
Recover: Then, they tried to take that compressed summary and "un-crumple" it back into a few new, synthetic pictures.
Relabel: Finally, they asked the computer that did the crumpling to look at these new pictures and guess what labels (names) they should have.

The Paper's Discovery: The "Blurry Photo" Effect

The authors of this paper, CIM, discovered a major flaw in this old method. They realized that the "Squeeze" and "Recover" steps act like a terrible photocopy machine.

The Information Leak: When you squeeze data into a model and then try to pull it back out as an image, you lose a lot of detail. It's like trying to recreate a high-definition painting just by describing it to someone who has to paint it from memory. The result is a blurry, distorted mess.
The Broken Labeler: Because the new images are so distorted (they look different from the real ones), the computer used to "Relabel" them gets confused. It's like asking a librarian who only knows the original books to label a blurry photocopy. The librarian guesses wrong, giving the student bad labels. This ruins the learning process.

The New Solution: CIM (The "Direct Copy-Paste" Approach)

Instead of crumpling the data up and trying to un-crumple it, the authors propose a new method called CIM.

Think of CIM as a smart collage maker.

Pick the Best Pages: Instead of trying to summarize the whole library, CIM first picks a small, perfect set of pages from the original books that represent the most important information.
Direct Alignment: Instead of trying to "recover" an image from a model, CIM directly compares the original pages with the new synthetic collage. It asks: "Does this new collage look and feel exactly like the original pages?"
Fixing the Gaps: It constantly tweaks the collage until the "information gap" is zero. It ensures that the texture (the grain of the paper) and the meaning (the story) are preserved perfectly.

Why This Matters

Because CIM skips the messy "un-crumpling" step, it doesn't lose information.

It's Faster: The authors claim they can condense the entire ImageNet-1K dataset (a massive collection of 1.2 million images) into a tiny set of synthetic images in just 80 minutes on a single powerful computer chip.
It's Smarter: The resulting synthetic images are so high-quality that when a student AI learns from them, it performs better than students trained on images made by previous methods.
It's Flexible: The method works well even when the student AI has a different "brain structure" (architecture) than the one that made the images.

In a Nutshell

The paper argues that trying to compress and then decompress data to create training images is like trying to make a perfect cake by first turning it into a liquid, freezing it, and then melting it back down—you lose the flavor. CIM skips the melting and freezing; it simply takes the best ingredients from the original cake and arranges them directly into a new, perfect, tiny cake that tastes exactly the same.

Technical Summary: Condensing Large-Scale Datasets Directly with Minimal Information Loss (CIM)

1. Problem Statement

Dataset distillation (DD) aims to compress massive training datasets into small synthetic sets while preserving generalization performance. Recent state-of-the-art (SOTA) approaches for large-scale datasets (e.g., ImageNet-1K) rely on information extraction pipelines (e.g., SRe2L), which employ a decoupled three-stage process: Squeeze (encoding data into a pre-trained model), Recover (inverting model parameters back to image space), and Relabel (re-assigning labels using the pre-trained model).

The authors identify two critical bottlenecks in these extraction-based methods:

Prohibitive Computational Overhead: The "Recover" stage requires complex, iterative optimization to invert model parameters, leading to high time and memory costs.
Poor Cross-Architecture Generalization: Distilled images often exhibit unrealistic textures or semantics tailored to specific networks.
The Root Cause: The authors argue that the implicit dual-compression process (Data $\to$ Model $\to$ Images) inherently induces severe information loss. This loss creates a distribution shift between the original and synthetic data. Consequently, the widely adopted Relabel strategy becomes unreliable; the pre-trained model, trained on the original distribution, acts as an "unreliable labeler" for the shifted synthetic samples, yielding sub-optimal labels and compromising performance.

2. Methodology: The CIM Framework

To address these issues, the authors propose CIM (Condensing Information with Minimal loss), a novel, metric-driven framework that abandons the flawed dual-compression paradigm. Instead of squeezing data into a model and recovering it, CIM directly minimizes the information gap between the original and distilled datasets.

Core Concepts

Effective Information Gap: The authors define the effective information of a sample as the distribution of features extracted by a group of observers (e.g., a pre-trained model under various transformations). They formulate the Pairwise Effective Information Gap ( $I_G$ ) between an original sample $x_i$ and a distilled sample $\tilde{x}_j$ using Kullback-Leibler (KL) divergence.
Theoretical Bound: Since direct KL estimation is intractable, the authors derive a theorem bounding the information gap by the expected squared Euclidean distance between feature vectors extracted by the observer group.
Optimization Objective: The distillation process minimizes the expected feature distance between a set of $N$ original images and a single distilled image $\tilde{x}_j$ , augmented via random crops and transformations:
$\mathcal{L} = \mathbb{E}_{(\xi_k \sim \mathcal{G})} \left\| \zeta_k \circ \phi_{\theta_T}(x_i) - \zeta_k \circ \phi_{\theta_T}(\tilde{x}_j^{(i)}) \right\|^2$
where $\phi_{\theta_T}$ is the pre-trained observer model and $\zeta_k$ represents transformations.

Key Technical Components

Direct Alignment: CIM explicitly aligns the data distributions of the original and distilled sets, ensuring high-fidelity information retention without the intermediate "Recover" step.
Semantic and Texture Balance: To prevent the loss of texture details (a common issue when aligning deep semantic features), CIM utilizes intermediate model features rather than final logits for the alignment objective.
Improved Relabeling: Unlike standard one-shot labeling, CIM employs a transformation-aware soft labeling strategy. It generates soft labels based on multiple transformed views of the distilled images, providing more diverse and robust knowledge injection.
Subset Selection: CIM adopts a subset selection mechanism (defaulting to RDED) to identify key samples from the original dataset that are most accurately relabeled by the pre-trained model, ensuring the distilled set starts with high-quality, representative data.

3. Key Contributions

Theoretical Insight: The paper rigorously reveals that the dual-compression process in existing extraction-based methods causes severe information loss and distribution shifts, which fundamentally undermine the efficacy of the Relabel strategy.
Novel Framework (CIM): A metric-driven approach that explicitly quantifies and minimizes the information gap between original and distilled datasets, eliminating the need for computationally expensive recovery processes.
State-of-the-Art Performance: Extensive experiments demonstrate that CIM establishes new SOTA across various scales and architectures.
- ImageNet-1K (IPC=10): Achieves 48.7% Top-1 accuracy on ResNet-18 in just 80 minutes on a single RTX-4090 GPU.
- Improvements: Outperforms previous SOTA methods like NRR-DD (+2.6%) and DELT (+2.9%).
- Generalization: Demonstrates superior cross-architecture generalization, maintaining high performance across diverse backbones (ResNet, ViT, MobileNet, etc.).
Efficiency: By avoiding the iterative recovery stage, CIM significantly reduces computational overhead and memory usage compared to optimization-based baselines, while maintaining or exceeding their performance.

4. Experimental Results

Small-Scale Datasets: CIM outperforms baselines (SRe2L, G-VBSM, RDED, etc.) on CIFAR-10 and CIFAR-100 across IPC settings (1, 10, 50) and various architectures (ConvNet, ResNet-18, ResNet-50).
Large-Scale Datasets: On Tiny-ImageNet and ImageNet-1K, CIM achieves the highest accuracy. Notably, on ImageNet-1K with IPC=10, it reaches 48.7% accuracy, a significant leap over the 46.1% of DELT and 45.8% of NRR-DD.
Cross-Architecture Generalization: When distilled on ResNet-18 and tested on unseen architectures (e.g., EfficientNet-B0, ViT-T/16), CIM consistently outperforms other methods, validating its ability to preserve both semantic and textural information.
Continual Learning: In a 5-step class-incremental learning setting, CIM significantly improves upon SRe2L, demonstrating the utility of its high-fidelity synthetic data for preventing catastrophic forgetting.
Ablation Studies: The authors verify that:
- Intermediate feature alignment (middle layers) yields the best balance between semantics and texture.
- Compressing 4 original images ( $N=4$ ) into one distilled image offers the optimal trade-off.
- The framework is robust to the number of compression iterations ( $M$ ) and subset selection strategies.

5. Significance and Claims

The paper claims that CIM represents a paradigm shift in dataset distillation. By moving away from the "Squeeze-Recover" heuristic, which inherently degrades information, CIM directly addresses the root cause of performance bottlenecks: information loss and distribution shift.

The authors emphasize that their approach:

Restores Reliability: By ensuring distributional proximity, it allows the Relabel strategy to function as intended, turning the pre-trained model back into a reliable labeler.
Scalability: It enables the distillation of massive datasets (ImageNet-1K) with unprecedented speed (80 minutes) and efficiency on a single GPU.
Generalizability: It produces synthetic datasets that are robust across different network architectures, solving a major limitation of previous extraction-based methods.

The work concludes that minimizing the information gap directly is the key to unlocking high-fidelity, efficient, and generalizable dataset distillation for large-scale applications.