💻 computer science

Dataset Distillation by Influence Matching

This paper introduces Influence Matching (Inf-Match), a dataset distillation method that learns compact synthetic datasets by aligning their final training outcomes with the full dataset via a novel, efficient, fully differentiable sample-level influence estimator, achieving state-of-the-art performance on both image classification and vision-language retrieval benchmarks.

Original authors: Haoru Tan, Wang Wang, Sitong Wu, Xiuzhe Wu, Yangtian Sun, Chirui Chang, Shaofeng Zhang, Xiaojuan Qi

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haoru Tan, Wang Wang, Sitong Wu, Xiuzhe Wu, Yangtian Sun, Chirui Chang, Shaofeng Zhang, Xiaojuan Qi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to recognize animals, but you have a library containing millions of books (images) to use as your textbook. While this massive library helps the robot learn perfectly, it's a nightmare to carry around, store, or send to other robots. This is the world of dataset distillation, a field of computer science dedicated to shrinking these giant libraries down into tiny, pocket-sized summaries that still teach the robot everything it needs to know.

For a long time, scientists tried to solve this by looking at the process of learning. They thought, "If we can make our tiny summary copy the exact steps, the mistakes, and the daily homework the robot did while studying the big library, then the robot will learn the same way." It was like trying to teach someone to play piano by forcing them to mimic every finger movement of a master, hoping the music would sound right. But sometimes, even if the finger movements match perfectly, the song sounds terrible. The big question researchers have been asking is: Can we skip the mimicry of the steps and instead focus on the result? Can we create a tiny summary that guarantees the robot ends up with the exact same "brain" (final knowledge) as if it had studied the whole library, regardless of how it got there?

This is exactly what the paper "Dataset Distillation by Influence Matching" (Inf-Match) tackles. The authors, a team from HKU, CUHK, and Stanford, argue that we should stop trying to copy the robot's daily training routine and start matching the final outcome. They introduce a clever new method called Influence Matching.

Think of the old methods as trying to copy a recipe by writing down every single chop, stir, and heat setting the chef used. If you miss one tiny stir, the cake might fail. The new method, Inf-Match, is more like asking: "If I remove this specific ingredient from the cake, how much does the taste change?" and "If I add this new ingredient, how does the flavor shift?" By measuring exactly how much each piece of data "pushes" the robot's brain in a specific direction, the team creates a tiny synthetic dataset that exerts the exact same total "push" as the massive original library.

To do this, they invented a special mathematical tool—a "influence estimator"—that acts like a super-fast calculator. Instead of needing to re-train the robot thousands of times to see what happens when you remove a picture (which would take forever), this tool uses a shortcut to predict the change instantly. It's like having a crystal ball that tells you, "If you swap this photo of a cat for a photo of a dog, the robot's brain will shift by exactly this much."

The results are impressive. When they tested this on standard image datasets like CIFAR-10, CIFAR-100, and the challenging Tiny-ImageNet, their method consistently beat the best previous techniques. For example, on the Tiny-ImageNet dataset with only 10 synthetic images per class, Inf-Match achieved an accuracy of 31.5%, which is a 4.7% improvement over the previous top method (NCFM). They also tested it on a vision-language task using the Flickr30K dataset (matching images to text), where their method outperformed strong baselines by 2.5% on average.

The paper suggests that by focusing on the final "influence" rather than the intermediate steps, we can create much more effective mini-datasets. The authors show that their approach works well across different types of neural networks and even scales up to complex tasks involving both images and text. While they don't claim to have solved every problem in AI, their experiments strongly suggest that matching the final outcome is a more reliable path to efficient data compression than trying to copy the training journey step-by-step. In short, they found a way to shrink the library without losing the story.

Technical Summary: Dataset Distillation by Influence Matching

Problem Statement

Modern deep learning relies on massive datasets, creating prohibitive costs for storage, transmission, and training. Dataset distillation addresses this by synthesizing a compact, highly informative dataset that encapsulates the knowledge of a much larger original dataset.

Existing approaches primarily rely on process alignment, where synthetic data is optimized to mimic intermediate training dynamics (e.g., per-step gradients or training trajectories) of the real data. While methods like Gradient Matching (GM) and Trajectory Matching (MTT) have shown success, they suffer from a fundamental limitation: matching intermediate processes does not guarantee matching the final outcome (the converged model parameters). This creates an optimization gap where synthetic data may perfectly reproduce training trajectories yet fail to achieve comparable downstream performance or generalization. Furthermore, existing methods for quantifying data influence often rely on computationally prohibitive inverse-Hessian computations and strict convexity assumptions that do not hold for deep neural networks.

Methodology: Influence Matching (Inf-Match)

The authors propose Influence Matching (Inf-Match), a framework that shifts the optimization objective from process alignment to outcome alignment. Instead of mimicking training steps, Inf-Match learns a synthetic set whose effect on the final converged parameters matches that of the full dataset.

1. Differentiable Sample Influence Estimator

The core innovation is a fully differentiable, sample-level influence estimator that quantifies how adding or removing data shifts the final model parameters.

Mechanism: The estimator unrolls the optimization dynamics (SGD trajectory) and applies a first-order Taylor approximation.
Efficiency: It avoids inverse-Hessian products and convexity assumptions, achieving linear-time complexity ( $O(p)$ where $p$ is the number of parameters) by approximating Hessian-gradient products via finite differences.
Theoretical Guarantees: The paper provides a theoretical upper bound on the estimation error, showing that the error scales polynomially with training steps $T$ (improving upon prior estimators with exponential error growth) and is controlled by the Lipschitz constant of the loss and gradient norms.

2. Optimization Objective

The distillation problem is formulated as minimizing the mismatch between the influence of the real dataset and the synthetic dataset.

Goal: Minimize the total parameter shift: $\| I_{-D} + I_{+S} \|$ , where $I_{-D}$ is the influence of removing the real dataset $D$ , and $I_{+S}$ is the influence of adding the synthetic dataset $S$ .
Formulation: By substituting the influence estimators, the objective becomes a differentiable loss function that aligns the final model parameters trained on $S$ with those trained on $D$ , without requiring actual retraining of the model on $S$ during the distillation loop.

3. Algorithm Pipeline

Initialization: The synthetic set $S$ is initialized with real images from $D$ (following Image-Per-Class, IPC, settings) and soft labels generated by a model trained on $D$ .
Trajectory Sampling: A base model is trained on $D$ to record a trajectory of checkpoints. During distillation, a subset of these checkpoints is sampled to estimate influence.
Optimization: The synthetic images and soft labels are updated via gradient descent to minimize the influence mismatch loss (Eq. 7). Mini-batches from both $D$ and $S$ are used to estimate Hessians and gradients efficiently.
Progressive Sampling: A time-step selection schedule is employed, sampling early checkpoints initially to capture basic patterns and shifting to later checkpoints to encode fine-grained structures.

Key Contributions

Outcome-Centric Paradigm: The paper redefines dataset distillation from heuristic process matching to rigorous outcome matching, directly targeting the final model state rather than intermediate dynamics.
Efficient Influence Estimator: It introduces a scalable, differentiable influence estimator that operates in linear time, removing the need for inverse-Hessian computations and convexity assumptions, making it applicable to deep neural networks.
State-of-the-Art Performance: The method establishes new benchmarks across standard classification and vision-language tasks, demonstrating that outcome alignment yields superior generalization compared to process alignment.

Experimental Results

The authors evaluated Inf-Match on image classification (CIFAR-10, CIFAR-100, Tiny-ImageNet) and vision-language tasks (Flickr30K).

Image Classification:
- Tiny-ImageNet (IPC=10): Inf-Match achieved 31.5% accuracy, a +4.7% improvement over the previous best (NCFM).
- CIFAR-100 (IPC=50): Achieved 57.4% accuracy, outperforming NCFM by 2.7% and DATM by 2.4%.
- Cross-Architecture Generalization: The synthetic data generated by Inf-Match consistently outperformed baselines (MTT, DATM) when tested on diverse architectures (ResNet, VGG, AlexNet), indicating robust generalization.
Vision-Language (Flickr30K):
- On image-to-text retrieval with 200 synthetic samples, Inf-Match achieved 7.4% Recall@1, surpassing DATM by 1.3%.
- With 1000 samples, it reached 16.4% on text-to-image retrieval, leading all baselines.
Ablation Studies: The study confirmed that the combination of real-data initialization, learnable soft labels, and progressive time-step sampling contributes synergistically to the final performance.
Feature Space Analysis: Visualizations indicate that unlike Distribution Matching (DM) methods which concentrate samples in high-density regions, Inf-Match generates samples that effectively occupy both high-density areas and distribution edges, leading to a more balanced representation.

Significance and Claims

The paper claims that Influence Matching represents a fundamental shift in dataset distillation. By directly quantifying and matching the influence of data on the final model parameters, the method bridges the optimization gap inherent in process-matching approaches.

The authors assert that their approach provides a principled way to distill data without relying on fragile heuristics or computationally intractable assumptions. The results suggest that matching the "outcome" (final parameter shift) is a more reliable objective for generating high-quality synthetic datasets, leading to state-of-the-art performance across diverse datasets and compression rates. The work positions outcome alignment as the next logical step for scalable and effective dataset distillation.