Technical Summary: Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weights

Problem Statement

Large-scale web-crawled datasets, which are fundamental to training modern multimodal models (e.g., CLIP, SigLIP, AIMv2), inherently contain noise, bias, and irrelevant information. Current data selection strategies suffer from significant limitations:

• Model-free methods (e.g., hand-crafted heuristics, semantic similarity to downstream datasets) often require costly trial-and-error, introduce unwanted data dependencies, and lack the granularity to assess individual sample utility.
• Model-based methods (e.g., specialized filtering networks, loss-based scoring, influence functions) increase pipeline complexity and are computationally expensive at scale. Influence functions, in particular, require expensive gradient computations over clean validation sets, which are difficult to curate and maintain.

There is a need for a simpler, dataset-agnostic, and compute-efficient mechanism to identify high-value data without relying on expensive validation splits or complex auxiliary networks.

Methodology: Mimic Score and Grad-Mimic

The authors propose a new data-quality metric called the Mimic Score and a two-stage framework named Grad-Mimic.

1. The Mimic Score

The core insight is that samples whose gradients pull the model in undesirable directions should be down-weighted. To define a "desirable" direction, the method leverages weight-space geometry from a pre-trained reference model ($\theta_{ref}$), which is assumed to reside in a more optimal region of the weight space than the current model ($\theta_t$).

For a given sample $s_i$ at training step $t$, the Mimic Score ($m_{i,t}$) is calculated by measuring the alignment between the sample's negative gradient ($-g_{i,t}$) and the vector pointing from the current weights to the reference weights ($v_t = \theta_{ref} - \theta_t$).

The score is computed as the projection length of the negative gradient onto the target vector:
$$\text{Mimic\_Score}(s_{i,t}) = m_{i,t} = \frac{\langle -g_{i,t}, v_t \rangle}{\|v_t\|}$$

This approach avoids additional forward passes or validation set computations, relying instead on the discrepancy between model weights (a simple matrix subtraction).

2. The Grad-Mimic Framework

Grad-Mimic utilizes the Mimic Score in two stages:

• Stage 1: Online Batch Re-weighting
  During training, Mimic Scores are computed on the fly for samples in a mini-batch. These scores are normalized using a softmax function with a temperature parameter $\tau$. The model parameters are then updated by re-weighting the gradients:
  $$\theta_{t+1} := \theta_t - \eta \sum_{i=1}^{b} \tilde{m}_{i,t} \cdot g_{i,t}$$
  where $\tilde{m}_{i,t}$ is the normalized score. This prioritizes samples that align with the reference model's optimal direction and down-weights those that do not.

• Stage 2: Offline Sample Selection
  After training, the Mimic Scores are aggregated across training steps to estimate overall sample utility. Continuous scores are binarized into retain/discard decisions using methods like thresholding, 1D clustering (k-means/GMM), or top-k percent selection. To handle noise in single-step assessments, the framework employs weak supervision techniques (specifically the Snorkel framework) to aggregate these binary votes into a probabilistic consensus, creating a robust ensemble filter for constructing smaller, higher-quality datasets.

Key Contributions

1. A New Data-Quality Metric (Mimic Score): The authors distill weight-space geometry into data-quality insights by exploiting the "access asymmetry" in AI (where pre-trained weights are public but curated datasets are not). It quantifies sample utility based on directional alignment with a reference model.
2. An Efficient Data Selection Framework (Grad-Mimic): A two-stage system that accelerates convergence via online re-weighting and automates effective data selection via offline aggregation.
3. Substantial Computational Gains: Unlike influence function-based methods or loss-difference approaches (e.g., Rho-Loss), Mimic Score computation incurs minimal runtime overhead and reduced GPU memory usage, as it operates on last-layer weight differences rather than full model inferences or Hessian approximations.
4. Reliable Ensemble Filtering: Aggregated Mimic Scores provide robust estimates of dataset quality and model performance gains, effectively strengthening existing selection strategies.
5. Extensive Ablation Studies: The paper validates the framework's generalization across diverse design choices (temperature, layer selection) and demonstrates robustness to the quality of the reference model.

Experimental Results

The authors evaluated Grad-Mimic across controlled mislabeled sample detection and large-scale web-crawled data curation (DataComp).

• Mislabeled Sample Detection: On six image datasets (e.g., CIFAR-100, Flowers102) with injected label noise (up to 60%), Grad-Mimic consistently outperformed baselines including Mini-batch SGD, GraNd, AGRA, Grad-Match, and Rho-Loss.

  • Performance: It achieved the highest average testing accuracy across all noise levels.
  • Detection: The aggregated Mimic Scores accurately identified mislabeled samples, achieving F1-scores above 95%.
  • Correlation: The Mimic Score-based retention rate showed a Pearson correlation of 0.903 with overall dataset quality.
  • Efficiency: Grad-Mimic reduced computational overheads by 2.6x compared to competing model-based methods.

• Large-Scale Data Curation (CLIP Training):

  • Convergence: When training CLIP models from scratch on DataComp scales (10M+ and 100M+ samples), Grad-Mimic accelerated convergence, requiring 20.7% fewer training steps to reach the same performance level as vanilla training.
  • Filtering: Mimic Score-based filters augmented existing CLIP Score filters. By removing low-value samples (bottom 30% by Mimic Score), the authors achieved improved model performance with 4.7 million fewer training samples in the medium-scale dataset.
  • Quality Estimation: The average Mimic Score of curated datasets correlated strongly (Pearson 0.958) with downstream performance gains.

• Robustness: The method remained effective even when the reference model was trained on significantly less data (372x fewer samples) or corrupted with noise, outperforming standard training in degraded scenarios.

Significance and Claims

The paper claims that Grad-Mimic offers a practical solution to the data selection bottleneck by leveraging the increasing availability of high-quality pre-trained weights. Its significance lies in:

• Decoupling from Data Dependencies: It removes the need for expensive, clean validation sets required by influence function methods.
• Scalability: By relying on simple weight-space geometry (last-layer subtraction) rather than complex inference or Hessian calculations, it scales efficiently to million-sample datasets.
• Automation: It automates the identification of low-value samples (e.g., short captions, misaligned images) that typically require human-designed heuristics.
• General Applicability: The framework is shown to be effective across different modalities (image-text, text-only) and training regimes (linear probing, full fine-tuning, training from scratch).

The authors conclude that while a high-performing reference model is ideal, the method is robust enough to function with weaker proxies, making it a flexible tool for improving data efficiency and model performance in the modern AI landscape.