Technical Summary: Label-Efficient Dataset Pruning via Semi-Supervised Pseudo-Labeling

Problem Statement

Deep learning models increasingly rely on massive datasets, leading to prohibitive storage and training costs. Dataset pruning addresses this by selecting an informative subset (a coreset) from a larger dataset. However, most existing pruning methods require fully labeled data to estimate example importance, which is impractical in real-world scenarios where annotation is costly and unlabeled data is abundant.

While recent "label-free" pruning methods attempt to bypass this by using features from pretrained models to estimate example difficulty, they suffer from a critical limitation: their reliability depends on the alignment between the pretrained feature distribution and the target dataset. When the target data differs significantly from the pretraining distribution (e.g., domain-specific datasets, image-corrupted variants, or long-tailed distributions), feature-based difficulty estimates often fail to capture the true semantic structure, leading to poor coreset selection.

Methodology: SemiPrune

The authors propose SemiPrune, a label-efficient dataset pruning framework that utilizes a small, randomly labeled subset to generate pseudo-labels for the entire dataset via Semi-Supervised Learning (SSL). The method operates in a two-stage pipeline:

1. SSL-based Pseudo-Label Generation:

   • A small subset of data (e.g., 10%) is randomly selected and labeled.
   • A model is trained on this labeled subset combined with the remaining unlabeled data using an SSL objective (specifically FixMatch).
   • The trained model assigns pseudo-labels to the unlabeled examples. These pseudo-labels are anchored to the target class space because the model is trained directly on the target distribution, unlike methods relying on fixed pretrained encoders.
   • The initial labeled examples (with ground-truth labels) and the pseudo-labeled examples form a unified training pool.

2. Supervised Pruning on Pseudo-Labeled Data:

   • A supervised model is trained from scratch on the pseudo-labeled training pool.
   • Example-wise importance scores are computed based on the training dynamics of this model.
   • The authors adapt two existing supervised pruning criteria to work with pseudo-labels:
     • Semi-AUM: Uses the Area Under the Margin (AUM) score with a double-end pruning strategy to select examples from the middle of the difficulty distribution.
     • Semi-DUAL: Uses the DUAL score (combining prediction difficulty and uncertainty) with Beta sampling to adjust the sampling distribution based on the pruning ratio.

Key Contributions and Findings

1. Limitations of Deep-Clustering-Based Pseudo-Labeling

The paper identifies that methods like ELFS, which rely on deep clustering of fixed pretrained features (e.g., DINO), are brittle in three specific scenarios:

• Domain Mismatch: Pretrained features may group examples by superficial visual similarity rather than semantic class (e.g., in Food-101), leading to noisy pseudo-labels.
• Image Corruption: Corruptions distort the feature space, making pretrained representations less separable and degrading clustering boundaries.
• Long-Tailed Distributions: Class imbalance causes clustering algorithms to fragment majority classes and merge minority classes, failing to preserve true class proportions.

In contrast, SSL-based pseudo-labeling, trained directly on the target data, consistently produces higher-quality pseudo-labels in these scenarios, as evidenced by higher accuracy, NMI, and ARI metrics compared to deep clustering.

2. Experimental Performance

The authors evaluate SemiPrune on diverse datasets, including domain-specific (Food-101, SUN397), corrupted (CIFAR-100-C, Tiny-ImageNet-C), long-tailed (Caltech-101, CIFAR-100-LT), and standard benchmarks (CIFAR-10/100, ImageNet-1K).

• Challenging Settings: In domain-specific, corrupted, and long-tailed settings, SemiPrune achieves state-of-the-art performance among methods that use either no labels or only a small fraction of labeled data. It significantly outperforms label-free baselines (Prototypicality, ZCore, ELFS) and few-label baselines (Score Extrapolation).
• Standard Benchmarks: On standard benchmarks where pretrained features are well-aligned, SemiPrune remains competitive, often matching or slightly exceeding the performance of fully supervised pruning baselines and label-free methods.
• Label Efficiency: The method demonstrates that a very small label budget (e.g., 1% to 5%) is sufficient to surpass strong label-free baselines, suggesting that minimal target-domain annotation is more effective than relying entirely on fixed pretrained representations or self-supervised encoders trained from scratch.

3. Ablation Studies

• Label Budget: Performance scales with the initial label budget, but even at 1%, SemiPrune outperforms ELFS with a self-trained encoder.
• Scoring Metrics: The advantage of SSL-based pseudo-labels persists across different scoring metrics (AUM, DUAL, EL2N, Forgetting), particularly in domain-mismatched settings.

Significance and Claims

The paper claims that SemiPrune effectively bridges the gap between fully supervised pruning (which requires full annotation) and label-free pruning (which relies on potentially misaligned pretrained features). By leveraging semi-supervised learning to generate pseudo-labels directly from the target dataset, the method:

• Captures the target distribution more accurately than fixed pretrained features.
• Provides more reliable signals for difficulty estimation and coreset selection, especially in non-ideal data conditions (domain shift, corruption, imbalance).
• Offers a practical, cost-efficient alternative to full annotation, requiring only a small fraction of labeled data to achieve performance comparable to or better than fully supervised methods in specific challenging regimes.

The authors note that the framework is not tied to FixMatch and can incorporate any SSL algorithm that provides reliable pseudo-labels. They also suggest that future work could explore incorporating active learning to select more informative samples for the initial annotation budget.