🤖 machine learning

Data Pruning: Redundant, Problematic, and Interdependent Samples

This paper empirically demonstrates that the effectiveness of popular data pruning methods is critically dependent on dataset redundancy, the absence of problematic samples, and sample interdependence, revealing that these methods often fail under significant label noise.

Original authors: Leon Freese, Marthinus W. Theunissen

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Leon Freese, Marthinus W. Theunissen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to create the perfect soup. You have a massive pot filled with thousands of ingredients (your data). You want to know: Do you need all of them to make a delicious soup, or can you throw some away and still get the same great taste?

This paper is about "Data Pruning," which is basically the act of throwing away the "less important" ingredients from your training pot to make the recipe more efficient. The researchers wanted to test two popular ways of deciding which ingredients to toss out.

Here is what they found, explained simply:

1. The "Garbage In, Garbage Out" Problem

The researchers tested their methods on two types of pots:

Clean Pots: Where every ingredient is labeled correctly (e.g., "this is a carrot").
Noisy Pots: Where someone sneaked in some wrong labels (e.g., labeling a potato as a carrot).

The Big Surprise: The two popular methods they tested worked okay in the clean pots, but they completely failed in the noisy pots. When there was a lot of wrong information, these methods didn't just fail to improve the soup; they made it inedible.

2. The Three Hidden Traps

The paper argues that deciding what to throw away isn't as simple as "keep the best, toss the rest." It depends on three tricky factors:

Redundancy (The Duplicate Ingredients): Imagine you have 1,000 identical carrots. If you throw away 900 of them, your soup still tastes the same. The data is "redundant." The researchers found that you can throw away a huge chunk of data (up to 90% in some cases) without hurting the model, simply because there were so many copies of the same thing to begin with.
Problematic Samples (The Rotten Apples): These are the mislabeled items. The popular methods tried to keep the "good" samples and throw away the "bad" ones. But in a noisy pot, the methods got confused and ended up keeping the rotten apples while throwing away the good ones.
Interdependence (The Teamwork Effect): This is the most interesting part. The paper shows that an ingredient's value depends on who else is in the pot. A carrot might seem "unimportant" if you already have 1,000 other carrots, but if you only have 5 ingredients left, that same carrot becomes vital. The popular methods didn't understand this teamwork; they judged ingredients in isolation.

3. The "Reverse" Trick

Here is the wildest discovery:
When the researchers had a very noisy dataset, the standard methods (which try to keep the "best" samples) failed miserably. However, when they reversed the list—meaning they threw away the "best" samples first and kept the "worst" ones—the soup actually tasted better.

Why? Because in a noisy dataset, the "best" samples according to the algorithm were actually the ones that were confusing the model. By flipping the script and keeping the "hard" or "weird" samples, the model learned to ignore the noise better.

4. The Random Baseline

The researchers also tested a "dumb" method: just throwing away ingredients completely at random.

In the middle: The smart methods were slightly better than the random method.
At the extreme: When they kept only a tiny, tiny amount of ingredients, the random method actually won.
The Lesson: The "smart" methods were too aggressive. They threw away too many "duplicates" (redundant data) that were actually needed to help the model generalize when the dataset got very small. The random method kept a weird mix of duplicates and unique items, which happened to work better in the extreme cases.

The Bottom Line

The paper concludes that we can't just look at a single piece of data and say, "You are important, stay," or "You are useless, go."

Redundancy means we can throw away a lot of data without worry.
Noise breaks the standard rules for deciding what to keep.
Context matters: A sample's value changes depending on what other samples are around it.

In short, the "smart" ways of cleaning data are currently too fragile. They work well when everything is perfect, but they break down when the data is messy or when you try to cut the dataset down to the absolute bare minimum. Sometimes, a little bit of randomness or even doing the exact opposite of what the algorithm suggests works better.

Technical Summary: Data Pruning: Redundant, Problematic, and Interdependent Samples

Problem Statement

Deep learning model performance relies on both the quantity and quality of training data. While data pruning aims to reduce dataset size by retaining only the most "important" samples to maintain or improve performance, existing literature often assumes a clear definition of sample importance. This paper investigates the efficacy of popular data pruning methods under varying conditions, specifically focusing on how redundancy, problematic samples (label noise), and interdependence between samples influence the ranking and subsequent removal of training data. The authors identify a critical gap: while previous works suggest specific scoring methods for pruning, their behavior in the presence of significant label noise and across the full spectrum of pruning (from moderate to near-total removal) remains under-explored.

Methodology

The authors employ a standard data pruning evaluation framework involving three steps: ranking samples, removing a subset based on that ranking, and retraining the model to evaluate test performance.

Datasets: Experiments were conducted on three multiclass classification datasets: a synthetic dataset (10 classes, isotropic Gaussian distributions), MNIST, and CIFAR-10.
Models: Multilayer Perceptrons (MLP) were used for the synthetic and MNIST datasets, while VGG-16 was used for CIFAR-10. All models utilized the Adam optimizer, batch normalization, and ReLU activations.
Scoring Methods: Two established scoring methods were evaluated:
1. Sample Forgetting Score (Toneva et al. [16]): Counts the number of times a model "forgets" a sample (correctly classifies at $t-1$ , misclassifies at $t$ ) during training.
2. Error L2-Norm (EL2N) Score (Paul et al. [6]): Calculates the average Euclidean norm of the error vector during early training stages.
Experimental Protocol:
- Pruning Strategy: Samples were ranked in ascending order of their scores (removing low-scoring/"unimportant" samples first) and in descending order (removing high-scoring samples first).
- Noise Injection: For specific experiments, symmetric label noise was applied to 30% of the training set.
- Baselines: A random pruning baseline and an "Oracle" baseline (which removes all noisy labels first) were used for comparison.
- Metric: The primary output is a $\psi$ -curve, plotting test set accuracy against the fraction of the training set pruned.

Key Results

1. Redundancy and Moderate Pruning

On clean datasets, the authors found that significant redundancy exists. Approximately 40% of the synthetic and CIFAR-10 training sets, and 90% of the MNIST set, could be pruned with negligible performance loss.

Observation: Under moderate pruning, the two scoring methods offered only marginal gains (approx. 2% for CIFAR-10) over a random baseline.
Extreme Pruning Anomaly: When pruning to extreme levels (keeping only the "most important" samples), the random baseline consistently outperformed the scoring methods. The authors hypothesize that samples deemed "redundant" at large dataset sizes are actually necessary for generalization when the dataset is small. The scoring methods excessively removed these samples, whereas the random baseline retained a necessary mixture of "easy" and "hard" samples.

2. Impact of Label Noise

In the presence of 30% label noise, the standard application of Sample Forgetting and EL2N scores failed catastrophically.

Failure Mode: Both methods tended to assign high scores (ranking them as "important" to keep) to label-corrupted samples. Consequently, when pruning in ascending order, the noisy samples were retained while clean samples were removed, leading to a sharp drop in model performance.
Ranking Stability: Clean samples maintained their relative ranking positions even after noise was introduced, whereas corrupted samples shifted significantly toward the top of the ranking (high scores).

3. The "Reverse Ranking" Phenomenon

A counter-intuitive finding emerged when the ranking order was reversed (pruning in descending order of score):

Noisy Data: Reversing the order allowed the model to prune the high-scoring noisy samples first. This resulted in model performance that matched or approached the Oracle baseline (which knows the location of all noise).
Extreme Pruning: Even on clean data, reversing the ranking improved performance at extreme pruning levels, suggesting that the "importance" of a sample is not intrinsic but dependent on the presence of other samples in the dataset.

Key Contributions

The paper contributes the following insights to the field of data pruning:

Context-Dependent Importance: The definition of an "important" sample is not absolute; it is heavily confounded by dataset redundancy and the presence of contradictory (noisy) samples.
Failure of Standard Metrics under Noise: The authors demonstrate that popular scoring methods (Sample Forgetting and EL2N) fail to identify and remove label noise when applied in their standard ascending-pruning configuration.
Interdependence of Samples: The study highlights that the contribution of a specific sample to model performance is interdependent on the rest of the dataset. Removing "redundant" samples can be detrimental when the dataset size becomes small.
Simple Remediation: The authors show that simply reversing the ranking order of these scoring methods can drastically improve performance in noisy environments and extreme pruning scenarios, effectively turning a failure mode into a success.

Significance and Limitations

The authors conclude that the success of data pruning is distinctly affected by redundancy, problematic samples, and interdependence. They argue that the order in which samples should be pruned is less clear in the literature than often assumed.

Limitations acknowledged by the authors:

Generalizability: It is undetermined if these findings hold for larger datasets like ImageNet.
Noise Type: The use of artificial, symmetric label noise may present an overly optimistic view compared to real-world, complex data corruption.
Domain Scope: Experiments were limited to image classification; applicability to other domains (e.g., time-series regression) remains uncertain.

The paper calls for future work to incorporate sample interdependence into ranking processes and to develop scoring methods that are robust to various types of problematic samples.