🤖 machine learning

Benchmarking Instance-Dependent Label Noise with Controlled Corruptions

This paper introduces CILN, a benchmark generation framework that creates controlled instance-dependent label noise through explicit input corruptions, revealing that noise structure significantly impacts algorithm performance and exposing failure modes in popular noisy-label learning methods that traditional rater-fallibility benchmarks miss.

Original authors: Shadman Islam, Agustinus Kristiadi, Mostafa Milani

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shadman Islam, Agustinus Kristiadi, Mostafa Milani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a robot to recognize animals in photos. You give it a picture of a dog, but the label says "cat." The robot gets confused. This is called label noise.

For a long time, researchers tested how well robots could handle these mistakes by just randomly flipping labels. It's like saying, "Okay, let's pretend 10% of the dog photos are actually cats, and 10% of the cat photos are actually dogs." But in the real world, mistakes aren't random. They happen because a picture is blurry, the lighting is bad, or the animal looks weird. A robot is more likely to mistake a blurry dog for a cat than a clear, sharp dog.

This paper introduces a new way to test robots, called CILN (Corruption-Induced Label Noise). Instead of just randomly swapping labels, the researchers deliberately "break" the pictures first, then see how the robots react.

Here is a simple breakdown of how it works and what they found:

1. The Old Way vs. The New Way

The Old Way (The "Bad Teacher"): Imagine a teacher who is just tired and makes random mistakes. They look at a clear picture of a dog and accidentally write "cat" on the paper. The mistake has nothing to do with the picture itself; it's just the teacher having a bad day. Existing tests used this method.
The New Way (The "Damaged Photo"): In the CILN experiment, the researchers don't just change the label; they ruin the photo first. They take a clear picture of a dog and add static, blur it, or cut off its ears. Now, the picture actually looks a bit like a cat. When they ask a group of "voters" (a pool of different AI models) to label this damaged photo, they disagree. Some say "dog," some say "cat." The researchers use this disagreement to create the "noisy" label.

The Key Difference: In the old way, the mistake is hidden inside the teacher's brain. In the new way, the mistake is visible in the broken photo. You know exactly why the robot got confused: because the image was blurry, not because the teacher was careless.

2. How They Did It (The "Controlled Breakage")

The researchers took clean datasets (like pictures of dogs, cats, and birds) and applied specific "corruptions":

Blur: Making the image fuzzy.
Noise: Adding grainy static.
Weather: Adding fog or snow.
Geometry: Stretching or rotating the image.

They did this at different levels of severity, from "mildly annoying" to "completely unrecognizable." Then, they asked a diverse team of AI models to guess what the damaged image was. If the team disagreed, that image became part of their "noisy" test set.

3. What They Discovered

The researchers ran these tests on three different types of data: pictures of animals (CIFAR-10), handwritten numbers (MNIST), and adult income data (Adult).

Mistakes Have a Pattern: They found that when you break a picture in a specific way, the mistakes aren't random. For example, if you add "impulse noise" (like salt-and-pepper static) to a handwritten number "5," the gaps fill in, and it starts to look like an "8." The robots consistently made the same mistake (5 becoming 8) because the damage made it look like an 8. This is different from random noise, where a 5 might randomly become a 2, 7, or 9.
The "Attractor" Effect: Some types of damage act like a magnet. No matter what the original picture was, if you damage it enough, the robots all start guessing the same wrong answer. For instance, heavy blur might make almost everything look like a "cat" or a "frog."
The "Small Loss" Trap: Many modern AI methods try to fix mistakes by assuming that if a robot is sure about a label (low "loss"), it's probably right. But the CILN tests showed a trap: If a picture is damaged enough to look like a cat, the robot becomes very sure it's a cat, even though it's actually a dog. The robot thinks it's right because the picture is broken, not because it's smart. This caused popular AI methods to fail in ways they never failed on the old "random mistake" tests.

4. Why This Matters

The paper argues that we can't just measure how many mistakes a dataset has (the "noise rate"). We also need to measure what kind of mistakes they are.

Analogy: Imagine testing a car's brakes.
- Old Test: You randomly press the brake pedal while the car is driving. Sometimes it stops, sometimes it doesn't.
- New Test (CILN): You put the car on a road covered in ice, then test the brakes.
- Result: The car might stop fine on dry pavement (low noise), but fail completely on ice (high noise). The old test wouldn't tell you the car struggles on ice. Similarly, the old AI tests didn't tell us that AI struggles when the data itself is damaged, only when the labels are wrong.

Summary

The paper presents CILN, a new tool that creates "broken" data to test AI. It shows that when data is damaged in specific ways (like blurring or noise), AI makes predictable, structured mistakes that are very different from random errors. This reveals weaknesses in current AI safety methods that were previously hidden, proving that how a mistake happens is just as important as how many mistakes there are.

Technical Summary: Benchmarking Instance-Dependent Label Noise with Controlled Corruptions (CILN)

Problem Statement

Modern machine learning systems rely on large labeled datasets, yet labels are frequently imperfect due to annotation errors, ambiguous examples, or weak supervision. While early research focused on symmetric and class-conditional noise (CCN), practical settings often exhibit Instance-Dependent Label Noise (IDN), where the probability of a labeling error depends on both the true class and the specific characteristics of the input instance.

Existing synthetic IDN benchmarks typically generate noise through two mechanisms:

Feature-space transformations: Applying latent transformations to increase corruption probabilities for specific examples.
Pseudo-labeling: Using diverse classifiers as synthetic annotators, where disagreement induces uncertainty.

The Limitation: In both approaches, the source of ambiguity remains implicit. In feature-space methods, the transformations are often hard to interpret. In pseudo-labeling, noise is controlled by modifying annotator properties (e.g., training procedures), making it difficult to isolate specific ambiguity mechanisms (e.g., blur, occlusion, missing data) or trace label errors to specific input changes. Consequently, existing benchmarks offer limited support for systematically studying how specific data degradation mechanisms affect classifier disagreement and downstream learning behavior.

Methodology: The CILN Framework

The authors introduce CILN (Corruption-Induced Label Noise), a benchmark-generation framework that creates IDN through controlled input corruptions. Unlike previous methods that alter the labeling process, CILN alters the input data itself while preserving semantic content.

Core Components

Controlled Corruptions: The framework applies specific corruption types ( $c$ $c$ ) at varying severity levels ( $\ell$ $ℓ$ ) to clean instances.
- Image Data: Utilizes corruption families from CIFAR-C and MNIST-C, including noise (Gaussian, shot, impulse), blur (defocus, glass, motion), weather (fog, frost, snow), geometric transformations, and digital artifacts (brightness, contrast, JPEG).
- Tabular Data: Utilizes corruption operators inspired by the Jenga benchmark, including missing data mechanisms (MCAR, MAR, MNAR) and value corruptions (noise injection, scaling).
Voter Pool: A diverse pool of $M$ models (e.g., ResNets, Vision Transformers, XGBoost) evaluates the corrupted instances.
Label Generation:
- Each voter produces a predictive distribution $p^{(m)}(y|x)$ .
- The benchmark releases an aggregated label distribution $\bar{p}_i$ (arithmetic mean of voter predictions) and individual voter distributions.
- For hard-label experiments, a single voter is sampled uniformly at random to provide a training label, mimicking the PL-IDN protocol.
Clean-Start Configuration: An optional mode filters the dataset to retain only instances unanimously classified correctly by all voters before corruption. This isolates corruption-induced uncertainty from pre-existing ambiguity in the original data.

Benchmark Construction

The framework was instantiated on three datasets: CIFAR-10, MNIST, and Adult.

CIFAR-10: 45 settings (15 corruption types $\times$ 3 severity levels).
MNIST: 30 released settings (11 continuous-severity types + 4 structural types).
Adult: 15 settings (5 corruption types $\times$ 3 severity levels).
Total: 90 benchmark settings spanning image and tabular domains.

Key Contributions

Mechanism-Controlled IDN Generation: CILN provides a framework where the source (corruption type) and severity of ambiguity are explicit and controllable, establishing a direct link between input degradation and label uncertainty.
Genuine Instance Dependence: The authors demonstrate that the framework generates genuine IDN where the degree of instance dependence can be systematically tuned via corruption type and severity.
Human-Like Uncertainty: Under appropriate settings, the label distributions generated by CILN are statistically closer to human annotation uncertainty (measured against CIFAR-10-H) than existing synthetic IDN benchmarks (specifically PL-IDN).
Exposure of Failure Modes: The benchmark reveals that state-of-the-art noisy-label learning methods (Co-Teaching, DivideMix) exhibit failure modes under high-severity corruption that are not observed under comparable noise rates generated by rater-fallibility models.

Experimental Results

1. Strength of Instance Dependence

The authors introduce Noise Transition Heterogeneity ( $n_{th}$ ) to measure instance dependence.

Findings: All CILN settings produce positive $n_{th}$ values, confirming genuine IDN.
Control: The magnitude of $n_{th}$ varies significantly across corruption families and severity levels. For many families, $n_{th}$ increases with severity but eventually saturates or decreases at high severity as prediction patterns converge.
Comparison: At high noise rates, representative CILN settings match or exceed the instance dependence of PL-IDN.

2. Similarity to Human Annotation Uncertainty

Using Total Variation (TV) distance against CIFAR-10-H:

Findings: CILN is consistently closer to human uncertainty than PL-IDN across all noise levels.
Corruption Specificity: Mild corruptions (particularly Weather and Digital families like frost, snow, fog, brightness, JPEG) produce label distributions most similar to human disagreement. Severe corruptions tend to diverge further from human patterns but still outperform PL-IDN.
PCA Analysis: The ten CILN settings closest to CIFAR-10-H in PCA space were all low-severity Weather or Digital corruptions; none of the PL-IDN releases appeared in the top ten.

3. Downstream Learning Behavior

The authors evaluated Empirical Risk Minimization (ERM), Co-Teaching, and DivideMix on CIFAR-10.

Low/Medium Noise: CILN and PL-IDN yield similar performance trends.
High Noise Divergence: At high noise rates, Co-Teaching and DivideMix suffer substantial performance degradation on specific CILN settings (e.g., high-severity contrast, pixelate, glass-blur), whereas they remain relatively robust on PL-IDN-H.
Mechanism of Failure:
- Attractor Classes: Certain corruptions create "attractor classes" (e.g., frog, cat, truck) where mislabeled examples from many true classes are pulled toward a single incorrect class.
- Loss Distribution Collapse: In these settings, the incorrect label becomes consistent with the corrupted input. Consequently, noisy examples do not necessarily produce higher training losses than clean examples. This violates the small-loss assumption (that clean examples have lower loss than noisy ones) which Co-Teaching and DivideMix rely on for sample selection.
- Clean-Start Validation: The failure modes persist even in the Clean-Start variant (CILN-C), confirming that the difficulty arises from the corruption structure itself, not pre-existing data ambiguity.

Significance and Claims

The paper argues that noise structure, not just noise rate, is a critical factor in benchmark difficulty and algorithm behavior.

Complementarity: CILN complements existing rater-fallibility benchmarks (like PL-IDN) by shifting the focus from how noisy labels are produced (annotator quality) to why they arise (data degradation).
Interpretability: By making ambiguity mechanisms explicit, CILN allows researchers to study how specific types of data degradation (e.g., blur vs. missing values) influence learning dynamics.
Realism: The framework demonstrates that controlled corruptions can produce label distributions that better approximate human uncertainty than current synthetic methods, particularly in low-to-moderate severity regimes.
Algorithmic Insight: The results suggest that current noise-robust methods may be overfitting to the specific noise structures of traditional benchmarks (where noise is largely independent of the input) and may fail when noise is structurally coupled with the input via corruption.

The authors conclude that CILN provides a practical, interpretable, and mechanism-controlled framework for evaluating noisy-label learning under diverse sources of instance difficulty, urging the community to consider noise structure alongside noise rate in benchmark design.