🤖 machine learning

Stop Replacing Noise with Noise: Two-Source Reliability Assessment for Label Correction and Sample Reweighting in Label-Noise Learning

The paper introduces TRACE, a framework that decouples the assessment of observed labels and pseudo-targets using distinct reliability metrics to prevent the common pitfall in noisy-label learning where correcting one unreliable signal inadvertently amplifies another.

Original authors: Wenxiao Fan, Kan Li

Published 2026-08-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wenxiao Fan, Kan Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to recognize animals by showing it thousands of photos. But here's the catch: a mischievous gremlin has scribbled the wrong names on some of the labels. A picture of a cat might be labeled "dog," and a dog might be labeled "cat." This is the world of "noisy-label learning," a tricky corner of computer science where machines try to learn the truth even when their teachers are lying.

To fix this, scientists have developed a clever strategy called "refurbishment." Think of it like a student taking a test. If the student is unsure about an answer, they don't just guess; they look at their own notes (the model's current knowledge) to see what it thinks the answer should be. The old way of doing this was to use a single "trust meter" for both the teacher's label and the student's guess. If the meter said, "Don't trust the teacher," the system would automatically say, "Great, trust the student's guess instead!" It assumed that if one source was bad, the other must be good. But what if the student's notes are also wrong? What if the robot learned the wrong answer from the teacher in the first place, and now it's confidently repeating that same mistake? That's the danger this paper explores.

The researchers behind this study, Wenxiao Fan and Kan Li from the Beijing Institute of Technology, discovered that this "all-or-nothing" trust switch is dangerous. They call the problem "replacing noise with noise." Imagine a detective who, upon realizing a witness is lying, immediately switches to trusting a second witness who was standing right next to the liar and heard the same lies. The second witness isn't necessarily telling the truth; they might just be echoing the first one.

The paper shows that in deep learning, when a model gets confused by noisy labels, its "deeper" layers (the parts of the brain that do complex reasoning) get scrambled and start believing the wrong things. However, the "shallower" layers (the parts that see basic shapes and edges) stay relatively calm and stable. The old methods were blindly trusting the scrambled deep layers to fix the labels, which often just made the errors worse.

To solve this, the authors propose a new framework called TRACE. Instead of using one trust meter for everything, TRACE uses two separate, independent checks.

Checking the Teacher: It looks at the original label using three clues: how well the model fits the data, how stable the basic shapes are (the shallow layers), and whether two different "student" models agree on the answer.
Checking the Student's Guess: It looks at the model's own guess using a completely different clue: how confident the model is in that specific guess.

The magic of TRACE is that it doesn't assume that if the teacher is wrong, the student is right. It asks, "Is the teacher wrong?" AND "Is the student actually right?" If the teacher is wrong but the student is also guessing wildly, TRACE says, "Hold on, let's not trust either of them right now." It only uses the student's guess to fix the label if the student is genuinely confident and reliable.

The team tested this idea on a variety of messy datasets, including synthetic noise (where they intentionally messed up the labels) and real-world messy data like photos from the internet. They found that TRACE consistently outperformed previous methods. For example, on a dataset called CIFAR-100 with 50% of the labels being wrong, TRACE improved the accuracy of the best existing method by about 1.16 percentage points. More importantly, it reduced the number of times the model made "high-confidence errors"—those moments where the robot is super sure it's right, but it's actually wrong.

In short, this paper suggests that when learning from messy data, we shouldn't just swap one unreliable source for another. We need to check each source independently. By separating the trust we place in the original labels from the trust we place in the model's own corrections, TRACE helps robots learn the truth without getting tricked by their own mistakes. It's a reminder that in the world of AI, sometimes the best way to fix a lie is to stop assuming the next guess is automatically the truth.

Technical Summary: TRACE – Two-Source Reliability Assessment for Label Correction and Sample Reweighting

Problem Statement

Current noisy-label learning methods, particularly those based on label refurbishment (e.g., DivideMix, SELFIE, RoLR), rely on a "one-size-fits-all" control mechanism. These methods typically interpolate between an observed noisy label and a model-derived pseudo target using a single sample-wise cleanliness score. This creates a hidden coupling: reducing trust in the observed label automatically increases the weight of the pseudo target.

The authors argue this coupling creates a risk of "replacing noise with noise." Because the pseudo target is generated by a model trained on the same corrupted supervision, it may inherit the model's noisy biases. Consequently, when the observed label is deemed unreliable, the system may blindly trust a pseudo target that is equally or more erroneous. Empirical analysis on CIFAR-100N demonstrates that even in "high-clean" bins, pseudo targets can follow the noisy observed label 99.7% of the time while remaining correct on only 74.7% of samples, often with high confidence.

Methodology: TRACE Framework

The paper proposes TRACE (Two-Source Reliability Assessment), a framework that decouples the reliability assessment of the two supervision sources: the observed label and the pseudo target. Instead of a single scalar controlling both branches, TRACE computes independent scores ( $s^{obs}_i$ and $s^{pseudo}_i$ ) to determine branch weights.

1. Preliminary Analysis & Motivation
The authors provide representation-level diagnostics to justify separating the two sources:

Depth-Dependent Bias: Noisy supervision perturbs deeper network layers more strongly than shallow layers. Gradient sensitivity and prototype flipping (where mislabeled samples align with incorrect class anchors) increase significantly in deeper layers as noise rates rise.
Shallow Stability: Shallow relations remain comparatively stable under noise, providing information about observed-label correctness that loss-based metrics miss.
Conclusion: Rejecting an observed label does not justify trusting a pseudo target derived from deep, noise-corrupted representations. Conversely, shallow structural stability can refine the reliability of the observed label without certifying the pseudo target.

2. The TRACE Algorithm
TRACE modifies the standard refurbishment loss $y^{rec}_i = \lambda_i \hat{y}_i + (1-\lambda_i)q_i$ by introducing source-specific weights:
$a_i = s^{obs}_i, \quad b_i = (1 - s^{obs}_i) s^{pseudo}_i$
The final corrected target is $\tilde{y}_i = \frac{a_i \hat{y}_i + b_i q_i}{a_i + b_i + \epsilon}$ .

Observed-Label Reliability ( $s^{obs}_i$ ):
This score combines three signals to assess the observed label $\hat{y}_i$ :
- Loss Fit ( $c^{loss}_i$ ): The standard small-loss prior (probability of being clean).
- Shallow Relation Stability ( $c^{str}_i$ ): Measures the stability of local relations in shallow layers across the network depth. High stability indicates the sample's representation aligns with clean prototypes, even if the loss is high.
- Prediction Agreement ( $g^{agr}_i$ ): A consistency gate based on agreement between two networks.
  These are combined into a weighted score that refines the loss posterior.
Pseudo-Target Reliability ( $s^{pseudo}_i$ ):
This score assesses the pseudo target $q_i$ independently. It is derived solely from the model confidence (maximum probability of the pseudo target), raised to a power $\rho$ to suppress low-confidence predictions. Crucially, a low observed-label score ( $s^{obs}_i$ ) creates a need for correction, but the pseudo branch is only activated if $s^{pseudo}_i$ is also high.

3. Training Process
The framework adjusts the supervision strength $w_i$ for each sample based on the sum of branch weights ( $a_i + b_i$ ). Samples where neither source is reliable receive lower weights, while reliable observed labels remain dominant.

Key Contributions

Identification of the "Replacing Noise with Noise" Risk: The paper formally identifies and demonstrates that using a single cleanliness score to control both the observed label and pseudo target leads to the propagation of noise, as the pseudo target is not an external oracle but a product of the same corrupted training process.
TRACE Framework: The proposal of a plug-in, two-source reliability assessment mechanism that decouples the evaluation of observed labels and pseudo targets. It utilizes shallow structural stability for the former and independent confidence gating for the latter.
Empirical Validation: Extensive experiments showing that TRACE improves representative refurbishment baselines across synthetic noise, human-annotated noise (CIFAR-N), and large-scale real-world noise (WebVision, Food-101N, Clothing1M).

Experimental Results

Synthetic Noise (CIFAR-10/100): TRACE improves the performance of multiple state-of-the-art baselines (DivideMix, RoLR, DISC, ANNE) across symmetric, pair, and instance-dependent noise settings. For example, on CIFAR-100 with 40% Pair noise, ANNE+TRACE achieves 79.3% accuracy compared to 78.4% for ANNE alone.
Real-World Noise: On CIFAR-10N and CIFAR-100N, TRACE consistently improves or matches the best reported results. On Clothing1M, DivideMix+TRACE reaches 77.6% accuracy, outperforming the baseline DivideMix (74.8%) and other plug-in methods.
Pseudo-Target Quality: Analysis reveals that TRACE significantly improves the accuracy of pseudo targets used for correction, particularly on "low-clean + noisy" samples. It reduces high-confidence errors (e.g., from 8.93% to 6.16% on CIFAR-100) and lowers calibration error, indicating that the selected pseudo targets are not just more frequent, but more reliable.
Ablation Studies: Removing the decoupling (reverting to a single score) or removing the pseudo-target confidence gate ( $s^{pseudo}_i$ ) results in measurable performance drops, confirming the necessity of independent assessment.

Significance and Claims

The paper claims that TRACE provides a safer interface for noisy-label learning by addressing the fundamental mismatch in current refurbishment methods. By separating the evidence used to evaluate the observed label from the evidence used to evaluate the pseudo target, TRACE prevents the system from blindly swapping one unreliable signal for another. The authors emphasize that their approach does not require a new learning objective or a new pseudo-target generator; rather, it acts as a modular reliability filter that enhances existing refurbishment pipelines. The findings support the view that source-specific reliability assessment is essential for robust learning in the presence of label noise.