Technical Summary: CANOLA – A Noise-Aware Framework for Correcting Corrupted Labels

1. Problem Statement

High-quality labeled data is the foundational resource for training reliable Machine Learning (ML) and Deep Learning (DL) models. However, real-world datasets frequently contain a significant proportion of corrupted labels due to manual errors or imperfect automated annotation tools. These mislabeled instances severely degrade model generalization, leading to biased or unreliable predictions.

While prior research has addressed label noise through model-centric approaches (designing robust algorithms to withstand noise) and data-centric approaches (detecting noisy instances), existing data-centric methods often stop at detection. Automatic label correction remains underexplored. Existing correction methods generally fall into two categories:

1. Neighbor-based correction: Relies on label consistency among similar samples (e.g., majority voting). These struggle near decision boundaries where similar instances may belong to different classes.
2. Model-based correction: Uses the model's own predictions to refine labels, often relying on early training iterations. These methods suffer from two major limitations: (a) high sensitivity to fluctuations in model predictions, and (b) the risk of "confirmation bias," where the model amplifies its own early errors, particularly in high-noise scenarios or with "hard-but-clean" instances.

2. Methodology: The CANOLA Framework

The authors propose CANOLA (Corrupted lAbel cOrrectiON via Noise-Aware Learning and iterative refinement), a novel framework that decouples feature learning from label refinement. CANOLA operates through an iterative pipeline consisting of two distinct phases:

Phase 1: Noise Transition Matrix Construction

The goal is to estimate the underlying noise distribution of the dataset without access to ground-truth clean labels.

• Asymmetric Co-Training: CANOLA maintains two models: a reliable model ($\mathcal{F}_r$) trained exclusively on a growing subset of likely clean data ($D_c$), and a noisy model ($\mathcal{F}_n$) trained on the entire noisy dataset ($\tilde{D}$).
• Clean Subset Identification: The process involves three stages:
  • Warm-up: Both models are trained on the full noisy dataset to establish basic patterns.
  • Exploration: Early in training, $\mathcal{F}_n$ (trained on all data) is used to identify likely clean samples (where prediction matches the observed label) to expand $D_c$.
  • Exploitation: Later in training, a stricter consensus criterion is applied: samples are added to $D_c$ only if both $\mathcal{F}_r$ and $\mathcal{F}_n$ agree with the observed label.
• Matrix Estimation: Using the reliable model $\mathcal{F}_r$ trained on $D_c$, the framework estimates the Noise Transition Matrix ($T$). This matrix models the probability $P(\tilde{y}=c' | y=c)$ that a clean label $c$ is corrupted into a noisy label $c'$.

Phase 2: Corrupted Label Correction

This phase trains a noise-aware model to perform cautious, iterative label refinement.

• Noise-Aware Training: The reliable model $\mathcal{F}_r$ is further trained on the entire noisy dataset. Instead of fitting directly to noisy labels, the model is trained to predict the clean label distribution. These predictions are projected through the estimated noise transition matrix $T$ to reconstruct the expected noisy distribution. The loss is computed between this reconstructed distribution and the observed noisy labels using Kullback-Leibler (KL) Divergence. This forces the model to explicitly account for noise structure, down-weighting unreliable supervision signals.
• Loss-Stabilization Trigger: Label refinement is not performed immediately. It is triggered only after the training loss stabilizes, ensuring the model has reached a mature, converged state before attempting corrections.
• Iterative Soft Relabeling: Once triggered, the model's predictions are blended with the observed labels to form soft labels (probability distributions) rather than hard one-hot vectors. This blending is controlled by a coefficient $\alpha$:
  $$\tilde{p}^{(t)}_i = \alpha \cdot p_i + (1 - \alpha) \cdot \tilde{p}^{(t-1)}_i$$
  This progressive, soft update strategy prevents premature commitment to potentially incorrect predictions and mitigates error propagation. The process repeats for $\tau$ iterations or until convergence, with the final soft labels converted to hard labels via maximum a posteriori estimation.

3. Key Contributions

The paper outlines three primary contributions:

1. Realistic Benchmark Construction: The authors constructed a benchmark of five datasets (covering image and text) labeled using diverse automated techniques (LLM-based, Supervised, Semi-Supervised, and Weak-Supervised) to facilitate evaluation under practical, non-idealized noise conditions. They also included the real-world Clothing1M dataset.
2. The CANOLA Framework: A novel correction framework that decouples noise estimation from label refinement. It combines a noise-aware DNN (regularized by an estimated noise distribution) with a loss-stabilized trigger to perform cautious, iterative soft label refinement.
3. Comprehensive Evaluation: Extensive experiments demonstrating that CANOLA consistently outperforms State-of-the-Art (SOTA) label correction methods and yields significant downstream performance gains, even when simple classifiers are trained on the corrected data.

4. Experimental Results

The framework was evaluated on six datasets (FashionMNIST, OrganAMNIST, RESISC45, AGNews, Yahoo! Answer, and Clothing1M) under various noise scenarios.

• Label Correction Accuracy: CANOLA consistently achieved the lowest error rates across all datasets and noise types. On average, it reduced the original dataset error rates by approximately 25%. Under challenging conditions (e.g., Weak-Supervised labeling), it achieved relative improvements of up to 31% over the original data.
• Comparison with SOTA: CANOLA outperformed existing correction methods (DOCTA, SELC, SIDYP) with relative improvements in error reduction ranging from 19% to 52%. Unlike some baselines (e.g., DOCTA or SIDYP in specific high-noise cases), CANOLA did not degrade data quality in any evaluation setting.
• Downstream Utility: Models trained on CANOLA-corrected data showed substantial performance gains. On average, downstream classifiers achieved a 10% increase in F1-Macro compared to training on original noisy data. In severe noise scenarios, CANOLA-corrected data enabled simple classifiers to outperform complex model-centric noise-robust approaches (e.g., DivideMix, Co-Teaching, SIDYP) by margins of 8% to 67%.
• Ablation Studies:
  • Two-Phase Design: Both phases were found necessary; Phase 1 provides a reliable foundation, while Phase 2 enables robust noise modeling and stable refinement.
  • Soft vs. Hard Updates: Soft label refinement significantly outperformed hard label updates, reducing error rates by an additional ~16% on average.
  • Loss Function: KL Divergence was found to be the most effective loss function for training the noise-aware model.
  • Sensitivity: The framework is robust to hyperparameter variations, with optimal performance observed at 15–20 warm-up epochs and approximately 10 correction iterations.

5. Significance and Claims

The paper claims that CANOLA represents a significant advancement in data-centric learning by explicitly addressing the root cause of label corruption. The authors emphasize that:

• Explicit cleaning is superior to robust training: In high-noise scenarios, cleaning the data via CANOLA yields greater downstream benefits than relying solely on noise-robust training algorithms.
• Stability is key: By decoupling the learning phase from the correction phase and utilizing a loss-stabilization trigger, CANOLA avoids the confirmation bias loops that plague earlier model-based correction methods.
• Practicality: The framework is computationally efficient (processing datasets in minutes to half an hour) and effective across diverse domains (image and text) and noise sources (automated pipelines and real-world datasets).

The authors conclude that CANOLA provides a principled, stable, and effective mechanism for repairing corrupted labels, making it a valuable tool for improving the reliability of ML models trained on imperfect real-world data.