Technical Summary: Ordinal Adaptive Correction (ORDAC)

Problem Statement

The paper addresses the critical challenge of label noise in ordinal image classification. In tasks such as age estimation and disease severity grading, class boundaries are often ambiguous, making the labeling process prone to error. Unlike nominal classification, where errors are discrete, ordinal errors have a semantic distance (e.g., misclassifying age 20 as 25 is less severe than misclassifying it as 60).

Existing approaches to noisy labels generally fall into two categories:

1. Model-centric: Designing noise-tolerant loss functions or regularization.
2. Data-centric (Sample Selection): Identifying and discarding potentially noisy samples (e.g., CASSOR, ICDF).

The authors argue that sample selection is suboptimal for ordinal data because it discards potentially valuable feature information contained in mislabeled instances. Furthermore, standard ordinal regression methods lack explicit mechanisms to handle incorrect rank annotations, leading to degraded performance and reliability. There is a specific gap in methods that can correct rather than discard noisy labels by adaptively modeling the inherent uncertainty of ordinal data.

Methodology: ORDinal Adaptive Correction (ORDAC)

The proposed solution is ORDAC, a data-centric framework that leverages Label Distribution Learning (LDL). Instead of treating a label as a single discrete value, ORDAC represents each label as a Gaussian distribution characterized by a mean ($\mu$) and a standard deviation ($\sigma$). The mean represents the label value, while the standard deviation quantifies the model's uncertainty.

Core Workflow

The framework operates iteratively using a K-fold cross-training strategy to prevent the model from being biased by its own confident but potentially incorrect predictions:

1. Initialization:

   • The dataset is split into $K$ folds.
   • $K$ identical Fixed-form LDL (FLDL) models are initialized.
   • Each sample's label is initialized as a Gaussian distribution $N(\mu_i, \sigma_i^2)$, where $\mu_i$ is the noisy label and $\sigma_i$ is a fixed constant representing uniform initial uncertainty.

2. Warm-up Phase:

   • All $K$ models are trained concurrently on their respective training sets for a specified number of epochs ($E_{corr}$) to learn initial feature representations on the noisy data.

3. Iterative Correction Phase:

   • After warm-up, the adaptive correction mechanism activates.
   • For each fold, the model trained on the remaining $K-1$ folds predicts the labels of the held-out validation fold.
   • These predictions are used to update the label distributions ($\mu$ and $\sigma$) of the validation samples.
   • The updated distributions are then propagated back to the training sets of the other models for the next epoch.

The Adaptive Correction Mechanism

The update process consists of two stages applied to each sample in the validation fold:

1. Class-wise Prediction Debiasing:

   • Ordinal models often develop a bias toward middle-rank classes due to loss functions and data imbalance.
   • To counteract this, the model's predictions are shifted to re-center class-wise predictions around the true class label.
   • The shifted prediction $\hat{y}^{shifted}_i$ is calculated as: $\hat{y}^{shifted}_i = \hat{y}_i - (\text{mean}_{\mu_i} - \mu_i)$.

2. Sample-wise Distribution Update:

   • A correction coefficient $\lambda^{corr}_i$ is computed based on model confidence ($\gamma_i$) and class frequency ($\pi_{\mu_i}$) to ensure rare classes are corrected more cautiously.
   • Standard Deviation Update ($\sigma$): If the prediction error $|e_i|$ is larger than the current uncertainty $\sigma_i$, the uncertainty is increased (indicating a likely noisy label). If the error is small, uncertainty decreases.
     • $\sigma^{new}_i = \sigma_i + \alpha_i \times (|e_i| - \sigma_i)$
   • Mean Update ($\mu$): The mean is shifted toward the debiased prediction.
     • $\mu^{new}_i = \mu_i + \beta_i \times e_i$
   • Here, $\alpha_i$ and $\beta_i$ are sample-specific learning rates derived from base rates $\alpha_{base}$ and $\beta_{base}$.

Variants

The authors introduce two variants to analyze the framework's behavior:

• ORDACC (Correct): Generates a static, cleaned dataset after training. A new model is trained from scratch on this dataset without further online corrections.
• ORDACR (Remove): Builds on ORDACC but removes samples that remain highly uncertain (where $\sigma^{new}_i \geq \sigma^{initial}_i$) after correction, treating them as outliers.

Experimental Results

The method was evaluated on two benchmark datasets: Adience (age estimation) and Diabetic Retinopathy (DR) (disease severity). Experiments involved injecting asymmetric Gaussian noise at rates of $\tau \in \{0.2, 0.4\}$.

Key Findings

• Performance under Noise: ORDAC and its variants significantly outperformed standard baselines (CORAL, DLDL-v2) and state-of-the-art sample selection methods (CASSOR) across all noise levels.
  • Example: On Adience with 40% noise, ORDACR reduced the Mean Absolute Error (MAE) from 0.86 (DLDL-v2) to 0.62 and increased Recall from 0.37 to 0.49.
• Intrinsic Noise Correction: Even with no injected noise ($\tau=0$), ORDAC outperformed baselines. This suggests the original datasets contain inherent label noise (e.g., self-reporting errors in Adience) that the method successfully identified and corrected.
• Correction vs. Selection: While sample selection (CASSOR) performed well, ORDACR generally achieved better results, particularly at higher noise levels, demonstrating that correcting labels is more data-efficient than discarding them.
• Ablation Studies:
  • Removing the class-wise debiasing step caused the correction process to collapse toward the majority middle class, significantly degrading performance.
  • Hyperparameter analysis showed that a small $\alpha_{base}$ (0.2) and large $\beta_{base}$ (0.8) yielded the best results, indicating the model should be conservative in updating uncertainty but aggressive in correcting the mean value.

Significance and Contributions

The paper claims the following contributions:

1. Paradigm Shift: It proposes a shift from sample selection (discarding data) to adaptive label correction for ordinal tasks, preserving valuable feature information.
2. Uncertainty Modeling: It introduces a mechanism to dynamically update both the mean and standard deviation of label distributions, allowing the model to explicitly represent and manage uncertainty during training.
3. Robustness: Extensive experiments demonstrate that adaptive correction using label distributions is an effective strategy to enhance the robustness and accuracy of ordinal classification models in the presence of noisy data.

The authors conclude that by focusing on improving data quality at a fundamental level through adaptive correction, ORDAC paves the way for more accurate and reliable ordinal classification, particularly in domains where clean data is scarce and label ambiguity is high. Future work is suggested to explore non-parametric distributions and automatic tuning of correction rates.