Leveraging Dissimilarity Invariance as a Robust Anchor for Learning with Noisy Labels

Technical Summary: Leveraging Dissimilarity Invariance as a Robust Anchor for Learning with Noisy Labels

Problem Statement
Deep learning models achieve high performance in visual recognition but suffer significant degradation when training labels are corrupted by noise. Existing approaches often fail because noisy labels distort the semantic structure of data, particularly the learned similarities between samples. The authors observe that semantic similarity (positive pairs) is inherently fragile under label noise; incorrect labels severely distort pairwise similarities, leading to erroneous affinity alignment and preventing the model from learning meaningful relationships. Conversely, the paper identifies an underexplored phenomenon where the semantic dissimilarity between unrelated samples (negative pairs) remains remarkably stable despite increasing noise rates.

Methodology: NegScale
To address the fragility of similarity, the authors propose NegScale, a plug-and-play framework that shifts the learning focus from fragile similarity to robust dissimilarity. The framework operates by leveraging "Dissimilarity Invariance" as a robust anchor. NegScale consists of two primary modules:

1. Structured Negative Orthogonality Penalty (SNOP):
   This module enforces orthogonality among negative pairs (semantically unrelated samples) within each minibatch. It operates on the principle that dissimilar samples should occupy distinct subspaces. SNOP comprises two components:

   • Global Orthogonality Loss ($L_{global}$): Encourages the set of normalized difference vectors between negative pairs to approximate an identity matrix, ensuring global repulsion.
   • Local Orthogonality Loss ($L_{local}$): A confidence-weighted penalty that directly penalizes the inner product alignment between negative pairs. This loss assigns higher penalties to pairs involving low-confidence (potentially noisy) samples, thereby enforcing stronger dissimilarity constraints where they are most needed.

2. Dissimilarity-Calibrated Similarity Adjustment (DCSA):
   This module suppresses spurious similarity caused by mislabeled positive pairs. It uses the stable dissimilarity patterns as anchors to impose soft upper bounds on the similarity of positive pairs.

   • For each positive pair, an adaptive calibration factor ($\delta_{ij}$) is computed based on the pair's proximity to negative samples.
   • If a positive pair is entangled with unrelated samples (high $\delta_{ij}$), its similarity is restricted.
   • The loss function penalizes positive pairs whose similarity exceeds a soft upper bound ($1 - \delta_{ij}$), preventing the model from internalizing false associations.

The total objective function combines the original loss ($L_{org}$) with the SNOP and DCSA losses, balanced by coefficients $\lambda$ and $\mu$. The method requires no external data or pre-training and can be seamlessly integrated into existing pipelines.

Key Contributions

• Dissimilarity Invariance: The paper formally identifies and analyzes the phenomenon where semantic dissimilarity between unrelated samples remains stable under label noise, contrasting with the volatility of semantic similarity.
• NegScale Framework: The authors propose a novel framework that utilizes this invariance to model robust dissimilarity and suppress spurious similarity.
• Theoretical Analysis: The paper provides theoretical proofs demonstrating that the expected variation in similarity for negative pairs is smaller than for positive pairs under symmetric noise. It further establishes bounds showing that NegScale mitigates feature perturbations and margin erosion caused by noisy labels.
• Empirical Validation: Extensive experiments show that NegScale consistently outperforms state-of-the-art baselines on both synthetic noise benchmarks (CIFAR-10/100) and real-world noisy datasets (CIFAR-10N, CIFAR-100N, Animal-10N, WebVision).

Experimental Results

• Synthetic Noise: On CIFAR-10 and CIFAR-100 with symmetric, pair, and instance-dependent noise, NegScale (combined with the RoLR baseline) achieves new state-of-the-art results. Notably, on CIFAR-100 with 80% noise, it improves accuracy by 3.6% over RankMatch.
• Real-World Noise: The method demonstrates superior robustness on real-world noisy datasets, outperforming competing approaches on CIFAR-10N, Animal-10N, and WebVision. On mini-WebVision and ILSVRC12, it surpasses the hybrid method UNICON by over 3% in top-1 accuracy.
• Plug-and-Play Capability: Integrating NegScale with other methods (DivideMix, RankMatch, ANNE) consistently boosts their performance, confirming its broad applicability.
• Ablation Studies: Removing either SNOP or DCSA leads to significant performance drops, particularly under high noise ratios, validating the necessity of both components. Sensitivity analysis confirms that balanced regularization weights and carefully calibrated thresholds for negative pair selection are critical for optimal performance.

Significance
The paper claims that by shifting focus from fragile similarity to robust dissimilarity, models can reliably learn and leverage stable patterns to improve performance under noisy supervision. The significance lies in providing a theoretical and practical foundation for using "negative" relationships as a stable anchor, offering a simple yet effective mechanism to correct the structural distortions caused by label noise without requiring complex sample selection or external data.