Intrinsic Gradient Suppression for Label-Noise Prompt Tuning in Vision-Language Models

1. Problem Statement

Vision-Language Models (VLMs) like CLIP exhibit remarkable zero-shot generalization. However, when adapting these models to specific downstream tasks via Prompt Tuning (learning continuous prompt vectors while keeping the backbone frozen), they remain highly sensitive to label noise.

The core issue identified by the authors is a "Gradient Surge Paradox":

• In standard Cross-Entropy (CE) loss, the gradient magnitude is proportional to the difference between the ground truth label and the model's prediction.
• Pre-trained VLMs possess strong priors. When a sample is mislabeled (e.g., an image of a "car" labeled as "dog"), the model's prior correctly predicts "car" with high confidence, resulting in a near-zero probability for the noisy label "dog."
• Paradoxically, this large discrepancy between the confident prediction and the noisy label generates a disproportionately large gradient.
• These massive gradients aggressively overwrite the pre-trained knowledge, causing the model to overfit to the noise and degrade performance, often falling below the zero-shot baseline.

2. Methodology: Double-Softmax Prompt Tuning (DSPT)

The authors propose Double-Softmax Prompt Tuning (DSPT), a hyperparameter-free framework designed to intrinsically suppress these disruptive gradients without complex architectural changes.

Core Mechanism:
Instead of the standard CE loss ($L = -\log(\text{softmax}(z)_{\tilde{y}})$), DSPT applies a sequential probabilistic normalization:
$$L_{ours} = -\log(\text{softmax}(\text{softmax}(z))_{\tilde{y}})$$
Where $z$ represents the logits from the prompt-tuned model, and $\tilde{y}$ is the noisy label.

How it Works:

1. First Softmax: Converts logits $z$ into a probability distribution $p = \text{softmax}(z)$.
2. Second Softmax: Applies softmax again to $p$ to generate a refined distribution $q = \text{softmax}(p)$.
3. Intrinsic Gradient Suppression:
   • For High-Confidence Mismatches (Noisy Samples): If the model is highly confident in a prediction that contradicts the noisy label (a likely mislabeled sample), the double-softmax pushes the logits into a saturation zone. Theoretical analysis shows that the gradient magnitude for these samples converges to zero, effectively "nullifying" their influence on the update.
   • For Informative/Uncertain Samples: For samples where the model is less confident (potentially ambiguous or correctly labeled but difficult), the gradients remain within an "active learning zone," allowing the model to learn subtle refinements.
4. Loss Boundedness: The method ensures the loss function is bounded, preventing the model from fitting extreme outliers.

3. Key Contributions

• Theoretical Insight: The paper provides a systematic analysis revealing the "gradient surge paradox" in VLM prompt tuning, where strong priors combined with noisy labels create destructive gradient updates.
• Novel Framework (DSPT): Introduction of a simple, drop-in Double-Softmax loss function that acts as an intrinsic noise filter. It re-purposes the traditionally problematic phenomenon of "gradient vanishing" as a principled mechanism for noise suppression.
• Hyperparameter-Free Design: Unlike existing methods (e.g., LogitNorm, LogitClip) that require careful tuning of thresholds or clamping parameters, DSPT requires no hyperparameters, making it robust across different datasets and noise rates.
• Theoretical Guarantees: The authors provide proofs demonstrating that DSPT induces gradient nullification for high-confidence mismatches and bounds the expected risk difference between clean and noisy distributions.

4. Experimental Results

The authors evaluated DSPT on 9 diverse datasets (including Caltech101, StanfordCars, OxfordPets, Flowers102, etc.) under various noise conditions (Symmetric and Pair-Flip noise at rates up to 90%).

• Performance: DSPT consistently achieved State-of-the-Art (SOTA) robustness. It outperformed baselines like CoOp, LogitNorm, Label Smoothing, and the complex NLPrompt method.
  • In high-noise scenarios (e.g., 80% symmetric noise), CoOp often failed (dropping below zero-shot performance), while DSPT maintained high accuracy.
  • DSPT showed a 0.4% to 5% improvement in average accuracy over the second-best method (NLPrompt) across most settings.
• Robustness to Extreme Noise: Even under extreme noise (90% symmetric, 80% pair-flip), DSPT significantly outperformed sample-selection methods and other loss-robust techniques.
• Gradient Analysis: Empirical evidence (Figure 3) confirmed that DSPT successfully suppresses the gradients of mislabeled samples to near-zero while maintaining gradients for correctly labeled samples, whereas standard CE loss allows noisy samples to dominate.
• Comparison with LogitClip: The paper highlights that LogitClip is highly sensitive to its threshold hyperparameter ($\tau$), which varies significantly across datasets. DSPT avoids this instability entirely.

5. Significance and Impact

• Simplicity and Generality: The work demonstrates that a simple mathematical modification (double softmax) can solve a complex problem (label noise in VLMs) more effectively than complex, hand-crafted architectures.
• Paradigm Shift: It challenges the conventional view that "gradient vanishing" is always a defect, showing it can be a feature for filtering noise in pre-trained models.
• Practical Applicability: As a "drop-in" solution requiring no hyperparameter tuning, DSPT is highly practical for real-world applications where label quality is unknown or poor.
• Future Directions: The authors suggest extending this mechanism to Large Vision-Language Models (LVLMs) for downstream tasks, positioning double-softmax as a foundational building block for robust multi-modal learning.

In summary, this paper presents a theoretically grounded, empirically superior, and practically simple solution to the critical problem of label noise in adapting Vision-Language Models, leveraging the models' own pre-trained confidence to filter out errors.