Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models

1. Problem Statement

Vision-Language Models (VLMs) like CLIP have demonstrated impressive zero-shot capabilities but remain highly susceptible to adversarial attacks, where minor pixel-level perturbations cause misclassification.

• Current Limitations: Existing methods for enhancing adversarial robustness, known as Adversarial Finetuning (AFT), typically follow a paradigm where the pretrained vision encoder is finetuned on a proxy classification dataset (e.g., ImageNet) using adversarial examples and a cross-entropy loss.
• The Core Issue: This approach deviates significantly from CLIP's original pretraining process, which relies on contrastive learning over massive, noisy image-text pairs from the web.
  • Consequence 1: Finetuning on a classification dataset with cross-entropy loss degrades the model's zero-shot generalization capabilities.
  • Consequence 2: The robustness gained is often limited in transferability across different domains and datasets because the training distribution (clean, labeled classes) does not match the pretraining distribution (noisy, open-vocabulary web data).

2. Methodology: AdvFLYP

The authors propose AdvFLYP (Adversarially Finetune Like You Pretrain), a paradigm that aligns the adversarial finetuning process with CLIP's original pretraining recipe.

A. Training Data Distribution

Instead of using a curated classification dataset like ImageNet, AdvFLYP utilizes web-scale image-text pairs (specifically, 1 million pairs sampled from LAION-400M). This mimics the noisy, diverse distribution of the original pretraining data.

B. Training Objective (Contrastive Loss)

Rather than minimizing cross-entropy loss to align images with class labels, AdvFLYP minimizes the contrastive loss (Eq. 1 in the paper).

• Process: For a batch of image-text pairs $\{(x_i, t_i)\}$, adversarial perturbations $\delta_i$ are generated to maximize the contrastive loss within the batch.
• Optimization: The vision encoder is then updated to minimize this contrastive loss, effectively forcing the model to match adversarial images with their corresponding texts while treating other images in the batch as negative samples.
• Key Difference: Unlike previous methods where perturbations are generated independently per image to maximize classification loss, AdvFLYP generates perturbations jointly for the batch to maximize the contrastive objective.

C. Regularization Strategies

Since web data is noisy, generating adversarial examples via contrastive loss can lead to distorted feature embeddings that harm the vision encoder. The authors introduce two regularization terms to mitigate this:

1. Feature-Level Regularization ($L_{feat}$):

   • Goal: Preserve the zero-shot capabilities and prevent the vision encoder from learning distorted features.
   • Mechanism: Penalizes the deviation of the adversarial image features (produced by the target model $f_\theta$) from two sources:
     1. The clean image features (from the same model).
     2. The adversarial features produced by the frozen original CLIP ($F_{\theta_0}$).
   • Effect: Significantly improves clean accuracy by keeping the feature space stable.

2. Logit-Level Regularization ($L_{logit}$):

   • Goal: Enhance robustness and generalization.
   • Mechanism: Applies KL-divergence loss between the probability logits of the target model and those of the frozen original CLIP (and clean counterparts).
   • Effect: Significantly improves adversarial robustness across downstream tasks.

Final Loss Function:
$$\theta' = \arg \min_\theta \{ \mathcal{L}_{CLIP} + \mathcal{L}_{logit} + \mathcal{L}_{feat} \}$$

3. Key Contributions

1. Paradigm Shift: Challenges the standard AFT practice of using cross-entropy loss on labeled datasets. Proposes that respecting the pretraining data distribution (web-scale image-text) and objective (contrastive loss) is crucial for robust VLMs.
2. Novel Regularization: Demonstrates that while logit-level regularization aids robustness, feature-level regularization is uniquely critical for preserving clean accuracy in the context of contrastive adversarial training on noisy data.
3. State-of-the-Art Performance: Establishes a new standard for adversarial finetuning that outperforms existing methods without requiring complex architectural changes or additional training data beyond web-scale pairs.

4. Experimental Results

The authors evaluated AdvFLYP on 14 downstream datasets spanning various domains (object recognition, fine-grained classification, scene recognition, etc.) under three attack types: PGD-10, CW-10, and AutoAttack.

• Robustness:

  • AdvFLYP (Basic): Achieved an average robust accuracy of 35.61% (vs. 33.00% for TeCoA, the previous SOTA).
  • AdvFLYPfull (with Regularization): Achieved 38.39% average robust accuracy, significantly outperforming all baselines including PMG-AFT and TGA-ZSR.
  • Stronger Attacks: Under stronger attack budgets ($\epsilon = 2/255$ and $4/255$), AdvFLYP consistently maintained a performance gap over baselines, demonstrating superior transferability.

• Clean Accuracy:

  • A major pain point in AFT is the trade-off between robustness and clean accuracy.
  • AdvFLYPfull achieved an average clean accuracy of 55.84%, outperforming supervised baselines (TeCoA: 51.35%, PMG-AFT: 54.25%) and approaching the performance of unsupervised methods while maintaining higher robustness.

• Ablation Studies:

  • Confirmed that the contrastive objective is central to robustness.
  • Showed that feature-level regularization is essential for retaining clean accuracy in this specific paradigm, a finding that contrasts with previous studies on cross-entropy-based AFT where feature regularization was less effective.
  • Demonstrated that simply applying the "Finetune Like You Pretrain" (FLYP) concept naively on ImageNet (ignoring batch overlap) does not work; the combination of web data + contrastive loss + joint perturbation is necessary.

5. Significance

• Rethinking Robustness: The paper argues that robustness should not be treated as a separate post-training phase that ignores the pretraining context. Instead, robustness is best achieved by continuing the pretraining trajectory (distribution and objective) even when introducing adversarial noise.
• Generalizability: The method is domain-agnostic and does not rely on specific class labels, making it highly transferable to open-vocabulary tasks.
• Efficiency: The approach is simple and intuitive, requiring only standard contrastive loss and regularization terms, yet it outperforms complex methods that rely on attention mechanisms or text-guided prompts.
• Community Impact: It sets a new benchmark for adversarial finetuning in VLMs, suggesting that future research should prioritize aligning finetuning objectives with pretraining objectives rather than forcing VLMs into traditional classification paradigms.