Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIP

1. Problem Statement

Vision-Language models (VLMs) like CLIP are foundational to modern AI, used in retrieval, zero-shot classification, and as encoders for Large Vision-Language Models (LVLMs). However, their reliance on massive, web-scraped datasets makes them highly vulnerable to backdoor attacks.

• The Threat: Attackers inject a small fraction of poisoned data (e.g., 0.01% to 0.5%) containing a specific "trigger" (e.g., a noise patch or text overlay) and a target label. When the trigger is present at inference, the model outputs the target label regardless of the actual content.
• The Challenge: Retraining these massive models from scratch to remove backdoors is computationally prohibitive. Therefore, fine-tuning with a small clean dataset is the preferred defense.
• The Vulnerability of Existing Defenses: Current state-of-the-art cleaning methods for CLIP, such as CleanCLIP and RoCLIP, rely heavily on strong data augmentations (e.g., AutoAugment, color jittering, heavy geometric transformations) to break the correlation between the trigger and the target label.
  • The Flaw: These methods assume triggers are random noise or unstructured patterns that get distorted by augmentation. The authors demonstrate that structured triggers (e.g., stripes, triangles, specific text overlays) are robust against these augmentations. Consequently, existing defenses fail to remove backdoors using these structured patterns while often degrading the model's standard performance (Clean Accuracy).

2. Methodology: Perturb and Recover (PAR)

The authors propose PAR, a fine-tuning-based defense mechanism that does not rely on strong augmentations or prior knowledge of the trigger. Instead, it uses a controlled perturbation strategy to "unlearn" the backdoor.

Core Mechanism

PAR operates on the hypothesis that if the model's embeddings are pushed sufficiently far away from the poisoned state, the spurious correlations (backdoor) will be destroyed. The method minimizes a composite loss function:
$$\mathcal{L}_{\text{PAR}} = \mathcal{L}_{\text{CLIP}} - \mathcal{L}_{\text{PERT}}$$

1. Perturbation Term ($\mathcal{L}_{\text{PERT}}$):

   • The goal is to maximize the distance between the embeddings of the cleaned model ($\phi, \psi$) and the poisoned model ($\phi_P, \psi_P$).
   • It calculates the $L_2$ distance between the features of the current model and the frozen poisoned model on a batch of clean data.
   • A threshold $\tau$ is applied: the loss only penalizes the model if the distance is below $\tau$. Once the model is sufficiently perturbed (distance $> \tau$), this term stops pushing, preventing over-perturbation that would destroy useful knowledge.
   • This forces the model to "forget" the specific trigger-target mapping learned during poisoning.

2. Recovery Term ($\mathcal{L}_{\text{CLIP}}$):

   • The standard CLIP contrastive loss is applied to ensure the model retains its general vision-language capabilities (standard performance) on clean data.

3. Data Requirements:

   • PAR requires a small set of clean image-text pairs (e.g., 250k samples).
   • Synthetic Data: A key finding is that PAR works effectively even when the clean data is synthetic (generated via text-to-image diffusion models), removing the need for costly, curated real-world clean datasets.

4. Augmentation:

   • Unlike CleanCLIP, PAR uses only very mild augmentations (Gaussian noise and small CutOut) that do not significantly distort image content, ensuring the text captions remain semantically valid.

3. Key Contributions

1. Exposing a Critical Vulnerability: The authors demonstrate that existing augmentation-based defenses (CleanCLIP, RoCLIP) are easily bypassed by structured triggers (e.g., BadNet-Stripes, Blended-Text). These triggers remain effective even after heavy augmentation, leading to high Attack Success Rates (ASR) and significant degradation in Clean Accuracy (CA) for existing methods.
2. Proposal of PAR: Introduction of a simple, effective fine-tuning scheme that removes backdoors by actively perturbing the model away from the poisoned state while recovering standard performance, without needing to know the trigger type.
3. Synthetic Data Viability: Demonstration that synthetic data (SynthCLIP) is sufficient to clean poisoned CLIP models, offering a cost-effective solution for scenarios where real clean data is unavailable.
4. Comprehensive Evaluation: Extensive experiments across different architectures (ResNet50, ViT-B/32, ViT-L/14, SigLIP), attack types (BadNet, Blended, WaNet, BadCLIP), and downstream tasks (Zero-shot classification, Retrieval).

4. Experimental Results

The paper evaluates PAR against CleanCLIP and RoCLIP across multiple metrics:

• Backdoor Removal (ASR):
  • Structured Triggers: For attacks like BadNet-Stripes and Blended-Text, CleanCLIP and RoCLIP fail to reduce the ASR significantly (often remaining >40-60%). In contrast, PAR reduces ASR to near 0% (e.g., 0.0% - 6.3%) across almost all structured attacks.
  • Random Triggers: PAR also performs well on standard random noise attacks, often outperforming baselines.
• Standard Performance (Clean Accuracy):
  • Existing methods often suffer a trade-off: reducing ASR leads to a drop in Clean Accuracy.
  • PAR maintains high Clean Accuracy (comparable to the original poisoned model or slightly lower but significantly better than RoCLIP). For example, on ImageNet with BadNet-Stripes, PAR achieves ~53% CA with 6.3% ASR, whereas CleanCLIP achieves ~53% CA but with 62.3% ASR.
• Robustness:
  • PAR generalizes across different encoder architectures (ResNet, ViT) and model variants (SigLIP).
  • Adaptive Attacks: Even when an attacker re-poisons a model cleaned by PAR, PAR successfully cleans the re-poisoned model without falling back to the original backdoor state.
• Synthetic Data: Using 250k synthetic samples, PAR achieves ASR reduction comparable to using real CC3M data, proving the feasibility of synthetic cleaning.

5. Significance and Impact

• Security for Foundation Models: As CLIP and similar models are increasingly used as the "eyes" for LVLMs (like LLaVA), ensuring their integrity is critical. This paper provides a practical, fine-tuning-based solution that is feasible for large-scale models where retraining is impossible.
• Paradigm Shift in Defense: The work challenges the prevailing assumption that "strong augmentation" is the silver bullet for backdoor removal in multi-modal models. It highlights that structured triggers require a different approach (perturbation) rather than just distortion.
• Cost-Effective Deployment: By showing that synthetic data suffices for cleaning, the paper lowers the barrier to entry for securing VLMs in real-world deployments where access to large, verified clean datasets is a bottleneck.
• Open Source: The authors provide code and models, facilitating further research and adoption of robust backdoor defenses.

In summary, PAR offers a robust, trigger-agnostic, and computationally efficient method to sanitize poisoned CLIP models, addressing a critical gap in the security of vision-language foundation models.