💻 computer science

Backbone is All You Need: Assessing Vulnerabilities of Frozen Foundation Models in Synthetic Image Forensics

This paper introduces the Surrogate Iterative Adversarial Attack (SIAA), a gray-box method demonstrating that knowledge of a Vision Transformer backbone alone is sufficient to effectively compromise frozen foundation models used in synthetic image forensics, thereby exposing a critical vulnerability in current deepfake detection systems.

Original authors: Chiara Musso, Joy Battocchio, Andrea Montibeller, Giulia Boato

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chiara Musso, Joy Battocchio, Andrea Montibeller, Giulia Boato

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: "The Backbone is All You Need"

Imagine you are trying to spot a fake painting. You hire a famous art expert (the Detector) to look at a picture and tell you if it's real or AI-generated.

For a long time, these experts were trained from scratch. But recently, researchers started using "pre-trained" experts. These are like art experts who have already studied millions of paintings in a museum (the Frozen Foundation Model or ViT Backbone) and know how to recognize shapes, textures, and patterns very well. To save time and money, we don't retrain their whole brain; we just give them a simple "Yes/No" test (a Classification Head) to decide if a specific new picture is fake.

The Paper's Discovery:
The authors of this paper found a dangerous secret: You don't need to know the expert's final "Yes/No" test to trick them. You only need to know how their brain processes the image (the backbone).

They created a new trick called SIAA (Surrogate Iterative Adversarial Attack). It's a "gray-box" attack, meaning the attacker doesn't know the secret sauce of the detector's final decision, but they do know the detector's underlying "brain" (the Vision Transformer).

How the Attack Works (The Analogy)

Think of the detector's brain as a massive library where every image is turned into a specific "book" (a feature vector).

The Setup: The detector has a library of "Real" books and "Fake" books. When you show it a picture, it finds the matching book and checks the label.
The Problem: Usually, to trick the detector, you need to know exactly how it reads the labels (the classification head).
The SIAA Solution: The attackers built a Surrogate Translator (the FP-Head).
- They take the "Real" and "Fake" books from the detector's library.
- They also use a text AI (CLIP) that knows the words "This is real" and "This is fake."
- They train their Translator to match the "Real" books with the text "This is real," and the "Fake" books with "This is fake."
The Trick: Once the Translator is ready, the attacker takes a fake image and adds tiny, invisible scratches (perturbations) to it. They tweak these scratches until the Translator says, "Hey, this fake image now looks exactly like the text 'This is real' inside the detector's library!"
The Result: The detector looks at the scratched image, sees the "Real" pattern in its library, and confidently says, "This is a real photo!" even though it's fake.

What They Tested

The researchers tested this trick on three different types of "expert brains" (CLIP, Swin, and DINOv2) and in several difficult situations:

The "Few-Shot" Test: What if the attacker only has 100 pictures to learn from instead of 10,000?
- Result: The attack still worked almost perfectly. The attacker didn't need a huge library to learn the trick.
The "Mismatched" Test: What if the attacker trains on a different set of pictures or uses different filters (data augmentation) than the detector?
- Result: Even when the attacker was working with different data than the detector, the attack still succeeded in fooling the detector more than 70% of the time.
The "Different Brain" Test: What if the attacker trains on a CLIP brain but tries to trick a Swin brain?
- Result: It worked surprisingly well between CLIP and Swin. However, the DINOv2 brain was much harder to trick. It's like a fortress with smoother walls; the "scratches" the attacker added didn't slide off as easily.

The Main Takeaway

The paper concludes that frozen pre-trained models are surprisingly fragile.

Even if you hide the final decision-making part of the detector (the "Yes/No" button), simply knowing the "brain" that processes the image is enough to break the system. The attackers didn't need to see the detector's final answer; they just needed to know how the detector sees the image.

In short: If you build a fake-image detector using a standard, frozen AI brain, you are vulnerable. An attacker who knows that specific brain can create invisible "noise" that makes the detector think a fake image is real, even without knowing the detector's secret final rules.

What This Means for Security

The paper warns that relying on these frozen backbones creates a "single point of failure." If the backbone is known, the whole detector can be bypassed. The authors suggest we need to build detectors that are more resilient to these specific types of "brain-only" attacks.

Technical Summary: Backbone is All You Need

Problem Statement

The rapid advancement of AI-generated synthetic images has necessitated the development of robust detection systems. While Vision Transformers (ViTs) have become the cornerstone of modern deepfake detection due to their ability to generalize across unseen generator classes, a critical vulnerability remains largely unexplored. Most current detectors rely on frozen, pre-trained ViT backbones with only a shallow classification head trained for the specific task.

Existing adversarial research has primarily focused on white-box attacks (full access to model weights and logits) or transferability between different architectures. However, the specific vulnerability of gray-box attacks—where an adversary possesses knowledge of the frozen ViT backbone but lacks access to the target detector's classification head, training data, or specific augmentation strategies—has been overlooked. This paper posits that knowledge of the backbone alone may be sufficient to compromise detector reliability.

Methodology: Surrogate Iterative Adversarial Attack (SIAA)

The authors propose the Surrogate Iterative Adversarial Attack (SIAA), a gray-box attack designed to operate entirely within the feature space of the target detector. The attack assumes knowledge only of the pre-trained ViT backbone and its weights.

The methodology is implemented in two distinct phases:

Phase 1: Training the Surrogate Feature-Processing (FP) Head

To bridge the gap between the unknown target classification head and the known backbone, the authors train a surrogate model consisting of:

Frozen Components: The target ViT backbone ( $F(\cdot)$ ) and the CLIP text encoder ( $G(\cdot)$ ).
Trainable Component: A Feature-Processing (FP) Head ( $H(\cdot)$ $H (\cdot)$ ) comprising two linear branches:
- $L_t$ : Processes text embeddings derived from prompts (e.g., "The image is real" vs. "The image is fake").
- $L_v$ : Processes visual features extracted by the ViT backbone.

The FP-Head is trained using a multi-term contrastive loss to align visual features ( $v'$ ) and textual features ( $t'$ ) into a shared embedding space. The objective is to approximate the decision boundaries of the unknown target classification head by clustering visual features around their corresponding textual embeddings. The loss function includes:

$L_{pull}$ : Minimizes distance between visual features and ground-truth text embeddings.
$L_{push}$ : Enforces a margin against the adversarial class text embeddings.
$L_{text}$ : Ensures separation between the two text embedding clusters.

Phase 2: Deployment and Perturbation Optimization

Once the FP-Head is trained, the attack generates adversarial examples without accessing the target's logits.

Objective Function: Instead of maximizing cross-entropy on logits, SIAA optimizes a custom distance-based objective $J$ in the surrogate feature space:
$J = \|v' - t'_y\|^2 - \|v' - t'_{1-y}\|^2$
Maximizing $J$ pushes the visual representation away from the ground-truth class and toward the adversarial target class.
Optimization: A Projected Gradient Descent (PGD) algorithm iteratively updates the input image $I_j$ to maximize $J$ , subject to an $\ell_\infty$ -norm constraint ( $\epsilon = 8/255$ ) to ensure imperceptibility.

Key Contributions

Novel Attack Vector: Introduction of SIAA, the first gray-box attack specifically tailored to exploit frozen ViT backbones in synthetic image forensics without requiring knowledge of the classification head.
Feature-Space Exploitation: Demonstration that adversarial perturbations can be effectively optimized in a surrogate feature space aligned via text-image contrastive learning, bypassing the need for logit access.
Comprehensive Evaluation: Systematic assessment of the attack under varying levels of gray-box knowledge, including few-shot learning scenarios, unknown data augmentations, and dataset mismatches.
Transferability Analysis: Evaluation of cross-model transferability across different ViT backbones (CLIP, Swin, DINOv2).

Experimental Results

The authors evaluated SIAA against detectors based on CLIP, Swin, and DINOv2 backbones using the Synthbuster dataset.

Performance vs. Baselines: SIAA significantly outperformed existing gray-box baselines (PGD-ViT and a linear variant PGD-L).
- Against CLIP and Swin detectors, SIAA achieved Attack Success Rates (ASR) of 1.00 and 0.99 respectively, with post-attack AUCs near 0.00, effectively matching the performance of white-box PGD attacks.
- Against DINOv2, SIAA achieved an ASR of 0.56. The authors attribute DINOv2's higher resilience to its self-supervised training objectives and the use of register tokens, which create a smoother loss landscape.
Robustness to Misalignment (Ablation Studies):
- Classification Head Variations: SIAA maintained ASR $\geq 0.98$ even when the target detector's classification head depth varied (1, 2, or 3 layers).
- Few-Shot Learning: With only 100 real and 100 fake training samples, SIAA achieved an ASR of 0.985.
- Unknown Augmentation: When data augmentation strategies were unknown, ASR remained high at 0.96.
- Dataset Mismatch: In the most restrictive scenario (different training dataset and unknown augmentation), SIAA still achieved an ASR of 0.70, successfully attacking over 50% of correctly classified images.
Transferability:
- Perturbations optimized on CLIP or Swin transferred well to each other (ASR $\approx 0.56$ ).
- Perturbations optimized on DINOv2 showed moderate transferability to CLIP (ASR 0.31) and Swin (ASR 0.28), despite DINOv2 being highly resistant as a target.

Significance and Claims

The paper claims that the reliance on frozen foundation models in synthetic image forensics introduces a "subtle yet critical vulnerability." The primary finding is that knowledge of the backbone alone is sufficient to undermine detector reliability, often yielding attack success rates comparable to white-box attacks.

The authors emphasize that this vulnerability persists even under conditions of total misalignment regarding training data, augmentation, and classification head architecture. Consequently, the paper argues for an urgent need to develop more resilient defenses that do not rely solely on the assumption that hiding the classification head provides adequate security. The work highlights that the feature space of frozen ViTs, when aligned with semantic prompts, is predictable enough to be exploited by adversaries with limited access.