One Surrogate to Fool Them All: Universal, Transferable, and Targeted Adversarial Attacks with CLIP

Imagine the internet is a giant, bustling city where computers act as the police, doctors, and shopkeepers. These computers use a special kind of brain called a Deep Neural Network (DNN) to make decisions, like recognizing a cat in a photo or diagnosing a sickness. For a long time, scientists discovered a weird trick: you can trick these computer brains with "adversarial attacks." It's like putting a tiny, almost invisible sticker on a stop sign that makes the computer think it's a speed limit sign. Usually, to pull off this trick, a hacker needs to either peek inside the computer's secret code (which is impossible in the real world) or ask the computer thousands of questions to figure out its secrets (which is too slow and expensive).

But what if you could trick any computer brain without ever asking it a single question? That's the big question this paper tackles. The researchers used a powerful tool called CLIP, which is like a super-smart librarian that has read billions of books and seen billions of pictures, learning how words and images connect. They wanted to see if this "universal librarian" could be used to create a master key—a single, tiny digital distortion—that could fool almost any computer vision system, from Google Search to high-tech robots, just by using simple words like "hen" or "cat" as a guide.

The Master Key: One Surrogate to Fool Them All

Meet UnivIntruder, a new attack framework introduced by Binyan Xu and their team. Think of it as a "universal master key" for computer vision. In the past, if a hacker wanted to trick a specific computer system, they had to build a fake copy of that system (a "surrogate model") using the same data the real system was trained on. It was like trying to pick a specific lock by first building a perfect replica of the lock's keyhole using the same metal and screws. If you didn't have the original lock's blueprints or the exact metal, your fake key wouldn't work.

UnivIntruder changes the game. The researchers realized they didn't need a perfect copy of the target. Instead, they used CLIP, a publicly available, super-smart AI that understands the relationship between images and text. They treated CLIP as their "surrogate" or practice lock. Here's the magic: they didn't try to copy the target's training data. Instead, they used a public dataset of random images (which might look nothing like the target's data) and simple text concepts.

How the Trick Works:
Imagine you want to trick a security camera into thinking a picture of a squirrel is actually a "hen."

1. The Text Guide: You tell the system, "I want this to look like a hen," and "I don't want it to look like a cat or a horse."
2. The Directional Shift: Instead of just looking at the picture, the system looks at the difference between the picture and the text. It calculates a "direction" in the computer's brain. It asks, "If I nudge this squirrel picture in this specific direction, will it move closer to the 'hen' concept and further away from the 'cat' concept?"
3. The Universal Sticker: The system creates a tiny, universal "sticker" (a perturbation) that, when added to any image, pushes it in that direction.
4. The Twist: To make sure this sticker works on any camera, not just the one they practiced on, they spin, zoom, and crop the images while creating the sticker. This forces the sticker to be robust, like a sticker that still works even if the car drives through a car wash or gets covered in mud.

The Results: Fooling the World

The team tested this "master key" on a massive scale. They didn't just test it on one computer; they tested it on 85 different models and real-world applications.

• The Numbers: On standard test sets, their attack was terrifyingly effective. On the CIFAR-10 dataset (a common test for image recognition), they achieved a success rate of 99.4%. On the massive ImageNet dataset (with 1,000 different categories), they hit 85.1%.
• The Real World: This is where it gets scary. They didn't just stop at lab tests. They tried to fool real services:
  • Search Engines: They uploaded a perturbed image of a monkey to Google, Baidu, Taobao, and JD. The search engines, instead of showing monkeys, started showing them chickens (hens). On Baidu, the success rate was 84%.
  • Super-AI: They tested it on GPT-4 and Claude-3.5, the most advanced AI chatbots. When shown a perturbed image of a monkey, these AIs confidently described it as a "hen" or a "chick." The success rate on Claude-3.5 was 80%.
  • Image Generators: Even AI that draws pictures, like DALL·E 3, was fooled. When asked to generate an image based on the perturbed input, it produced images of chickens instead of monkeys.

Why Other Methods Failed

The paper is very clear about what doesn't work. Previous methods tried to use CLIP as a helper, but they failed because they didn't account for the "bias" of the data.

• The Bias Problem: If you train a fake lock using a dataset that is slightly blue-tinted, your key might only work on blue-tinted locks. The researchers found that by using "directions" (subtracting the original image's features from the new ones) instead of raw features, they canceled out these biases.
• The Surrogate Problem: Other methods tried to use CLIP but failed to transfer the attack to other models. The authors showed that without their specific "directional" and "random transformation" tricks, even using CLIP resulted in failure rates of over 70% on advanced models. UnivIntruder is the only method that successfully bridged the gap between the public CLIP model and the diverse, unknown models used in the real world.

What This Means for Security

The paper suggests that our current idea of security might be broken. Usually, we think if we hide our training data and limit how many questions people can ask us, we are safe. UnivIntruder proves that an attacker can bypass these defenses entirely. They don't need your data, and they don't need to ask you questions. They just need a public AI (CLIP) and a few words.

The researchers also tested if we can stop this. They tried "adversarial training" (teaching the computer to resist tricks) and "test-time defenses" (cleaning the image before it's processed). While these methods helped a little, they often made the computer worse at its actual job (like recognizing real cats) or were too expensive to run on huge systems like GPT-4. The paper concludes that we need to rethink how we secure AI, because a single, publicly available model can now be used to fool almost anything.

In short, UnivIntruder is a wake-up call. It shows that with the right "universal key," the digital world's vision systems are far more fragile than we thought, and the only thing standing between a monkey and a chicken in the eyes of an AI might be a tiny, invisible digital glitch.