🤖 machine learning

Enhancing Adversarial Transferability through Block Stretch and Shrink

This paper proposes FRO, a frontend response-oriented input transformation method that enhances adversarial transferability by employing block-wise stretch-and-shrink and perspective deformation operators to enrich model frontend responses through an implicit ensemble perspective.

Original authors: Quan Liu, Feng Ye, Chenhao Lu, Shuming Zhen, Guanliang Huang, Lunzhe Chen, Xudong Ke

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Quan Liu, Feng Ye, Chenhao Lu, Shuming Zhen, Guanliang Huang, Lunzhe Chen, Xudong Ke

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to sneak a tiny, invisible sticker onto a security camera's lens. If you put it in just the right spot, the camera might get confused and think a stop sign is a speed limit sign. This is what hackers do with "adversarial attacks" on AI. But here's the tricky part: if you design your sticker to fool Camera A, it might not work on Camera B, even if they look the same.

For a long time, researchers tried to make these stickers work on any camera by taking the image, twisting it, stretching it, or spinning it in random ways before showing it to the AI. The idea was, "If we show the AI a bunch of different versions of the sticker, maybe it will learn a trick that works on everyone."

But a new paper by Quan Liu and his team suggests we've been looking at this the wrong way. They argue that simply making the image look different isn't the magic key. Instead, the real secret lies in how the AI's "front door" (its early processing stage) reacts to those changes.

The "Front Door" Mystery

Think of an AI model like a two-story house.

The Front Door (Frontend): This is where the AI first sees the image. It's like a bouncer checking IDs. It looks at shapes, edges, and textures.
The Living Room (Backend): This is where the AI makes its final decision, like "That's a cat!" or "That's a dog!"

Previous studies thought that if you just made the image look weird (diverse), the bouncer would get confused, and the trick would work on other houses too. But Liu's team says, "Wait a minute." Just because the image looks different to you doesn't mean the bouncer sees it differently. If the bouncer at House A and the bouncer at House B both react the same way to a twisted image, your trick fails.

The paper suggests that to make a sticker that works on everyone, you need to mess with the bouncer's reaction in a specific way. You need to create "Frontend Response" changes that are unique enough to be interesting, but similar enough that other bouncers will also notice them.

Enter FRO: The Double-Acting Trickster

To solve this, the team created a new method called FRO (Frontend Response-Oriented). Instead of just randomly twisting the image, FRO uses two specific moves to poke the AI's front door in just the right spots:

The Local Scaling Operator (The "Stretchy Block" Move): Imagine the image is made of Lego blocks. This operator grabs a few blocks here and there and stretches them out or shrinks them down. It changes the local texture and the size of small details without ruining the whole picture. It's like zooming in on a specific part of a face to see the pores, then zooming out again, but doing it to just one cheek.
The Projection Operator (The "Funhouse Mirror" Move): This one takes the whole image and gives it a gentle, coherent perspective shift, like looking at a building from a weird angle. It changes the global shape and how things are arranged in space, but it keeps the whole scene looking like a single, logical view.

By combining these two moves, FRO creates a set of "transformed views." It's like asking the AI, "What if this cat's ear was stretched? What if the whole cat was leaning sideways?" The AI has to process all these slightly different versions and combine its reactions to figure out the best way to trick it.

Did It Work? The Numbers Don't Lie

The team tested this on a subset of the famous ImageNet dataset. They didn't just guess; they ran the numbers.

The Setup: They used a "white-box" model (a model they could see inside) to generate the attacks, and then tested them on "black-box" models (models they couldn't see inside).
The Competition: They compared FRO against other popular methods like DeCoWA, CWT, SIA, SID, and OPS.
The Result: FRO consistently beat the others. When tested on a mix of different AI architectures (including CNNs and Vision Transformers), FRO achieved the highest success rates in 6 out of 11 target models.
- For example, against a model called ResNet-50, FRO hit a 95.6% success rate, while the next best was 93.2%.
- Against Inception-v3, it hit 96.3%, beating the runner-up at 96.2%.
- Even against tricky, defended models (AI that has been trained to resist attacks), FRO managed to break through 96.2% of the time against one specific defended model, outperforming everyone else.

What They Ruled Out

The paper is very clear about what doesn't work as the main explanation.

It's not just about "Image Diversity": The authors argue that just making the image look different isn't enough. If the AI's front door doesn't react differently, the trick fails.
It's not about "Semantic Preservation" alone: While keeping the meaning of the image (like keeping a cat looking like a cat) is important, it's not the reason the attack works. The reason is how the front door handles the changes.
It's not about "Hypothesis Space Augmentation": They suggest that simply adding more possibilities to the math isn't the key; the key is the specific response of the front door.

How Sure Are They?

The paper is quite confident, but they are careful with their words. They demonstrated these results through experiments on an ImageNet subset. They didn't just simulate it; they actually ran the attacks and measured the success rates.

They also introduced a new way to measure things called the "unified implicit ensemble size" (let's call it N). They found that the number of transformed views matters. When they tested different numbers (like N = 10, 17, 21, 26, 37, and 101), FRO stayed strong even with smaller numbers, suggesting its method is efficient.

The authors suggest that their approach works because it targets the "shared front-end factors" that both CNNs (the old-school AI) and ViTs (the new-school AI) use. They showed that FRO reduces the "gap" in how different AI models react to these transformations, making the attack more transferable.

The Bottom Line

This paper suggests that to hack an AI's brain, you shouldn't just throw random distortions at it. You need to understand how its "front door" processes information. By using a clever mix of local stretching and global perspective shifts, FRO creates a "universal key" that fits many different AI locks.

The authors propose that this "Frontend Response" perspective is the missing piece of the puzzle. While they haven't solved the problem of AI security forever (and they don't claim to), they have shown a significantly better way to craft these attacks, proving that understanding the AI's early reactions is the real secret sauce.

Technical Summary: Frontend Response-Oriented Input Transformation for Transferable Adversarial Attacks (FRO)

Problem Statement
Deep Neural Networks (DNNs) are vulnerable to adversarial examples, where imperceptible perturbations cause misclassification. While white-box attacks are effective, they often suffer from limited transferability to black-box target models. Input transformation-based attacks attempt to improve transferability by aggregating gradients over multiple transformed views of an input. However, existing explanations for their efficacy—such as image diversity, semantic preservation, attention variance, or hypothesis space augmentation—primarily focus on image-level characteristics or high-level model responses. These approaches overlook the critical role of the model frontend (early feature encoding). Visually distinct transformations do not necessarily induce useful or transferable differences after frontend encoding; if multiple views produce nearly identical frontend responses, the resulting gradients remain redundant. Furthermore, existing comparisons between transformation methods often lack fairness because they vary in the number of transformed views (ensemble size), conflating the benefits of transformation design with the benefits of simply increasing the ensemble scale.

Methodology
The paper proposes FRO (Frontend Response-Oriented input transformation), a method grounded in an implicit ensemble perspective.

Implicit Ensemble Formulation: The authors reinterpret input transformation-based attacks as a functional implicit ensemble. Given a source model $f_s$ and transformations $\{T_i\}$ , the attack optimizes over a set of composite members $\{f_s \circ T_i\}$ . Unlike conventional ensembles that use different model parameters, this implicit ensemble uses the same source model but different pre-model operators. The diversity of this ensemble is determined not by model parameters, but by the frontend response diversity induced by the transformations.
Frontend Response Perspective: The authors argue that for transferability, transformations must induce response differences in the source frontend that are not purely source-specific but are partially shared with heterogeneous target architectures (e.g., CNNs and Vision Transformers).
Shared Frontend Factors: Through preliminary analysis, the authors identify two factors common to both CNN and ViT frontends that transformations should target:
- Local Content Sampling: How local scale, texture density, and boundary positions are presented to the receptive fields or patch tokens.
- Spatial Organization: Global geometry, object shape, and spatial layout.
FRO Operators: To perturb these factors, FRO employs two complementary operators:
- Local Scaling Operator: Partitions the image into blocks and applies random stretch-and-shrink operations to individual blocks. This alters local scale, texture density, and boundary positions while preserving the global image size.
- Projection Operator: Applies a globally coherent perspective deformation (tilt and focal length variations) to the entire image. This modifies global geometry, object shape, and spatial layout.
- Generation Process: At each iteration, $N$ transformed views are generated by sequentially sampling the Local Scaling and Projection operators. Gradients from these views are aggregated to update the adversarial perturbation.
Unified Implicit Ensemble Size ( $N$ ): To ensure fair comparisons, the authors define $N$ as the unified number of transformed views used for gradient aggregation. This allows performance differences to be attributed to transformation design rather than ensemble scale.
Evaluation Metric (T-FRDM): To quantify cross-architecture consistency, the paper introduces a Transformation-induced Front-end Response Dissimilarity Matrix (T-FRDM). Instead of comparing raw heterogeneous features, it compares the relations (dissimilarity structures) between response displacements of transformed views in CNN and ViT frontends. A smaller gap between these relation matrices indicates better preservation of response factors across architectures.

Key Contributions

Theoretical Reinterpretation: The paper formulates input transformation-based attacks as a transformed-input implicit ensemble and introduces the frontend response perspective to explain adversarial transferability, shifting focus from image diversity to how transformations are encoded by the model's early layers.
FRO Algorithm: The proposal of FRO, which combines Local Scaling and globally coherent Projection to induce complementary local and global frontend response variations.
Unified Evaluation Protocol: The introduction of a unified implicit ensemble size $N$ to decouple the effects of transformation design from ensemble scale, enabling fairer comparisons among input transformation-based attacks.
Analysis Tool: The development of a relation-based metric (T-FRDM) to measure cross-architecture frontend response consistency without aligning raw feature spaces.

Experimental Results
Experiments were conducted on an ImageNet subset using various source models (ResNet-18/101, DenseNet-121, Inception-v3, ViT-B/16) and target models (6 CNNs and 5 Vision Transformers).

Transferability: FRO consistently outperformed state-of-the-art input transformation-based attacks (DeCoWA, CWT, SIA, SID, OPS) across diverse source and target architectures. It achieved the best black-box results on 6 out of 11 target models and maintained strong performance even with smaller ensemble sizes (e.g., $N=10$ ).
Robustness to Defenses: FRO demonstrated superior transferability against defended models, including ensemble adversarial training, randomized smoothing (RS), feature squeezing (HGD), neural representation purifiers (NRP), adversarial training (AT), and diffusion-based purification (DiffPure). It achieved the highest success rate in 8 out of 9 defended settings.
Ablation Studies: The study confirmed that the specific hyperparameters for the Local Scaling and Projection operators (e.g., block size, tilt angle) significantly impact performance. It also validated that increasing $N$ generally improves performance but that FRO's design yields high efficiency even at lower $N$ .
Frontend Analysis: The T-FRDM analysis showed that FRO induces response relations in the source frontend that are more consistently preserved in the target frontend compared to other methods, supporting the paper's theoretical motivation.

Significance
The paper claims that the primary significance of FRO lies in its shift of perspective: rather than viewing input transformations merely as image augmentation or diversity generators, they should be designed to perturb frontend response factors that are relevant across heterogeneous architectures. By explicitly targeting local content sampling and spatial organization—factors shared by both CNNs and ViTs—FRO generates perturbations that are less prone to source overfitting. The work suggests that designing transformations from the perspective of front-end response ensembles is a more effective strategy for improving adversarial transferability than relying solely on image-level diversity or semantic preservation. The introduction of the unified ensemble size $N$ further provides a rigorous framework for future research in this domain.