💬 NLP

Pay Less Attention to Function Words for Free Robustness of Vision-Language Models

This paper proposes Function-word De-Attention (FDA), a method that mitigates cross-modal adversarial vulnerabilities in Vision-Language Models by differentially subtracting function-word attention from original attention, thereby achieving significant robustness improvements with negligible or even positive impacts on downstream task performance.

Original authors: Qiwei Tian, Chenhao Lin, Zhengyu Zhao, Chao Shen

Published 2026-04-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qiwei Tian, Chenhao Lin, Zhengyu Zhao, Chao Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot assistant that can look at a picture and understand what you're saying about it. This robot is a Vision-Language Model (VLM). It's great at connecting images to words.

However, there's a problem. Just like a human can be tricked by a magician's sleight of hand, these robots can be tricked by "adversarial attacks." These are tiny, almost invisible changes to an image that make the robot completely misunderstand what it's seeing.

The researchers in this paper discovered a funny secret: The robot gets distracted by the "boring" words.

The Problem: The Robot's "Noise"

Think of a sentence like a movie scene.

Content words (like "dog," "running," "park") are the main actors. They carry the story.
Function words (like "the," "is," "and," "of") are the stagehands. They hold the set together, but they don't do much acting.

The researchers found that when hackers try to trick the robot, they sneak their "poison" into those boring stagehand words. Because the robot pays attention to everything equally, it gets confused by the noise in the words "the" and "is," causing it to ignore the actual picture of the dog.

The Solution: The "Mute Button" for Boring Words

The team invented a new technique called FDA (Function-word De-Attention).

Imagine you are trying to listen to a friend in a crowded, noisy room.

The Old Way: You try to listen to everyone in the room, including the people shouting nonsense. You get overwhelmed and can't hear your friend.
The FDA Way: You put on special noise-canceling headphones that specifically mute the people shouting "um," "uh," and "the." You don't stop listening to your friend; you just stop listening to the background chatter that isn't important.

In technical terms, the robot calculates what it should be looking at (the whole sentence) and then subtracts what it doesn't need to look at (the function words). It's like taking a photo and using a filter to blur out the background clutter so the main subject pops out clearly.

Why is this "Free" Robustness?

Usually, making a robot more secure (robust) makes it dumber (less accurate). It's like putting a heavy helmet on a runner; they are safer from falling, but they run slower.

This method is special because it's "Free Robustness."

The Helmet: The robot becomes much harder to trick (it stops falling for the magic tricks).
The Speed: The robot doesn't slow down at all. In fact, because it's not distracted by the noise, it sometimes runs faster and gets better scores!

The Results: A Magic Trick

The researchers tested this on three different robot brains and two different jobs (finding images based on text, and finding objects in pictures).

The Attack: They tried to trick the robots with six different types of "magic spells" (attacks).
The Defense: The robots with the "Mute Button" (FDA) ignored the spells almost completely.
- On one task, the robots became 90% harder to trick.
- On another, they became 50% harder to trick.
The Performance: The robots didn't lose any ability to do their actual jobs. They were just as smart as before, but now they were unshakeable.

The Big Picture

Think of this like teaching a child to read. If you tell the child, "Don't get distracted by the words 'the' and 'a', focus on the nouns and verbs," they become a much better reader. They stop getting confused by the tiny words and start understanding the real story.

This paper shows that by simply telling our AI to pay less attention to the boring words, we can make them incredibly strong against hackers without making them any less smart. It's a simple, clever trick that makes our digital friends much safer.

1. Problem Statement

Vision-Language Models (VLMs) are increasingly vulnerable to cross-modal adversarial attacks, where slight perturbations to input images or texts cause the model to generate incorrect outputs. Existing defense mechanisms, particularly Adversarial Training (AT), often suffer from a significant trade-off: they improve robustness at the cost of degraded clean performance (accuracy on unperturbed data) and high computational overhead.

The authors hypothesize that this vulnerability stems from function words (e.g., "the," "is," "of," "and"). These words carry minimal semantic information but have high frequency. In the context of cross-modal alignment, function words create "semantic gaps" between clean images and targeted texts, making them easier targets for adversarial perturbations to exploit. The paper argues that these function words act as distractions that weaken the model's alignment, allowing attacks to succeed.

2. Methodology: Function-word De-Attention (FDA)

The proposed solution is Function-word De-Attention (FDA), a plug-and-play module designed to refine vision-language alignment by explicitly reducing the model's attention to function words.

Core Concept: Inspired by Differential Transformers, FDA operates by calculating the difference between the standard cross-attention and the attention specifically focused on function words.
Mechanism:
1. Parallel Pipeline: Within the fusion encoder's multi-head attention mechanism, a parallel pipeline is deployed alongside the standard calculation.
2. Function Word Extraction: A mask ( $M_{Tf}$ ) is applied to the input text to isolate only function words (based on a predefined dictionary), creating a feature representation $F_{Tf}$ .
3. Distraction Calculation: The model calculates cross-attention scores between these function words and the visual features ( $F_V$ ). These scores represent "distractions" or misleading alignments.
4. Softmax Highlighting: Softmax is applied along visual and textual dimensions to highlight the most misleading tokens.
5. Differential Subtraction: The calculated distraction attention is subtracted from the original attention scores. A control gate ( $G$ ) modulates the intensity of this subtraction. The final attention score is the minimum of the original attention minus the distraction (from either text or visual perspective).
6. Implementation: FDA is applied to specific layers and attention heads (typically early layers $L_{0-1}$ and the first half of heads $H_{0-5}$ ) to avoid disrupting contextual integrity while removing noise early in the processing pipeline.

3. Key Contributions

Novel Hypothesis & Identification: The paper identifies function words as a primary source of vulnerability in VLMs. Experiments show that removing function words reduces Attack Success Rate (ASR) significantly while causing the least drop in clean performance compared to removing verbs or adjectives.
FDA Method: Proposes the first method to achieve "free robustness" (robustness without performance trade-off) by refining alignment through differential subtraction of function-word attention.
Comprehensive Evaluation: Extensive experiments across 3 models (ALBEF, TCL, BLIP), 2 downstream tasks (Text-to-Image/Image-to-Text Retrieval, Visual Grounding), 3 datasets (Flickr30k, COCO, RefCOCO+), and 6 attack types (PGD, APGD, MAPGD).
Zero-Shot & Generalization: Demonstrates that FDA improves zero-shot performance and generalizes well across different model backbones and attack scenarios.

4. Experimental Results

The results demonstrate that FDA significantly outperforms State-of-the-Art (SOTA) defenses like TeCoA and FARE, particularly in maintaining clean performance while drastically reducing ASR.

Retrieval Tasks (Flickr30k & COCO):
- Robustness: FDA achieved an average ASR drop of 18% (ALBEF), 13% (TCL), and 53% (BLIP) across all attacks.
- Performance: Unlike baselines which suffered 4–9% drops in clean accuracy, FDA caused negligible performance drops (0.2% – 0.6%).
- Scalability: The robustness gain scaled with model size; BLIP (124M pre-trained images) saw a 54% increase in $\Delta$ ASR compared to smaller models.
Visual Grounding (RefCOCO+):
- FDA achieved an ~90% ASR drop (e.g., 93.16% for $\epsilon=2/255$ ) while actually improving clean performance by 0.3% compared to the vanilla model.
Untargeted Attacks: When combined with TeCoA or FARE, FDA further boosted their robustness against untargeted attacks, acting as a complementary defense.
Ablation Studies:
- Layer Selection: Applying FDA to shallow layers ( $L_{0-1}$ ) and the first half of attention heads ( $H_{0-5}$ ) yielded the best results, preventing "absorption" of distractions in deeper layers.
- Dictionary: A shortlisted dictionary of 91 crucial function words performed slightly better than a full stopwords list, suggesting context distortion from overly broad masking.
- Comparison: Directly masking function words (without differential subtraction) hurt performance. FDA's differential approach is crucial for preserving semantic integrity.

5. Significance and Conclusion

"Free Robustness": The paper challenges the conventional wisdom that robustness requires a performance trade-off. By targeting the specific redundancy of function words, FDA secures the model without sacrificing accuracy.
Mechanism Insight: The work provides a new perspective on VLM vulnerabilities, suggesting that adversarial attacks exploit low-information, high-frequency tokens to break cross-modal alignment.
Practicality: FDA is computationally efficient (parallel calculation with subtraction) and requires no retraining of the backbone, making it a "plug-and-play" solution for existing VLMs.
Future Work: The authors note that while FDA works on fusion-encoder models (like ALBEF/BLIP), extending it to projector-based models (like LLaVA) and optimizing the function word dictionary are promising directions for future research.

In summary, this paper presents a highly effective, low-cost defense mechanism that leverages the semantic redundancy of function words to fortify Vision-Language Models against adversarial attacks, achieving superior robustness with minimal to no loss in clean task performance.