FORCE: Transferable Visual Jailbreaking Attacks via Feature Over-Reliance CorrEction

This paper proposes FORCE, a feature over-reliance correction method that enhances the cross-model transferability of visual jailbreaking attacks against multimodal large language models by analyzing and mitigating their tendency to reside in high-sharpness regions caused by improper reliance on narrow layer representations and semantically poor frequency components.

The Gist

The Big Picture: The "Master Key" Problem

Imagine Multimodal Large Language Models (MLLMs) as incredibly smart, security-guard robots. They can read text and look at pictures. Their job is to be helpful but safe—they won't tell you how to build a bomb or hack a bank.

Researchers (the "Red Team") try to trick these robots into breaking their rules. They do this by adding tiny, invisible "noise" to an image (like static on an old TV) that the robot can't see, but which confuses its brain into saying, "Okay, here is how to build a bomb."

The Problem:
Currently, these "trick images" are like custom-made keys.

• If you make a key that opens Robot A's door, it usually won't open Robot B's door.
• The key is too specific to Robot A's internal mechanics. If you try to use it on Robot B, it just doesn't fit.
• This makes it hard to test if the new commercial robots (like GPT-5 or Claude) are safe, because we can't easily make a key that works on them without seeing their internal code.

The Discovery: Why the Keys Break

The authors of this paper investigated why these custom keys are so fragile. They found two main reasons the keys are "too specific":

1. The "Early Layer" Trap (The Shallow Roots):
   Think of the robot's brain as a multi-story building. The bottom floors (early layers) handle basic details like edges and colors. The top floors handle complex concepts like "bomb" or "poison."

   • The Issue: The trick images rely too heavily on the bottom floors. They exploit tiny, specific quirks in how Robot A sees a "red line" or a "sharp edge."
   • The Result: When you move to Robot B, who sees edges slightly differently, the trick fails immediately. It's like trying to open a door by picking a specific screw on the hinge; if the hinge is a different shape, the trick doesn't work.

2. The "High-Frequency" Addiction (The Static Noise):
   Images are made of frequencies. Low frequencies are the smooth shapes and colors (the "meaning"). High frequencies are the tiny, jagged details and static (the "noise").

   • The Issue: As the researchers tried to make the trick work better, the robot started relying more and more on the high-frequency noise (the static) rather than the actual meaning of the picture.
   • The Result: The robot is being tricked by "visual static" rather than the image itself. Since different robots handle static differently, the trick stops working when transferred.

The Solution: FORCE (The "Universal Key" Maker)

The authors created a new method called FORCE (Feature Over-Reliance CorrEction). Think of it as a Keysmith that reforges the custom key into a Universal Key.

FORCE does two things to fix the problems:

1. Deepening the Roots (Layer Correction):
   Instead of letting the trick rely on the bottom floors (the specific edges), FORCE forces the trick to find a solution that works on the top floors (the high-level concepts).

   • Analogy: Instead of picking a specific screw on the hinge, the keysmith designs a key that turns the main lock mechanism. This mechanism is similar in almost all robots, so the key works on everyone.

2. Cleaning the Static (Spectral Correction):
   FORCE looks at the "noise" in the image and says, "Stop relying on the static!" It dials down the high-frequency noise and forces the trick to rely on the meaningful parts of the image (the low frequencies).

   • Analogy: If a song is being played through a bad speaker with lots of static, the listener might get confused. FORCE turns down the static volume so the listener hears the actual melody. Since the melody is the same for everyone, the trick works on any robot.

The Result: A Flatter, Safer Landscape

By fixing these two issues, FORCE creates a "flatter" path to tricking the robot.

• Before: The path was a narrow, steep cliff. If you took one tiny step sideways (changed the robot slightly), you fell off.
• After: The path is a wide, flat plateau. You can walk around a bit, change the robot slightly, and you are still on the safe (or unsafe, in this case) ground.

Why This Matters

1. Better Safety Testing: Because these new "Universal Keys" work on many different robots, security experts can now test closed-source commercial robots (like the ones you might use at work) to see if they are truly safe, without needing to see their secret code.
2. Real-World Threat: It shows that visual attacks are becoming a serious threat. We can't just rely on text filters; we need to make sure robots can't be tricked by "invisible" picture noise.

Summary in One Sentence

The paper found that current tricks to fool AI robots are too fragile because they rely on tiny, specific details; the authors created a new method (FORCE) that forces the tricks to rely on the big-picture meaning, making them work on almost any robot, not just the one they were designed for.

Technical Summary

1. Problem Statement

Multimodal Large Language Models (MLLMs) have integrated visual capabilities, but this has introduced new vulnerabilities. While optimization-based visual jailbreaking attacks (e.g., Projected Gradient Descent, PGD) can successfully bypass safety guardrails in open-source MLLMs with imperceptible perturbations, they suffer from extremely limited cross-model transferability.

• The Core Issue: Attacks optimized on a source model (e.g., LLaVA) fail to jailbreak target models (e.g., InstructBlip, commercial APIs like GPT-5 or Claude).
• The Gap: Existing attacks rely on model-specific features, making them ineffective against closed-source models where parameters are inaccessible. This limits their utility for real-world red-teaming and safety evaluation.
• Hypothesis: The authors hypothesize that these attacks reside in high-sharpness regions of the loss landscape, making them highly sensitive to minor parameter changes during transfer.

2. Methodology: Feature Over-Reliance CorrEction (FORCE)

The authors propose FORCE, a method designed to correct the "over-reliance" on non-generalizable features. The approach is grounded in two key analyses of the attack's feature representations:

A. Analysis of Failure Modes

1. Layer-Space Analysis:
   • The authors analyzed intermediate layer features and found that successful attacks rely heavily on early-layer features (shallow layers).
   • These early layers exhibit narrow feasible regions; interpolating even a small amount of natural image features into the attack causes the loss to spike, rendering the attack ineffective.
   • In contrast, deeper layers show flatter, more robust feasible regions.
2. Spectral Domain Analysis:
   • During optimization, the attack's effectiveness increasingly depends on high-frequency components.
   • High-frequency features often lack semantic meaning (representing noise or texture) compared to low-frequency components.
   • The optimization process converges on these superficial, high-frequency patterns, which are specific to the source model's architecture and do not generalize.

B. The FORCE Algorithm

FORCE addresses these issues through two complementary mechanisms integrated into the standard PGD optimization loop:

1. Layer-Aware Regularization (Addressing Layer Over-Reliance):

   • Goal: Encourage the attack to explore broader feasible regions in early layers.
   • Mechanism: The method samples reference points within a noise neighborhood of the adversarial image. It maximizes the $L_2$ distance between the features of the adversarial image and these reference points across all layers.
   • Weighting: A decreasing regularization strength ($\lambda_l$) is applied, penalizing early layers more heavily than deep layers. This forces the attack to rely less on fragile, model-specific early features and more on robust, generalizable representations.
   • Constraint: The reference points must also successfully jailbreak the model (minimize loss) to ensure the expanded region is valid.

2. Spectral Rescaling (Addressing Frequency Over-Reliance):

   • Goal: Suppress the excessive influence of semantically poor high-frequency components.
   • Mechanism: The perturbation is transformed into the frequency domain (Fourier Transform). The spectrum is divided into bands.
   • Rescaling: If a high-frequency band's influence on the loss exceeds a scaled threshold relative to adjacent low-frequency bands, its magnitude is down-scaled.
   • Result: This restores a frequency distribution closer to natural images, forcing the attack to rely on semantically meaningful low-frequency content.

3. Key Contributions

1. Diagnosis of Transferability Failure: The paper identifies that visual jailbreaking attacks fail to transfer because they reside in high-sharpness loss landscapes caused by an improper reliance on early-layer features and high-frequency spectral components.
2. FORCE Method: A novel correction method that combines layer-aware regularization and spectral rescaling to guide attacks toward flatter, more generalizable feasible regions.
3. Comprehensive Evaluation: Extensive experiments across diverse MLLM architectures (Adapter-based vs. Early-fusion) and commercial APIs (GPT-5, Claude, Gemini), demonstrating significant improvements in transferability.

4. Experimental Results

The authors evaluated FORCE against standard PGD and other baselines on three benchmarks: MaliciousInstruct, AdvBench, and HADES.

• Transferability to Adapter-Based MLLMs:
  • FORCE improved the Attack Success Rate (ASR) by an average of 12.3% compared to PGD.
  • It reduced the query cost (number of attempts needed) by over 15%.
• Transferability to Early-Fusion MLLMs:
  • Standard PGD struggled significantly (93% failure rate).
  • FORCE achieved nearly a 100% relative improvement in ASR over the baseline, successfully jailbreaking models like LLaMA-3.2-Vision and Qwen2.5-VL.
• Commercial Models (Black-Box):
  • FORCE demonstrated consistent improvements on closed-source models (Claude-Sonnet-4, Gemini-2.5-Pro, GPT-5), achieving an average 70% relative improvement in ASR.
  • Notably, it achieved a 200% improvement on GPT-5 for specific tasks where the baseline failed almost entirely.
• Ablation Studies:
  • Both components (Layer and Frequency) contributed independently, but their combination yielded a synergistic effect (20.6% total improvement over baseline).
  • The method remained robust under blank image initialization and zero-shot transfer settings.
• Efficiency: The computational overhead is negligible, with only a slight increase in memory usage and optimization time.

5. Significance and Impact

• Red-Teaming Utility: FORCE provides a practical tool for evaluating the safety of closed-source MLLMs without requiring access to their internal parameters. It bridges the gap between open-source research and commercial model safety.
• Theoretical Insight: The work shifts the understanding of adversarial attacks in MLLMs from a purely pixel-space optimization problem to a feature-space representation problem. It highlights that "sharp" minima in the loss landscape are the primary cause of poor transferability.
• Future Directions: The paper suggests that future safety research must account for the specific feature dependencies of different model architectures (e.g., early-fusion vs. adapter-based) and that improving transferability is key to robust safety evaluation.

In summary, FORCE effectively transforms fragile, model-specific visual attacks into robust, transferable threats by correcting the underlying feature over-reliance, thereby enabling more rigorous and realistic safety assessments for the next generation of multimodal AI.