🤖 AI

Forced Deferral: Manipulating Routing Decisions in Multimodal LLM Cascades

This paper introduces the Forced Deferral Attack (FDA), an adversarial method that manipulates multimodal LLM cascades by lowering a weak model's confidence through universal border triggers, thereby forcing computationally expensive routing to a strong model without directly compromising answer correctness.

Original authors: Zhongye Liu, Yaopei Zeng, Yurui Chang, Lu Lin

Published 2026-06-16

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhongye Liu, Yaopei Zeng, Yurui Chang, Lu Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a high-end restaurant with a two-tier kitchen system.

The Setup: The "Fast Food" vs. The "Master Chef"
To save money, the restaurant uses a clever system. When a customer orders a dish, a junior cook (the Weak Model) tries to make it first.

If the junior cook is confident they can make it perfectly, they serve it immediately. This is cheap and fast.
If the junior cook is unsure or thinks the order is too tricky, they pass the ticket to the Master Chef (the Strong Model). The Master Chef is expensive and slow, but they always get it right.

The rule is simple: Only the difficult orders get the Master Chef. This keeps the restaurant running efficiently.

The Vulnerability: The "Confidence Hack"
The researchers in this paper discovered a sneaky way to break this system. They realized that the whole process depends on the junior cook's confidence. If the junior cook thinks, "I'm not sure about this," the ticket goes to the Master Chef.

The researchers found that an attacker doesn't need to trick the Master Chef or ruin the food. They just need to trick the junior cook into feeling insecure.

The Attack: The "Confidence-Flattening Frame"
The researchers created a special, invisible "frame" (a border) that can be placed around any picture a customer submits.

The Trick: When the junior cook looks at a picture with this special frame, their brain gets confused. They start to feel less certain about their answer, even if the picture is perfectly clear.
The Result: Because the junior cook suddenly feels "unsure," the system automatically passes the order to the expensive Master Chef.
The Outcome: The attacker gets a high-quality answer from the Master Chef, but the restaurant owner ends up paying the high cost for every single order, even the easy ones.

How They Did It (The "Magic Frame")
Instead of scribbling all over the picture (which might ruin the image for the Master Chef), they optimized a universal border.

They taught a computer to find a specific pattern for the edge of the image that makes the junior cook's brain "flat" and indecisive.
They didn't try to make the junior cook give the wrong answer; they just made the junior cook hesitate.
Because the center of the picture remains untouched, the Master Chef still sees the clear image and gives a perfect answer.

Why This Matters
The paper shows that this "confidence hack" works on many different types of AI models and different kinds of questions.

It's universal: The same border works on almost any image.
It's invisible: The picture still looks normal to humans.
It's costly: It forces the expensive AI to do work that the cheap AI was supposed to handle, draining the provider's resources.

In Short:
The researchers found a way to put a "confidence-sapping" sticker on the edge of an image. This sticker doesn't change the picture, but it makes the cheap AI feel too nervous to do its job, forcing the expensive AI to step in and do the work for free. It's a way to manipulate the system's budget by hacking its self-doubt.

Technical Summary: Forced Deferral in Multimodal LLM Cascades

Problem Statement
Multimodal Large Language Models (MLLMs) offer powerful visual reasoning capabilities but incur substantial inference costs when deployed for every query. To mitigate this, recent systems employ MLLM cascades, where a cheaper, weaker model ( $M_w$ ) handles queries first, deferring only uncertain cases to a stronger, more expensive model ( $M_s$ ). The deferral decision relies on a confidence signal derived from the weak model's output (e.g., token probabilities, perplexity, or consistency).

The paper identifies a critical vulnerability in this architecture: the confidence signal acts as a compute-allocation mechanism that can be manipulated. Unlike traditional adversarial attacks that aim to degrade answer correctness or bypass safety alignment, this threat targets the efficiency of the system. An adversary can manipulate the weak model's confidence to force unnecessary deferrals to the strong model. This allows the attacker to obtain high-quality responses while shifting the computational cost to the service provider, undermining the economic benefits of cascaded inference.

Methodology: Forced Deferral Attack (FDA)
The authors propose the Forced Deferral Attack (FDA), an adversarial image attack designed to lower the weak model's confidence without altering the semantic content of the query. The attack addresses three primary challenges:

Unknown Metrics: The attacker does not know the specific uncertainty metric (e.g., Answer Probability, Inverse Perplexity, Jaccard Similarity) or the deferral threshold used by the deployed system.
Semantic Preservation: The perturbation must not degrade the visual content, as the strong model must still be able to answer the query correctly.
Universality: The attack must be a single, reusable trigger optimized once, rather than a per-query optimization.

Core Components:

Metric-Agnostic Objective: Instead of optimizing against a specific confidence score, FDA targets the underlying token distribution. The attack constructs a temperature-flattened target distribution ( $q^T_t$ ) from the weak model's clean logits ( $z_t$ ) using a temperature $T > 1$ . This flattens the distribution (reducing concentration) while preserving the relative ranking of tokens from the clean response.
Teacher-Forcing Optimization: The attack minimizes the Kullback-Leibler (KL) divergence between the weak model's distribution on the perturbed input and the flattened target distribution. This forces the model to generate less confident predictions along the clean output trajectory.
Universal Border Trigger: To preserve semantics, the perturbation is constrained to a universal border region surrounding the image. The central visual content remains intact, ensuring the strong model receives a valid query. The trigger is learned as a fixed pattern applied to the image border.

Key Contributions

Threat Formulation: The paper identifies and formalizes the "cascade deferral attack" as a new efficiency-oriented threat. It shifts the adversarial focus from prediction correctness to the manipulation of compute allocation.
Novel Attack Mechanism: The authors propose an answer-agnostic universal border trigger optimized via a temperature-flattened teacher-forcing objective. This enables metric-agnostic deferral manipulation while maintaining the strong model's performance on triggered inputs.
Empirical Validation: The study demonstrates that FDA is effective, transferable, and robust across multiple datasets (MMBench, ScienceQA, MMMU), model families (Gemma4, Qwen3-VL), and deferral metrics.

Experimental Results
The evaluation compares FDA against baselines including generic image perturbations (defocus blur) and prompt injection.

Reroute Rate: FDA significantly increases the rate at which queries are routed to the strong model. Across datasets and model pairs, FDA consistently outperforms baselines, pushing the actual reroute rate close to 1.0 even at low calibrated deferral rates.
Confidence Reduction: The attack drastically lowers the weak model's confidence scores (e.g., reducing Answer Probability from ~0.94 to ~0.03 in some cases) without changing the input semantics.
System Accuracy: Crucially, the increased routing does not degrade the final system accuracy. Because the strong model receives the original semantic content (only the border is perturbed), it continues to provide high-quality answers.
Transferability: The attack transfers effectively across different model families and uncertainty metrics, confirming that the attack targets a fundamental property of the confidence mechanism rather than a specific implementation detail.

Significance and Claims
The paper claims that MLLM cascades, while efficient, expose a new attack surface where the deferral decision itself becomes vulnerable. The significance lies in demonstrating that an adversary can manipulate the system's resource allocation to force expensive inference without needing to compromise the model's safety or answer correctness. This highlights a gap in existing research, which has focused on cost-quality tradeoffs under benign conditions or robustness against output-level errors, but has largely overlooked the robustness of confidence-based routing mechanisms. The work suggests that current deferral strategies are insufficient to prevent adversaries from exploiting the compute-allocation signal.

Technical Summary: Forced Deferral in Multimodal LLM Cascades

More like this