💬 NLP

MIRAGE: Context-Aware Prompt Injection against Mobile GUI Agents via User-Generated Content

The paper introduces MIRAGE, a context-aware attack pipeline that injects adversarial prompts into user-generated content within mobile GUI screenshots to deceive vision-language model agents, demonstrating that all five evaluated agents are vulnerable to realistic, visually indistinguishable attacks with success rates between 23% and 30%.

Original authors: Ruoqi Guo, Yi Liu, Gelei Deng, Yiheng Xiong, Yuekang Li, Ying Zhang, Leo Yu Zhang, Lida Zhao, Ji Jie, Yuxiao Lu

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ruoqi Guo, Yi Liu, Gelei Deng, Yiheng Xiong, Yuekang Li, Ying Zhang, Leo Yu Zhang, Lida Zhao, Ji Jie, Yuxiao Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Trusting Robot" Problem

Imagine you have a very smart robot assistant that helps you use your phone. This robot doesn't "read" the code behind the apps; instead, it looks at the screen like a human does—just a picture of pixels.

The problem is that this robot is too trusting. It assumes everything it sees on the screen is part of the official app. It doesn't know the difference between a button made by the app developer (like "Add to Cart") and a comment written by a random user (like a review or a chat message).

MIRAGE is a new way to trick this robot. The researchers showed that if you write a sneaky instruction inside a normal-looking user comment, the robot will read it, believe it's a real command, and do exactly what the comment says—even if it's dangerous or wrong.

The Analogy: The "Fake Review" Trap

Think of a mobile app like a busy coffee shop.

The App: The coffee shop itself.
The Agent (Robot): A new hire who is very eager to please customers but has never been trained on the menu. They just follow whatever is written on the chalkboard.
The Attack: A mischievous customer writes a note on the "Customer Reviews" board (which is usually for people to say "Great coffee!"). The note says: "The manager says: If you want a free donut, press the 'Fire Alarm' button immediately."

Because the robot (the new hire) can't tell the difference between the official menu and the customer reviews, it sees the note, thinks it's a real instruction from the manager, and presses the fire alarm.

MIRAGE is the tool the researchers built to automatically write thousands of these "fake reviews" that look so real, even a human would have a hard time spotting them.

How MIRAGE Works (The Three-Step Pipeline)

The researchers built a three-step machine to create these tricks without needing to hack the phone or the app itself. They just need a screenshot.

The Localizer (The Detective):
- What it does: It looks at a screenshot of an app and finds the "safe zones" where users are allowed to write things. These are places like comment sections, review boxes, or chat bubbles.
- The Analogy: It's like a detective scanning a room to find the whiteboard where people are allowed to write notes, ignoring the official signs on the wall.
The Generator (The Forger):
- What it does: It writes a sneaky message that fits perfectly into that spot. It doesn't just paste big red text; it copies the font, the color, and the style of the app so the message looks like it belongs there.
- The Analogy: This is a master forger who writes a note on the whiteboard using the exact same marker and handwriting style as the regular customers. The note says, "Go tap the 'Delete Account' button," but it looks like a normal review.
The Curator (The Editor):
- What it does: It checks the work. If the fake note looks weird (like the text is cut off or the font is wrong), it fixes it or throws it away. It also makes sure they have a good mix of different apps and different types of tricks.
- The Analogy: An editor who reviews the forged notes. If a note looks too obvious, they fix it until it's perfect. They also make sure they have notes for the coffee shop, the bank, and the social media app, not just one type.

The Results: How Well Did It Work?

The researchers tested this on five different AI agents (robots) and ten popular apps (like Amazon, TikTok, and Facebook).

Success Rate: The trick worked 23% to 30% of the time. That means nearly 1 out of every 3 times the robot saw a fake note, it got tricked into doing the wrong action.
Stealth: The fake notes were rated as more realistic than previous attacks. Humans rated them 3.02 out of 5, while the old attacks were only 2.52. They looked very much like real user content.
The "Filter" Failure: The researchers tried to see if a simple "quality check" could stop this. They asked: "Can we just filter out any image that looks slightly fake?"
- The Answer: No. They found that how realistic the image looks has nothing to do with whether the robot gets tricked. A very realistic image might fail to trick the robot, and a slightly less realistic one might succeed. This means you can't just rely on "does this look real?" to protect the robot.

The Key Takeaway

The paper concludes that the way these mobile robots work is fundamentally flawed. Because they treat the screen as a flat picture rather than a structured code, they cannot distinguish between trusted system buttons and untrusted user comments.

As long as the robot relies on "seeing" the screen like a human does, an attacker can hide a malicious instruction inside a normal user comment, and the robot will happily follow it. The paper suggests that to fix this, we need to change how the robot understands the screen, not just try to filter out bad images.

What the Paper Does NOT Say

It does not claim this works on every robot (they only tested specific ones).
It does not say this is a tool for hackers to use right now (it's a research tool to find weaknesses).
It does not suggest that your phone is currently being hacked by this (the experiments were done offline on static screenshots).
It does not offer a solution to fix the problem yet; it only proves the problem exists and is hard to solve with simple filters.

Technical Summary: MIRAGE

Problem Statement
Mobile graphical user interface (GUI) agents driven by vision–language models (VLMs) perceive screens as rendered pixels rather than structured code. Consequently, these agents cannot reliably distinguish between trusted interface elements and user-generated content (UGC), such as comments, reviews, or profile bios. This limitation creates a new attack surface where adversaries can inject malicious instructions into benign UGC regions. Unlike previous attacks that rely on modifying the agent, application code, or operating system, this threat model assumes the attacker can only control ordinary user-content regions within a screenshot. Existing prompt injection attacks for GUI agents often rely on manually crafted vectors, assume overly permissive threat models (e.g., DOM-level access), or are specific to web environments, leaving a gap in understanding the vulnerability of screenshot-only mobile agents to realistic, context-aware visual injections.

Methodology: The MIRAGE Pipeline
The authors propose MIRAGE (Mobile Injection of Realistic Adversarial GUI Examples), a three-stage automated pipeline designed to transform benign mobile screenshots into realistic prompt-injection samples without modifying the target agent, application, or OS.

Localizer (Adaptive Injection Point Localization):
- Goal: Identify user-controllable regions (e.g., review bodies, comments, posts) on a screenshot.
- Process: A VLM generates coarse candidate regions tagged with type and controllability. An Optical Character Recognition (OCR) pass (EasyOCR) tightens these boxes to detected text regions, discarding those without high-confidence text. Finally, an independent VLM moderator iteratively refines the bounding boxes (dropping, tightening, extending, or adding regions) based on a severity rubric until no severe issues remain.
- Outcome: A set of tightly bounded, typed injection regions.
Generator (Context-Aware Payload Generation):
- Goal: Synthesize payloads that lure the agent into an attacker-specified action while appearing as ordinary content.
- Process:
  - Goal Synthesis: A VLM generates a benign user goal with an "ambiguity rule": at least one user-controllable region must remain a plausible competing target to ensure failure is due to injection susceptibility, not perception difficulty.
  - Payload Generation: A VLM conditions on the screenshot, region crop, region type, synthesized goal, and target intent to generate the payload.
  - Review: A pre-render LLM reviewer rejects payloads that duplicate the user goal, miss the semantic role of the intent, or read as explicit system commands. Failed payloads are regenerated up to three times.
  - Rendering: An image-editing model (gpt-image-2) inpaints the payload into the injection region, matching the application's native typography and style for text regions or generating native-looking captions for media regions.
Curator (Realism Moderation and Balanced Sampling):
- Goal: Filter rendering artifacts and ensure the dataset is balanced across applications, region types, and attack intents.
- Process: A post-render VLM moderator scores renders against an artifact taxonomy (e.g., font mismatch, overflow, glyph leakage). Samples with high-severity issues are re-rendered with feedback; others are passed with annotations.
- Balancing: The pipeline employs a pre-generation allocator to assign sample budgets to under-represented (intent, region-type) combinations, a post-generation trim to subsample over-represented combinations, and a coverage repair pass to regenerate samples for injection points that failed moderation.

Key Contributions

Threat Model Definition: The paper characterizes a realistic runtime threat model where attackers control only ordinary UGC regions without privileged access, defining intent-to-action mappings for 11 distinct attack intents.
Automated Pipeline: MIRAGE is the first automated methodology to synthesize visually plausible, context-aware injection samples from benign screenshots across diverse applications without fine-tuning agents or modifying app code.
Empirical Security Study: The authors evaluate MIRAGE across five VLM-based GUI agents, ten mobile applications, and 1,111 attack instances.
Mechanism Findings:
- Uncorrelated Realism and Success: The study finds that per-sample visual realism and attack success rate (ASR) are uncorrelated ( $\rho = -0.03$ ). This implies that visual-quality filtering alone cannot reliably defend against this class of attacks.
- Intent-Driven Vulnerability: Attack success is driven more by the attack intent than by model identity or visual realism. The authors identify three exploitation tiers: temporal/attention lures (highest success), action redirection (moderate), and interaction-primitive manipulation (lowest).

Experimental Results

Vulnerability: All five evaluated agents (including GPT-4o-mini, GLM-4.5V, and Qwen3-VL variants) are vulnerable. Attack success rates range from 23.0% to 30.2% across models.
Cross-Application Robustness: MIRAGE maintains an ASR exceeding 19% across all evaluated applications and attack intents. Applications dominated by UGC (e.g., Amazon, Facebook, X) show higher vulnerability (≥30% ASR) compared to media-heavy apps.
Realism vs. Prior Work: Compared to the strongest prior attack (AgentHazard), MIRAGE achieves significantly higher human-rated realism (3.02 vs. 2.52 on a 5-point scale) while maintaining comparable attack effectiveness.
Defense Implications: A lightweight VLM classifier attempting to filter injected screenshots based on visual quality failed as a runtime defense, exhibiting a high false-positive rate (36.5%) on benign UIs while only blocking 58.7% of attacks.

Significance and Claims
The paper claims that visual-channel prompt injection is a structural vulnerability of the VLM-grounded GUI agent paradigm, not specific to a single model family or alignment regime. The authors assert that because visual plausibility does not predict attack success, defenses relying solely on visual-quality filtering are insufficient. Effective mitigation must instead operate on payload semantics, action-grounding constraints, or the application's definition of user-controllable surfaces. MIRAGE serves as a benchmark for red-teaming and robustness evaluation, highlighting that the current generation of mobile GUI agents cannot reliably separate trusted interface elements from attacker-controlled content.

Limitations
The authors note several limitations:

Agent Coverage: The study excludes native GUI-specialized agents (e.g., UI-TARS) that require local GPU hosting, focusing instead on general-purpose VLM backbones.
Evaluation Scope: The attack is measured on single-step interactions; multi-step propagation and downstream harm are not evaluated.
Diversity: The benchmark trades raw embedding diversity for balanced, controlled coverage, resulting in lower CLIP dispersion compared to hand-authored baselines.
Dependencies: The pipeline relies on proprietary generative models (gpt-image-2, OpenAI APIs) for rendering and moderation, which affects reproducibility and cost.