💻 computer science

MissClick: Exploiting Digit-Serialized Coordinates to Attack GUI Grounding Models

The paper introduces MissClick, a white-box adversarial attack that exploits the digit-serialized coordinate generation process of GUI grounding models by leveraging place-value sensitivity to achieve significantly higher untargeted and targeted success rates compared to existing methods.

Original authors: Yu Ran, Wentao Zhao, Xin Zhang, Yi Pan

Published 2026-08-05

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Yu Ran, Wentao Zhao, Xin Zhang, Yi Pan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where your computer screen isn't just a picture, but a map of commands waiting to be tapped. In this digital landscape, a new kind of "smart assistant" has emerged: the GUI agent. Think of these agents as incredibly talented, but slightly literal, tour guides. You tell them, "Click the red button to buy the shoes," and they look at the screen, figure out exactly where that button is, and tap it. To do this, they don't just point; they speak in a secret code of numbers. They generate a string of digits, like "814, 515," which the computer translates into a precise location on your screen. This process is called "visual grounding," and it's the magic that lets robots and AI navigate our apps, websites, and phones without human hands. But just like any map, if you can trick the guide into reading the numbers wrong, you can send them to the wrong destination entirely.

This is the story of a new discovery called MissClick. The researchers behind it realized that the way these AI agents write down coordinates is a bit like a game of "place value" in math class. When an AI writes the number "814," the '8' isn't just a '8'; it's an '8' that stands for 800 because it's in the hundreds spot. The '1' is just 10, and the '4' is just 4. The team found that if you can trick the AI into changing that '8' to a '9', the click doesn't just move a little bit—it jumps 100 pixels across the screen! By exploiting this mathy quirk, they created a "hacker" tool that can nudge the AI's numbers just enough to make it click on a "Checkout" button instead of the "Quantity" field, or wander off into the blank space of the screen. They tested this on two different AI models across desktop, web, and mobile screens, and found that by understanding the math behind the digits, they could break the system much more effectively than anyone had before.

The Secret Code of the Screen

To understand how MissClick works, we first need to look at how these AI agents "see" a screen. When you give an instruction like "adjust the item quantity with 5," the AI doesn't just guess a spot. It acts like a typewriter, spitting out a sequence of characters one by one: (, 8, 1, 4, ,, 5, 1, 5, ). It's generating the numbers "814" and "515" as a text string. Once it's done typing, the computer takes that string and turns it into a real coordinate, telling the mouse to jump to that exact pixel.

The researchers noticed something fascinating about this process. The AI treats every digit as a separate character, but the computer treats them as a mathematical value. If the AI gets the first digit of a coordinate wrong, the result is a massive jump. For example, if the AI is supposed to click at 814 but gets tricked into clicking at 914, the click moves 100 pixels to the right. If it gets the last digit wrong, changing 814 to 815, it only moves one pixel. This means that messing with the "hundreds" digit is like pushing a boulder, while messing with the "ones" digit is like nudging a pebble.

The Two Faces of MissClick

The team, led by Yu Ran and colleagues from the National University of Defense Technology, built an attack tool called MissClick. They realized that to hack this system, you can't just treat the numbers like random letters. You have to respect the math. They designed two different strategies, or "modes," depending on what the hacker wants to achieve.

Mode 1: The "Miss" (Untargeted Attack)
Imagine you want to make the AI trip and fall off a cliff. You don't care where it lands, as long as it's not where it's supposed to be. This is called an "untargeted" attack. MissClick-U (the "U" stands for Untargeted) works by calculating a "soft" version of the coordinate. Instead of waiting for the AI to pick a single digit, it looks at the probability of every possible digit. It asks, "What if the AI picks a 9 instead of an 8? What if it picks a 7?" It then calculates a weighted average, creating a "ghost coordinate" that floats between the numbers. The goal is to push this ghost coordinate as far away as possible from the correct spot. By maximizing this distance, the AI is forced to pick a digit that sends the real click way off course.

Mode 2: The "Hijack" (Targeted Attack)
Now, imagine you want to trick the AI into clicking a specific, dangerous button, like a "Delete Account" button, instead of a "Save" button. This is a "targeted" attack. Here, MissClick-T (the "T" stands for Targeted) plays a different game. It doesn't just want to move the click; it wants to land it in a specific zone. The researchers found that simply trying to make the "average" number match the target wasn't enough. Sometimes the AI might "think" the average is 5, but actually pick a 1 or a 9 because those were the most likely options. So, MissClick-T focuses on the "place value." It puts extra pressure on the high-value digits (the hundreds and tens places). It tells the AI, "You must get the '8' in the hundreds spot right, because that's what moves the click the most." By weighting the importance of each digit based on its mathematical power, it forces the AI to select the exact digits needed to land on the attacker's target.

The Results: A Big Jump in Success

The team tested MissClick on two popular AI models, OS-Atlas and UGround, across three different environments: desktop computers, web browsers, and mobile phones. They compared their new method against older hacking techniques that treated the coordinates like plain text, ignoring the math.

The results were striking.

For the "Miss" attacks: MissClick-U succeeded in making the AI click the wrong place 75.07% of the time on OS-Atlas and 72.93% on UGround. The old methods only managed about 58% and 42% respectively. That's a huge jump, improving success rates by over 30 percentage points in some cases.
For the "Hijack" attacks: MissClick-T was even more impressive. It successfully redirected the click to the attacker's chosen target 44.86% of the time on OS-Atlas and a whopping 62.67% on UGround. The old methods struggled, succeeding only about 13% to 15% of the time.

The researchers also found that the two goals needed different tools. Trying to use the "Miss" strategy for a "Hijack" didn't work well, and vice versa. This confirmed that understanding the specific goal of the attack is just as important as understanding the math behind the digits.

Why This Matters

This paper doesn't just show that AI can be tricked; it shows how to trick it better by understanding the hidden structure of its language. The authors suggest that because the AI generates coordinates as a sequence of digits, the "place value" of those digits creates a vulnerability. A small change in a high-value digit creates a massive shift in the final action.

However, the researchers are careful to note the limits of their work. They only tested on models that write coordinates as digit-by-digit text. They didn't test on models that use different methods to point at things. They also assumed they had full access to the AI's "brain" (a "white-box" scenario), which is easier than hacking a system you can't see inside. But the core idea—that the way we ask AI to speak numbers matters for security—seems to be a solid discovery.

In the end, MissClick is a reminder that in the digital world, a number isn't just a number. It's a coordinate, a command, and sometimes, a trap. By understanding the math behind the magic, the researchers showed that we can make these smart agents stumble, or steer them exactly where we want them to go.

Technical Summary: MissClick

Problem Statement
Recent GUI visual grounding models, such as OS-Atlas and UGround, generate screen coordinates as sequences of digit tokens (e.g., "8", "1", "4" for x=814) which are subsequently parsed into numerical values to execute clicks. The security implications of this specific generation pipeline have been largely overlooked. Existing adversarial attacks against GUI grounding typically treat coordinate outputs as ordinary text sequences, optimizing objectives based on token-level cross-entropy or embedding distances. However, this approach fails to account for the numerical and place-value structure of the coordinate generation process.

A critical observation driving this work is that a change in a high-order digit (e.g., the hundreds place) results in a significantly larger displacement of the executed click (e.g., 100 units) compared to a change in a low-order digit (e.g., 1 unit for the ones place), despite both being single-token changes. Furthermore, untargeted and targeted attacks have distinct success conditions: untargeted attacks require the click to fall outside the ground-truth region, while targeted attacks require the click to fall inside a specific attacker-specified region. Existing methods do not explicitly model the parsing of digit strings into numerical coordinates or the mapping of these coordinates to executable clicks, leading to suboptimal attack objectives.

Methodology: MissClick
The authors propose MissClick, a white-box adversarial attack framework designed to exploit the digit-serialized nature of coordinate generation. MissClick employs two distinct, goal-specific objectives tailored to the success conditions of untargeted and targeted attacks.

Coordinate-Digit Structure Modeling:
The method models the coordinate $c_k$ as a sum of digit values weighted by their decimal place values ( $a_{k,j} = 10^{m_k-j}$ ). The attack utilizes teacher forcing during optimization to obtain differentiable logit vectors for each digit position. From these, it extracts digit probability distributions $p_{k,j}(v)$ for digits 0–9.
MissClick-U (Untargeted Attack):
- Objective: Maximize the displacement between the executed click and the ground-truth location.
- Mechanism: Instead of optimizing token probabilities directly, MissClick-U constructs soft coordinates ( $\tilde{c}$ ) by computing the expected value of each digit position weighted by its place value: $\tilde{d}_{k,j} = \sum v \cdot p_{k,j}(v)$ .
- Loss Function: The attack maximizes the squared Euclidean distance between the soft click $r(\tilde{c})$ and the ground-truth click $u_{gt}$ . This approach is differentiable and explicitly accounts for the scale of displacement caused by digit changes at different decimal positions.
MissClick-T (Targeted Attack):
- Objective: Minimize the loss to ensure the executed click falls within a specific target region.
- Mechanism: The authors argue that minimizing the distance between soft coordinates and a target does not guarantee that the argmax selected digits will match the target digits (due to the non-linearity of the argmax operation). Therefore, MissClick-T directly optimizes the probability of the target digits.
- Loss Function: It employs a place-weighted target-digit loss. This is a cross-entropy loss against the target digit tokens, where the weight for each digit position is its decimal place value ( $a_{k,j}$ ). This ensures that optimization prioritizes high-order digits, which have the most significant impact on the final coordinate value.

Key Contributions

Security Analysis of Coordinate Pipelines: The paper identifies that treating coordinate outputs as ordinary text ignores the critical numerical structure (place-value) that determines the magnitude of click displacement.
Goal-Specific Attack Objectives: The proposal of MissClick-U and MissClick-T, which utilize soft-coordinate displacement and place-weighted digit optimization, respectively. These objectives are derived directly from the success conditions of untargeted and targeted attacks.
Empirical Validation: Comprehensive experiments across desktop, web, and mobile platforms on two representative models (OS-Atlas and UGround) demonstrate the efficacy of these objectives.

Experimental Results
The authors evaluated MissClick on the ScreenSpot-v2 dataset against three baselines: Random Noise, Token Cross-Entropy (CE), and Representation Attack.

Untargeted Performance: MissClick-U achieved Attack Success Rates (ASR) of 75.07% on OS-Atlas and 72.93% on UGround. This represents improvements of +16.62 and +30.72 percentage points over the best baseline (Representation Attack), respectively.
Targeted Performance: MissClick-T achieved ASRs of 44.86% on OS-Atlas and 62.67% on UGround. These results outperformed the Token CE baseline by +31.73 and +47.06 percentage points, respectively.
Objective Comparison: The study confirmed that soft-coordinate displacement is superior for untargeted attacks, while place-weighted target-digit optimization is superior for targeted attacks, validating the need for distinct objectives.
Efficiency: MissClick reached or exceeded baseline performance with significantly fewer optimization iterations (10–30 iterations vs. 100 for baselines).

Significance and Claims
The paper claims that the security of GUI grounding models is fundamentally linked to how coordinate strings are parsed into numerical values. By explicitly modeling the place-value structure of digit tokens, MissClick demonstrates that attacks can be significantly more effective than those treating coordinates as generic text. The work highlights that untargeted and targeted attacks have distinct "objective preferences," necessitating specialized designs rather than a one-size-fits-all approach.

Limitations
The authors acknowledge three primary limitations:

The evaluation is restricted to models that serialize coordinates as per-digit decimal tokens, excluding coordinate-free interfaces or different tokenization schemes.
The place-weighted objective treats non-target digits uniformly and does not encode the ordinal distance between non-target digits and the target.
The threat model assumes white-box access to the victim model's architecture and weights.

The Secret Code of the Screen

The Two Faces of MissClick

The Results: A Big Jump in Success

Why This Matters

Technical Summary: MissClick

More like this