💬 NLP

GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding

The paper introduces GUI-Primitives, a benchmark revealing that vision-language models primarily fail at localizing candidate interface elements rather than understanding spatial relations, as evidenced by their high accuracy when restricted to correct regions but poor performance on unconstrained coordinate prediction.

Original authors: Md Abrar Jahin, Md Rizwan Parvez

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Abrar Jahin, Md Rizwan Parvez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a computer that can look at a screenshot of a software program, read a sentence like "click the button to the right of the save icon," and then move a mouse cursor to that exact spot. This is the promise of a new generation of artificial intelligence agents designed to operate our computers for us. They are being built to perform tasks ranging from filling out forms to navigating complex menus, acting as digital assistants that can see and interact with the graphical interfaces we use every day. For these agents to work, they must possess a fundamental skill called "grounding": the ability to connect a specific word in a sentence to a specific shape or button on a screen. If the agent cannot reliably find the "button to the right," it cannot perform the task, no matter how smart it is at planning or typing.

Researchers have long known that these computer agents sometimes make mistakes, but they have struggled to pinpoint exactly why. Is the computer failing because it cannot read the text on the screen? Is it confused about what "right" means? Or is it simply unable to locate the buttons in the first place? Existing tests for these systems often mix all these skills together, making it impossible to tell which part of the brain is broken when the agent clicks the wrong thing. To solve this puzzle, a team of scientists created a new, highly controlled test designed to isolate a single, basic ability: understanding spatial relationships in a computer interface.

The researchers built a dataset called GUI-PRIMITIVES, which consists of nearly one thousand pairs of questions. Each pair uses the exact same screenshot and the exact same starting point, or "anchor," but changes only one word in the instruction. In one version, the computer is asked to click the element to the left of a specific icon; in the paired version, it is asked to click the element to the right of that same icon. Because the image and the anchor are identical, the only difference is the direction. If the computer is truly understanding the language, it should click the left button for the first question and the right button for the second. If it clicks the same button for both, or clicks a completely different area, it reveals a failure in understanding the spatial relationship. The team tested nineteen different artificial intelligence models on this challenge, ranging from powerful, expensive commercial systems to smaller, open-source versions.

The results were stark. Even the most advanced models performed poorly, with the best system getting the answer correct in only about thirty-two percent of cases. This is a massive gap compared to human performance, where people answered correctly nearly ninety-seven percent of the time. The researchers dug deeper to understand where the failures were happening. They discovered that the problem was not primarily that the models were confused by the words "left" or "right." Instead, the models were failing to find the correct area on the screen in the first place. In sixty to ninety-two percent of the errors, the computer's predicted click landed outside both the correct target and the wrong distractor, often in a completely empty part of the image.

When the researchers filtered out these "lost" predictions and looked only at the times the computer actually pointed to one of the two candidate buttons, the picture changed. For horizontal and vertical directions, the models were quite good at choosing the right one once they had narrowed it down to the correct area. However, for more complex relationships, such as determining if one item is inside a panel or hidden behind a pop-up window, the models performed no better than random guessing, even when they were looking at the right candidates. This suggests that while the models can sometimes understand the words, they struggle significantly with the visual task of locating where those words apply on a complex screen.

The study also tested whether these spatial skills mattered for real-world performance. They found a strong link: models that were better at these basic spatial tests were also better at a separate, more complex benchmark for clicking on real desktop applications. This confirms that the ability to understand simple spatial instructions is a foundational building block for more advanced computer use. The researchers also tried several tricks to help the models, such as asking them to think step-by-step or adding visual markers to the screen to highlight the options. One method, which involved placing numbered tags directly on the candidate buttons, dramatically improved the scores, lifting some models from thirty percent to nearly ninety percent. However, this was a diagnostic tool rather than a practical solution, as it required the computer to already know where the buttons were to place the tags in the first place.

Ultimately, the study reveals that the current generation of computer-using agents is missing a crucial piece of the puzzle. They are not failing because they cannot reason about language or because they cannot see the screen; they are failing because they cannot reliably map a simple spatial instruction to a specific location on a crowded, complex interface. Until this fundamental skill of connecting words to visual locations is fixed, these digital assistants will continue to struggle with the most basic tasks of navigating our digital world.

Technical Summary: GUI-PRIMITIVES

Problem Statement

Computer-use agents rely on vision-language models (VLMs) to "ground" natural language instructions in screenshots, locating interface elements to execute clicks or keystrokes. While existing benchmarks (e.g., OSWorld, ScreenSpot) evaluate end-to-end agent performance or coarse pixel-grounding, they fail to isolate whether models possess the elementary spatial reasoning skills required for this task. When an agent clicks the wrong button, it is unclear whether the failure stems from a lack of spatial understanding (e.g., confusing "left" with "right") or from an inability to localize the candidate elements themselves. Current benchmarks conflate these skills, preventing a precise diagnosis of model limitations.

Methodology

The authors introduce GUI-PRIMITIVES, a controlled diagnostic benchmark designed to isolate seven elementary spatial relations in graphical user interfaces (GUIs).

1. Benchmark Construction

Scale and Scope: The dataset contains 994 contrastive instruction pairs covering seven spatial primitives: horizontal position (left/right), vertical position (above/below), containment (inside/outside), alignment (same row/column), proximity (nearest/farthest), list ordinal (first/second), and occlusion (visible/hidden).
Minimal-Pair Design: Each item is paired with a "twin" that shares the exact same screenshot and anchor element but flips the spatial relation (e.g., "Click the element left of X" vs. "Click the element right of X"). This construction ensures that the correct target moves between two designated candidates. This design controls for screenshot-specific salience and fixed answer preferences; a model that relies on heuristics (e.g., always clicking the center) will succeed on one twin and fail the other, resulting in a pair-level score of zero.
Data Sources: The benchmark combines 290 items from real desktop screenshots (UI-Vision) and 704 procedurally rendered synthetic images. Synthetic data is used to cleanly generate containment and occlusion scenarios, which lack scalable metadata in real-world datasets.
Human Verification: A stratified subset of 196 items was validated by five annotators, achieving high agreement (Fleiss $\kappa = 0.94$ for well-formedness; $\kappa = 0.79$ for target selection). This resulted in a "human-clean" core of 185 items.

2. Evaluation Protocol

Models: 19 vision-language models were evaluated, including proprietary APIs (Claude Opus 4.7, GPT-5, etc.) and open-weight models (Qwen, Gemma, Llama, etc.).
Metrics: The primary metric is strict point-in-box accuracy (prediction must fall within the ground-truth bounding box). The authors also employ a candidate-level classification to distinguish between "relation errors" (selecting the wrong candidate) and "localization errors" (predicting a coordinate outside both candidate regions).
Interventions: The study tests three training-free interventions: Set-of-Mark (SoM) prompting, primitive-aware Chain-of-Thought (CoT), and activation steering.

Key Results

1. Performance Gap and Localization Failure

Low Overall Accuracy: Even the strongest model, Claude Opus 4.7, achieves only 32.4% strict accuracy on the human-clean core, leaving a 65-point gap compared to human annotators (96.9%).
Candidate Localization is the Primary Bottleneck: Analysis reveals that 60–92% of model predictions fall outside both the target and distractor candidate regions.
- For primitives like horizontal/vertical position and proximity, models that do fall within a candidate region select the correct target with high accuracy (0.82–0.90).
- For containment and occlusion, selection accuracy within a candidate region is indistinguishable from chance (0.50), indicating a genuine failure to understand the relation words for these specific primitives.
Real vs. Synthetic: On real desktop screenshots, strict accuracy drops to near zero for all models. This is primarily attributed to candidate localization failures (96.2% of predictions fall outside candidate regions) rather than relation-word errors.

2. Primitive-Specific Performance

List Ordinal: This is the only primitive reliably solved by top models (0.80–0.83 accuracy), likely because it reduces to a counting task once the list is identified.
Spatial Relations: All models perform below the 0.50 two-candidate reference level on containment, occlusion, alignment, and proximity.
Pair Consistency: Models exhibit a "relational failure signature" where pair consistency is significantly lower than the square of individual accuracy ( $y < x^2$ ). This indicates that models are negatively correlated on twin pairs; being correct on "left of" makes them more likely to be wrong on "right of," suggesting the relation word itself is the source of confusion.

3. Correlation with Downstream Tasks

GUI-PRIMITIVES accuracy correlates strongly with ScreenSpot-Pro grounding accuracy across ten models (Spearman $\rho = +0.74$ , $p = 0.015$ ). This suggests that elementary spatial competence is a predictor of broader GUI grounding performance.

4. Intervention Analysis

Set-of-Mark (SoM): Providing explicit marks on the candidate regions (an "oracle" diagnostic) dramatically improves performance, raising accuracy by 35–57 percentage points (e.g., GPT-5 from 30% to 87%). This confirms that the primary failure mode is locating the candidates, not interpreting the relation. However, this is a diagnostic upper bound, not a deployable solution, as it supplies the candidates the model is supposed to find.
CoT and Activation Steering: Primitive-aware Chain-of-Thought and activation steering yielded no significant gains, suggesting the failure lies closer to perception (visual encoding and cross-modal binding) than to language-side reasoning or residual stream representation.

Significance and Claims

The paper claims that GUI grounding failures in current VLMs are not primarily due to a lack of reasoning capability regarding spatial relations, but rather a failure in candidate localization. Models often fail to identify the relevant interface elements before attempting to reason about their spatial relationship.

Diagnostic Utility: GUI-PRIMITIVES provides a necessary control to separate localization failures from relational reasoning failures, a distinction obscured by existing benchmarks.
Limitations of Scale: Increasing model scale does not monotonically close the gap; even the largest proprietary models struggle with basic spatial primitives.
Future Directions: The authors posit that the bottleneck is likely perceptual (visual encoder or cross-modal binding) rather than linguistic. They suggest that future work should focus on improving the visual grounding of candidate elements before attempting to refine spatial reasoning.

The paper explicitly avoids claiming that these findings solve the problem of GUI agents; rather, it offers a rigorous diagnostic tool to identify where and why current systems fail, highlighting that the "candidate search" problem is the dominant hurdle in GUI grounding.