💻 computer science

POINTS-GUI-G: GUI-Grounding Journey

This paper introduces POINTS-GUI-G-8B, a state-of-the-art GUI grounding model trained from a base with minimal spatial awareness by leveraging refined data engineering, improved training strategies, and reinforcement learning with verifiable rewards to achieve superior performance across multiple benchmarks.

Original authors: Zhongyin Zhao, Yuan Liu, Yikun Liu, Haicheng Wang, Le Tian, Xiao Zhou, Yangxiu You, Zilin Yu, Yang Yu, Jie Zhou

Published 2026-02-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhongyin Zhao, Yuan Liu, Yikun Liu, Haicheng Wang, Le Tian, Xiao Zhou, Yangxiu You, Zilin Yu, Yang Yu, Jie Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot assistant that can read text and look at pictures. You want to teach it to use your computer or phone, just like you do. But there's a problem: the robot can read the instructions ("Click the red button"), but it doesn't know where the red button actually is on the screen. It's like giving someone a map of a city but not telling them which street they are currently standing on.

This paper introduces POINTS-GUI-G, a new version of that robot assistant that has finally learned to look at a screen and say, "Ah, I see the button! It's right there at these exact coordinates."

Here is how the researchers taught this robot to become a master of the screen, explained through simple analogies:

1. The Starting Point: A Blank Slate

Most other researchers took a robot that was already good at finding things (like a seasoned detective) and just gave it a few more clues. The authors of this paper decided to start with a robot that was bad at finding things. They wanted to build the skill from the ground up, like teaching a child to read rather than just correcting a teenager's spelling. They started with a base model called "POINTS-1.5" that had almost no ability to point at things on a screen.

2. The Three Pillars of Success

To turn this "blind" robot into a "sighted" one, they used three main strategies:

A. Cleaning and Organizing the Textbooks (Refined Data Engineering)

Imagine you are trying to teach a student using a pile of textbooks. Some books use inches, some use centimeters, some are written in English, and others in French. Some pages are torn, and some have scribbles all over them. It would be a nightmare to learn from.

The researchers did three things to fix this:

Standardization: They took all the messy, different datasets and forced them into one single format. Now, every "coordinate" (location) is measured the same way, like converting everything to a standard ruler.
Noise Reduction: They acted like strict editors. They used a special tool (called OmniParser-v2) to check the textbooks. If a textbook said "The button is here" but the picture clearly showed the button was somewhere else, they threw that page out. They only kept the perfect examples.
Making it Harder: Once the robot got good at finding big, obvious buttons, the researchers started feeding it "hard mode" puzzles. They created screens with tiny, crowded buttons and confusing layouts (like a messy desk with papers everywhere) to force the robot to get sharper and more precise.

B. Tuning the Eyes (Improved Training Strategies)

Usually, when training these AI models, the "eyes" (the part of the computer that processes images) are frozen in place. The researchers realized that for finding buttons on a screen, the eyes needed to be flexible.

Unfreezing the Vision: They let the "eyes" learn and adjust while the rest of the brain learned. This allowed the robot to get better at spotting tiny details.
Resolution Consistency: Imagine practicing for a basketball game by shooting hoops from 5 feet away, but then showing up to the real game where the hoop is 10 feet away. You would miss. The researchers noticed their robot practiced on small, low-resolution images but was tested on huge, high-definition screens. They fixed this by training the robot on larger, clearer images so its "vision" was ready for the real thing.

C. The Video Game Reward System (Reinforcement Learning)

This is the most unique part. Usually, Reinforcement Learning (RL) is used to teach robots how to think or solve logic puzzles. Here, they used it to teach the robot how to see.

Think of it like a video game:

The robot guesses where a button is.
The computer checks: "Did you hit the button?"
Yes? +100 points.
No? 0 points.

Because the answer is either right or wrong (there is no "maybe"), the robot gets very clear feedback. The researchers found that this "try, get a score, try again" loop made the robot much more accurate than just showing it examples and saying "this is right."

The Result

By combining these three methods, the POINTS-GUI-G model became a champion at finding things on screens.

It beat many other models that were much bigger and more expensive.
It scored incredibly high on tests that measure how well a robot can find buttons, text, and icons on phones, computers, and websites.

In short: The paper shows that if you clean your training data, let your model's "eyes" learn freely, and use a strict "right-or-wrong" reward system, you can build a small, efficient robot that is better at navigating your computer screen than many giant, expensive alternatives.

Technical Summary: POINTS-GUI-G: GUI-Grounding Journey

Problem Statement

The rapid advancement of Vision-Language Models (VLMs) has enabled the emergence of GUI agents capable of automating complex digital workflows, from online shopping to flight booking. A foundational prerequisite for these agents is GUI grounding: the ability to precisely locate interface elements (text, icons, buttons) within a screenshot to execute operations like clicking or typing.

While current research often relies on models with pre-existing, strong spatial awareness (e.g., Qwen3-VL) to focus on end-to-end task execution, this approach may obscure critical technical insights necessary for iterative development. Furthermore, existing approaches often overlook the specific challenges of grounding, such as data heterogeneity, resolution mismatches between training and inference, and the potential of reinforcement learning in perception-heavy tasks. This paper addresses these gaps by aiming to master the full technical pipeline of GUI grounding, starting from a base model with minimal native grounding capabilities (POINTS-1.5) rather than a pre-optimized foundation.

Methodology

The authors introduce POINTS-GUI-G-8B, a model developed through a "full-stack" approach comprising three core pillars:

1. Refined Data Engineering

The authors implemented a multi-stage pipeline to address the limitations of existing open-source GUI grounding datasets:

Standardization: Diverse datasets were unified by normalizing all coordinates to a $[0, 1]$ range with three-decimal precision and reformulating tasks into consistent UI element localization formats (bounding box prediction or center point localization).
Noise Reduction: An automated filtering pipeline using OmniParser-v2 was employed to prune inaccurate annotations. For point annotations, a coverage score was calculated against detected UI elements; only samples with a high overlap ( $S \ge \tau$ ) were retained to eliminate hallucinated coordinates.
Complexity Enhancement: To push performance ceilings, "simple" samples (sparse layouts, oversized regions) were filtered out using Layout Entropy metrics (1D projection and 2D grid entropy). Additionally, challenging data was synthesized via:
- GUI-CodeGen: Rendering professional software interfaces (e.g., VS Code) with high-density functional components.
- GUI-Overlay: Layering multiple application windows onto desktop backgrounds to introduce realistic distractors and occlusions.

2. Improved Training Strategies

Two critical adjustments were made to the training pipeline to enhance perceptual accuracy:

Unfrozen Vision Encoder: Unlike previous iterations where the vision encoder was frozen, the authors fully unfroze the encoder (based on Qwen2-VL-NaViT) during training. This allowed the model to adapt its feature extraction specifically to the GUI domain, yielding substantial performance gains.
Resolution Consistency: To mitigate performance degradation on high-resolution benchmarks (e.g., ScreenSpot-Pro) caused by training on lower-resolution data, the authors increased the maximum training resolution cap to 3072 × 3072 pixels and constrained inference resolution to under 2000 × 2000 pixels. This alignment resulted in a performance boost of over 10 points on high-resolution benchmarks.

3. Reinforcement Learning with Verifiable Rewards (RLVR)

The paper demonstrates the application of RL to GUI grounding, a task traditionally dominated by Supervised Fine-Tuning (SFT).

Verifiable Rewards: The output space (points or bounding boxes) allows for precise, objective reward functions. The reward is binary ($1$ or $0$) based on whether the predicted coordinate falls within the annotated bounding box.
Algorithm: The authors employed Group Relative Policy Optimization (GRPO).
Curriculum Learning: A curriculum-based strategy was used to progressively increase the difficulty of training samples, stabilizing the learning process.
No Reasoning Trace: Following GTA1, the model does not generate a reasoning trace before outputting coordinates, focusing purely on the perception-to-action mapping.

Key Results

POINTS-GUI-G-8B achieves state-of-the-art performance across five major benchmarks, often outperforming significantly larger models (e.g., 32B and 72B parameter models):

ScreenSpot-Pro: Scored 59.9, outperforming GTA1-7B by 9.8 points and GUI-Owl-7B by 5 points.
OSWorld-G: Achieved 66.0, ranking first among SOTA models and outperforming MAI-UI-8B by approximately 6 points.
ScreenSpot-v2: Achieved 95.7.
UI-Vision: Scored 49.9, often exceeding competitors by margins greater than 10 points.
MMBench-GUI-L2: Achieved 87.0, showing robust performance across mobile, desktop, and web environments.

The ablation studies (Figure 7) confirm that Data Engineering (DE), Unfrozen Vision Encoder (UVE), Image Resolution consistency (IR), and Reinforcement Learning (RL) each contribute incrementally to the final performance, with RL providing consistent improvements over the supervised baseline.

Significance and Claims

The paper claims that POINTS-GUI-G serves as a robust foundation for future GUI agent development. Its significance lies in:

Full-Stack Mastery: Demonstrating that starting from a base model with minimal grounding ability and building up through refined data and training strategies can yield SOTA results, offering a comprehensive alternative to relying solely on pre-optimized large models.
RL in Perception: Proving that Reinforcement Learning with Verifiable Rewards is highly effective for perception-intensive tasks like GUI grounding, where rewards are easily verifiable and accurate, challenging the notion that RL is solely for reasoning tasks.
Technical Nuance: Highlighting often-overlooked factors such as resolution consistency and the necessity of unfreezing vision encoders for specialized domains.

The authors have open-sourced the model and evaluation suite to support the research community and address inconsistent evaluation standards, aiming to catalyze progress in end-to-end GUI agent execution.