💻 computer science

FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Setting

This paper introduces FineState-Bench, a comprehensive benchmark with 2,209 instances and a novel four-stage diagnostic pipeline that reveals significant limitations in current LVLMs' ability to achieve precise, state-conditioned GUI interactions while demonstrating that improved visual grounding can substantially enhance performance.

Original authors: Fengxian Ji, Jingpu Yang, Zirui Song, Yuanxi Wang, Zhexuan Cui, Yuke Li, Qian Jiang, Xiuying Chen

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Fengxian Ji, Jingpu Yang, Zirui Song, Yuanxi Wang, Zhexuan Cui, Yuke Li, Qian Jiang, Xiuying Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to use a computer. In the past, we mostly tested if the robot could find a button and click it. If the robot found the right button, we said, "Great job!"

But this new paper, FineState-Bench, argues that finding the button isn't enough. It's like telling a robot, "Turn the volume knob to exactly 75%." If the robot finds the volume knob but clicks the wrong spot on it, the volume might end up at 60% or 90%. To the robot, it "clicked the knob," but to you, the task failed because the state (the volume level) wasn't exactly right.

Here is a simple breakdown of what the researchers did and found:

1. The Problem: "Close Enough" Isn't Good Enough

Current tests for AI agents (robots that use screens) are like a game of "Hot or Cold." They check if the AI found the right area. But in real life, we need precision.

The Old Way: Did the robot click the "Volume" slider? Yes? Pass.
The New Way (FineState-Bench): Did the robot click the exact spot on the slider that makes the volume 75%? If it clicked 74% or 76%, it fails.

The authors say current AI models are actually quite bad at this. Even the smartest models only get the "exact state" right about 23% of the time on average. They are good at finding the general area but terrible at hitting the tiny, precise target needed to change the setting exactly as requested.

2. The Solution: A New "Gym" for Robots

To fix this, the team built FineState-Bench, a massive testing ground with over 2,200 different tasks.

The Playground: It covers computers, phones, and websites.
The Tasks: Instead of just "click here," the tasks are specific: "Set the date to December 25th," "Drag this slider to 77.7%," or "Pick the exact shade of red."
The Map: For every task, they created a super-precise map. They didn't just draw a box around the whole slider; they drew a tiny box around the exact part of the slider you need to touch to get the right result.

3. The Diagnostic Tool: The "Visual Detective"

The researchers realized that when robots fail, we don't know why. Did they not understand the instruction? Did they find the wrong button? Or did they find the right button but miss the tiny target?

To solve this, they created a Visual Diagnostic Assistant (VDA). Think of this as a helpful human coach standing next to the robot.

Without the Coach: The robot looks at the screen and guesses. (Success rate: Low).
With the Coach: The coach whispers, "Hey, the red button you need is actually in this tiny square right here," and points to it.
The Result: When the robot gets this extra hint, its success rate jumps up significantly (by about 15%).

What this tells us: The robots aren't necessarily "dumb" or unable to understand the goal. Their main problem is visual blindness. They can't see the tiny, precise spot they need to click. If we give them better visual hints, they get much better at the job.

4. The Main Takeaway

The paper concludes that while AI agents are getting better at navigating screens, they are still struggling with fine-grained precision.

They are like a person trying to thread a needle in the dark: they can find the needle (the button), but they can't get the thread through the eye (the exact state).
Current tests are too easy because they accept "close enough."
To build truly reliable robots that can do complex tasks (like adjusting a professional video editor's settings or filling out a precise medical form), we need to test them on exactness, not just general location.

In short: The robots are getting better at finding the door, but they still need help to turn the key in the lock perfectly.

1. Problem Definition

Despite rapid advancements in Large Vision-Language Models (LVLMs) for GUI agents, fine-grained, state-conditioned interaction remains a significant bottleneck. Current agents often struggle to set a UI control to an exact target state (e.g., adjusting a slider to precisely 74%, selecting a specific date, or choosing a specific color) based on a single predicted interaction point.

Limitations of Existing Benchmarks:

Coarse Evaluation: Most existing benchmarks (e.g., ScreenSpot, AITW) focus on end-to-end task completion or simple "click-level" grounding. They often lack precise definitions of the target state.
Ambiguous Verification: They rarely distinguish between successfully locating a control and successfully hitting the specific "interactable core" required to change the state.
Lack of Diagnostics: Aggregate success rates obscure where and why agents fail (e.g., did the agent misidentify the control, or did it click the wrong part of the correct control?).

2. Methodology

A. FineState-Bench Dataset

The authors introduce FineState-Bench, a cross-platform benchmark comprising 2,209 instances across Desktop, Web, and Mobile platforms.

Scope: Covers 4 interaction families and 23 specific UI component types (e.g., Sliders, Date Pickers, Tree Views, Color Pickers).
Single-Step Protocol: Unlike trajectory-based benchmarks, FineState-Bench isolates the problem to a single-step, point-based interaction. The agent must predict two points:
1. $p_0$ : Location of the target control in the current UI.
2. $p_1$ : The specific operation point required to reach the exact goal state.
Dual-Region Annotation: Each instance provides four geometric annotations:
- Locate Box ( $B_{loc}$ ): The visible extent of the control.
- Interact Box ( $B_{int}$ ): The specific "interactable core" region that triggers the state change.
- These are annotated for both the current and target configurations to account for layout shifts.
Exact Goal-State Labels: Every instance includes a precise, quantifiable target state ( $s_{goal}$ ), such as a specific numeric value or binary toggle status.

B. FineState-Metrics

To move beyond binary success/failure, the authors propose a four-stage diagnostic pipeline:

SR@Loc (Localization Success Rate): Measures if $p_0$ falls within the control's visible extent ( $B^0_{loc}$ ).
SR@Int (Interaction Success Rate): Measures if $p_1$ falls within the state-changing core ( $B^0_{int}$ ).
ES-SR@Loc (Exact State Success at Locate): Measures exact state attainment conditioned on correct localization ( $p_0 \in B^1_{loc}$ ).
ES-SR@Int (Exact State Success at Interact): Measures exact state attainment conditioned on correct interaction ( $p_1 \in B^1_{int}$ ).

This decomposition allows researchers to attribute failures specifically to perception (finding the control), precision (hitting the core), or state-setting logic (failing to achieve the goal even after hitting the core).

C. Visual Diagnostic Assistant (VDA)

To quantify the impact of visual grounding errors, the authors introduce a plug-and-play VDA.

Function: VDA generates a structured Description of the target element and a Localization Hint (a bounding box) based on the instruction and screenshot.
Controlled Comparison: Agents are evaluated with and without these hints. The performance gap ( $\Delta_{vis}$ ) isolates the portion of errors caused specifically by insufficient visual grounding versus other factors.

3. Key Contributions

FineState-Bench: The first benchmark explicitly designed for exact goal-state verification in GUI agents, featuring dual-region geometric supervision and precise state labels across 23 component types.
FineState-Metrics: A stage-wise diagnostic framework that disentangles localization accuracy, point-level precision, and exact state attainment, enabling granular failure attribution.
Visual Diagnostic Assistant (VDA): A tool for controlled ablation studies that quantifies how much performance can be recovered by improving visual grounding, revealing that grounding errors are a primary bottleneck.
Comprehensive Benchmarking: Evaluation of 8 representative agents (including GPT-4o, Claude-3.5, Gemini-2.5-Flash, and open-source models like OS-Atlas and CogAgent) revealing significant performance gaps.

4. Experimental Results

Overall Performance

Low Success Rates: Even the strongest models struggle with exact state setting. The best-performing model, UGround-7B, achieved only 32.8% on Web and 22.8% on average across platforms for the primary metric (ES-SR@Int).
Platform Variance: Performance varies wildly; for example, Gemini-2.5-Flash scored 17.6% on Mobile but dropped to 0.7% on Desktop.

Diagnostic Insights

The "Interactable-Core" Bottleneck: The largest performance drop occurs between SR@Loc (coarse localization) and SR@Int (fine-grained interaction). Agents often correctly identify the general control but fail to click the precise region needed to change the state.
Component Sensitivity: Continuous controls (Sliders, Seek Bars, Knobs) show the largest gaps between localization and interaction success. Discrete controls (Switches, Radio Groups) perform better but still struggle with exact state verification.
VDA Impact: Providing VDA localization hints significantly improved performance. Gemini-2.5-Flash gained +14.9% in ES-SR@Int when provided with VDA hints. This suggests that a substantial portion of failures are due to visual grounding inaccuracies rather than reasoning or planning deficits.

5. Significance and Future Directions

Redefining Evaluation: The paper argues that "task completion" is insufficient for GUI agents. Reliable automation requires exact state verification, which current benchmarks fail to capture.
Identifying the Bottleneck: The study conclusively shows that the primary barrier to reliable fine-grained GUI interaction is visual grounding precision, not high-level reasoning.
Roadmap for Improvement: By providing a diagnostic framework, the authors enable the community to target specific failure modes (e.g., improving point prediction for sliders) rather than relying on aggregate metrics.
Limitations: The current benchmark focuses on static screenshots and single-step interactions. Extending these diagnostic capabilities to long-horizon, multi-step, and real-time interactive environments remains an open challenge.

In summary, FineState-Bench provides the necessary rigor to evaluate and improve the next generation of GUI agents, shifting the focus from "can the agent click something?" to "can the agent set the interface to the exact desired state?"