Technical Summary: WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point

Problem Statement

Recent advancements in Multimodal Large Language Models (MLLMs) have improved the visual grounding capabilities of GUI agents. However, robust planning remains a significant challenge, particularly when the environment deviates from a canonical initial state. In real-world applications, users frequently invoke assistance mid-workflow, where software may be partially configured, steps may have been executed in a different order, or the interface may differ from its default setup.

Existing online GUI benchmarks (e.g., WebArena, OSWorld, WindowsAgentArena) primarily evaluate agents from a single, canonical initial state and measure success only at the end of the trajectory. This static evaluation overlooks critical real-world scenarios: (1) interfaces often lack default configurations, (2) users request help from intermediate task states, and (3) agents with low success rates may exhibit vastly different robustness and planning stability that cannot be revealed under a single start-state protocol. While some benchmarks address interface-level perturbations (e.g., layout shifts), they do not address task-progress-level variations (e.g., missing, redundant, or reordered steps).

Methodology

1. WorldGUI Benchmark

To address the gap in evaluating task-state variability, the authors introduce WorldGUI, a benchmark covering ten widely used desktop and web applications (Office, Windows Usage, Web, Coding, and Media).

• Data Construction: The benchmark consists of 611 tasks derived from 111 meta-tasks. Each meta-task is augmented five times to create diverse initial states.
• Augmentation Strategy: Instead of altering UI skins or resolutions, WorldGUI modifies the task progress itself using controlled pre-actions. These pre-actions are executed before the agent begins, creating three types of variations:
  • Add-step: Introducing unrelated state augmentations (e.g., opening random tabs) to simulate starting from an unrelated context, requiring the agent to replan and add necessary steps.
  • Trim-step: Completing specific steps of a long task to create an intermediate starting state.
  • Adjust-step: Making small modifications to the existing state (e.g., clicking a different tab or opening an unrelated dropdown) to challenge the agent's state understanding without necessarily requiring new steps.
• Components: Each task includes a user query, an instructional video, a project file for reproducibility, and ground-truth (GT) plans used solely for generating pre-actions, not provided to the agents.

2. WorldGUI-Agent Framework

The authors propose WorldGUI-Agent, a model-agnostic framework designed to improve reliability in dynamic environments through a "critical-thinking" approach. Unlike prior agents that rely on a single post-action checker, WorldGUI-Agent organizes planning and execution around three distinct critique stages:

1. Post-Planning Critique (Planner-Critic): After an initial plan is generated based on the user query and instructional video, this module verifies the plan against the current screenshot. It identifies "Wrong Steps," "Missing Steps," or "Redundant Steps" and self-corrects the plan before execution begins.
2. Pre-Action Validation (Step-Check): Before executing each subtask, this module validates whether the subtask is still necessary given the current state. It determines if a step should be executed (Continue), skipped (Pass), modified (Modify), or if the task is already finished (Finished). If the screenshot is ambiguous, a "Region Search" module is invoked to crop and re-evaluate specific areas.
3. Post-Action Evaluation (Actor-Critic): After an action is executed, this module compares the pre- and post-action screenshots to verify success. If the action fails, it triggers an iterative correction loop (up to a limit) to locate GUI elements and regenerate the action before proceeding.

Key Contributions

1. WorldGUI Benchmark: A systematic benchmark that introduces task-progress variability (Add/Trim/Adjust steps) to evaluate agent robustness, recoverability, and planning stability in non-default contexts. It is the first to explicitly evaluate agents on intermediate starting states across diverse desktop and web applications.
2. WorldGUI-Agent Framework: A lightweight, universal agent framework that integrates three critique stages (Planning, Step, and Action) into the standard perception-planning-execution loop. It demonstrates that structuring verification processes significantly improves performance without requiring model retraining.
3. Diagnostic Evaluation: The paper provides a comprehensive analysis of current state-of-the-art (SOTA) agents, revealing that while they perform reasonably well on canonical states, they suffer substantial performance degradation under non-default initial conditions.

Experimental Results

Experiments were conducted on WorldGUI and the WindowsAgentArena benchmark using various MLLMs (Gemini-2.0, GPT-4o, Claude-3.5-Sonnet, GPT-5.1) and specialized GUI models (UI-TARS-1.5, Agent-S2).

• Human vs. Agent Gap: A significant gap remains between human experts (85.3% success rate) and the best-performing agent (WorldGUI-Agent with GPT-5.1 at 45.8%).
• Robustness to Variability: Performance degrades sharply under augmented (non-default) conditions. For example, Claude-3.5-Sonnet dropped from 50.0% on Meta tasks to 19.1% on Augmented tasks in Windows Usage.
• Framework Effectiveness: WorldGUI-Agent consistently outperforms naive Plan-Act baselines and specialized GUI models. With GPT-4o, it improved the overall success rate by +17.5% over the Plan-Act baseline.
• Ablation Studies: Removing any of the three critique modules led to significant performance drops. The Actor-Critic module was found to be the most critical, with its removal collapsing the success rate to 9.7%, highlighting the necessity of action-level feedback.
• Specialized Models: Models like UI-TARS-1.5 and Agent-S2, which perform well on static benchmarks, struggle in WorldGUI, indicating that training pipelines tuned for single-start states do not automatically transfer to dynamically perturbed environments.
• Computational Cost: WorldGUI-Agent incurs a competitive running time (~129s) compared to other SOTA agents, with the primary cost attributed to the underlying MLLM calls.

Significance and Claims

The paper claims that WorldGUI and WorldGUI-Agent provide a foundation for developing more adaptable and reliable GUI agents. The authors emphasize that:

• Task-state variability is a pervasive but under-explored aspect of GUI interactions that current benchmarks fail to capture.
• Multi-level critique (planning, step, and action) is essential for handling dynamic environments where the state may deviate from expectations due to user behavior or system configurations.
• Current GUI agents exhibit fragile planning behaviors and limited robustness when faced with non-default initial conditions, revealing a critical area for future research in both perception and planning components.

The work suggests that future progress in desktop GUI automation requires moving beyond static, canonical evaluations toward benchmarks that reflect the messy, intermediate, and variable nature of real-world human-computer interactions.