GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks

1. Problem Statement

Current Graphical User Interface (GUI) agents primarily focus on full automation, where an AI executes a sequence of clicks and keystrokes to complete a predefined task. While efficient, this paradigm overlooks the human intention and agency required in real-world, open-ended workflows (e.g., creative design in Photoshop or data analysis in Excel).

• The Gap: Users often explore, iterate, and refine ideas, frequently revising their goals mid-task. Automated agents often treat this iterative behavior as redundant errors rather than essential parts of the creative process.
• The Challenge: To move from automation to collaboration, AI agents must understand what a user is doing, why they are doing it (intent), and when they need assistance. Existing benchmarks focus on task automation using expert demonstrations, failing to capture the confusion, trial-and-error, and decision-making processes of novice users.

2. Methodology

A. The GUIDE Dataset

The authors constructed GUIDE (GUI User Intent Detection Evaluation), a comprehensive benchmark designed to evaluate multimodal Large Language Models (MLLMs) on their ability to perceive and assist users.

• Data Collection: 67.5 hours of screen recordings from 120 novice users across 10 software applications (e.g., Photoshop, Figma, PowerPoint, Excel).
• Task Design: 40 open-ended tasks were designed to elicit natural, non-linear behaviors (e.g., "Design a music festival poster" rather than "Click these 5 buttons").
• Multimodal Input: Each session includes screen recordings and think-aloud narrations, providing ground truth for user cognitive states and intentions.
• Annotation Strategy: A human-AI collaborative pipeline was used. Initial annotations were generated by Gemini-2.5-Pro using the narrations and video, then verified and refined by human annotators (achieving >96% agreement on behavior states).

B. The Three-Stage Evaluation Framework

GUIDE defines three progressive tasks to test an agent's capability to move from perception to action:

1. Behavior State Detection:

   • Goal: Classify the user's current behavioral mode based solely on visual cues.
   • Taxonomy: A hierarchical taxonomy of 9 distinct states grouped into four phases:
     • Planning: Task Understanding, Ideation.
     • Execution: Exploration/Decision-Making, Performing Actions.
     • Problem-Solving: Frustration, Debugging, Seeking External Help.
     • Evaluation: Waiting/Monitoring, Assessment.
   • Format: Multi-class classification.

2. Intent Prediction:

   • Goal: Infer the user's immediate short-term goal (e.g., "Adjust the spacing between form elements").
   • Format: Multiple-Choice Question (MCQ) with 1 correct intent and 3 distractors.

3. Help Prediction:

   • Goal: Determine if the user needs assistance and, if so, what type.
   • Subtasks:
     • Help Need Detection: Binary classification (Need vs. No Need).
     • Help Content Prediction: MCQ identifying the specific help required (e.g., "Explain how to use layer masks").
   • Format: Sequential decision-making based on visual input.

3. Key Contributions

• First Open-Ended GUI Benchmark for Assistance: Unlike prior work focusing on expert automation, GUIDE focuses on novice users in open-ended workflows, capturing authentic moments of confusion and discovery.
• Structured User Understanding Taxonomy: Introduces a novel 9-state taxonomy aligned with cognitive theories (Norman's Seven Stages of Action, Bloom's Taxonomy) to model user cognitive states in GUIs.
• Context-Aware Evaluation: Demonstrates that providing structured context (behavior state + intent) significantly improves model performance, establishing a baseline for "context-aware" agents.
• Comprehensive Dataset: A large-scale, high-quality dataset of 120 hours of video with think-aloud protocols, available for the research community.

4. Experimental Results

The authors evaluated 8 state-of-the-art MLLMs (including Gemini-2.5-Pro, GPT-4o, Claude-4.5-Sonnet, and open-source models like Qwen3-VL and InternVideo) in a zero-shot setting.

• Baseline Performance (Visual Only):

  • Behavior State Detection: All models struggled, with the best model (Claude-4.5-Sonnet) achieving only 44.6% accuracy. Models frequently misclassified "Frustration" or "Debugging" as "Performing Actions," failing to detect subtle signs of struggle.
  • Help Prediction: Accuracy for predicting Help Need was around 55% for top models.
  • Intent Prediction: Performed relatively better (~60-70% accuracy) but dropped significantly under stricter Multi-Binary Accuracy (MBAcc) metrics.

• Impact of Context:

  • Providing structured user context (previous behavior state and inferred intent) significantly boosted performance.
  • Help Prediction saw the most dramatic improvement: providing behavior and intent context raised accuracy by up to 50.2 percentage points for lower-performing models.
  • Online vs. Offline: Models showed consistent performance gains as they were given more temporal context (progressively seeing 25%, 50%, 75%, 100% of the video), highlighting the importance of temporal history in proactive assistance.

5. Significance and Future Directions

• Shift from Automation to Collaboration: GUIDE argues that effective AI assistants should not replace user agency but rather accelerate exploration and provide timely, context-aware support.
• Limitations of Current MLLMs: The results reveal that current models lack the nuanced understanding of human cognitive states (e.g., distinguishing between deliberate iteration and confusion) when relying solely on visual inputs.
• Path Forward: The paper suggests that future GUI agents must leverage structured context engineering (layering behavior, intent, and history) to make accurate assistance decisions. This benchmark provides the necessary foundation for training the next generation of collaborative, user-aware AI agents.

In conclusion, GUIDE establishes that while current AI models can recognize basic actions, they fail to understand the human context of open-ended tasks. The benchmark serves as a critical step toward building AI that acts as a true collaborative partner rather than a rigid automation tool.