A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directions

1. Problem Statement

Agents for Computer Use (ACUs) are AI systems designed to execute complex tasks on digital devices (PCs, mobile phones) by interpreting natural language instructions and performing low-level actions (e.g., mouse clicks, keystrokes, touch gestures) to control software interfaces.

Despite rapid progress driven by Foundation Models (LLMs, VLMs), ACUs are not yet mature for everyday deployment. The field faces significant hurdles:

• Generalization Failure: Agents struggle to adapt to dynamic UI changes, multiple applications, and dense environments (e.g., spreadsheets).
• Robustness Issues: Success rates are significantly lower than human performance due to an inability to handle non-standardized interfaces.
• Fragmented Research: The field lacks a unified terminology, taxonomy, or standardized evaluation, making it difficult to compare approaches or identify systemic gaps.
• Research-Reality Gap: Many benchmarks rely on sanitized, static environments that do not reflect real-world complexity, leading to agents that fail in practical deployment.

2. Methodology

The authors conducted a broad, technology-agnostic survey to map the current state of the art.

• Data Collection: They curated a dataset of 87 original research papers on ACUs and 33 relevant datasets published between 2018 and 2024.
• Selection Criteria:
  • Focus on deep learning-based agents (excluding rule-based systems).
  • Requirement for active execution (generating executable code or actions) rather than passive advice.
  • Exclusion of non-computer-use domains (e.g., video games, server control, pure coding assistants).
• Process: An iterative, semi-structured collection process was used, combining expert keyword searches with "snowballing" (backward and forward citation tracking) to ensure thematic saturation.
• Analytical Framework: The authors developed a multifaceted taxonomy grounded in intelligent agent theory (Russell & Norvig; Sutton & Barto) to structure the literature across three distinct perspectives.

3. Key Contributions: The Proposed Taxonomy

The paper's primary contribution is a comprehensive taxonomy organizing ACUs across three complementary perspectives (visualized in Figure 4 of the paper):

A. Domain Perspective (The Environment)

Classifies the operating environments and their interfaces:

• Domains: Web, Android, and Personal Computers (Desktop).
• Observation Types: Image (screenshots), Text (HTML, accessibility trees), and Indirect (file system, network traffic).
• Action Types: Mouse/Touch/Keyboard (low-level coordinates), Direct UI Access (element IDs), Task-Tailored Actions (high-level commands), and Executable Code.
• Finding: Research heavily favors Web and Android; Desktop environments are underexplored despite high practical value.

B. Interaction Perspective (Agent ↔ Environment)

Analyzes how agents perceive and act:

• Observation Spaces: A shift is occurring from text-based (HTML) to image-based (screenshots) observations, driven by Vision-Language Models (VLMs). Text-based observations often fail in real-world settings due to inconsistent DOM structures.
• Action Spaces: While coordinate-based actions align with visual inputs, many current systems use "modality bridging" (e.g., converting screenshots to text to use element IDs), which the authors argue is a temporary workaround.
• Action Grounding: The process of converting abstract plans (e.g., "click submit") into executable actions. Current methods rely heavily on "Set-of-Marks" prompting or rule-based matching, which may become obsolete as VLMs improve spatial grounding.

C. Agent Perspective (Internal Structure)

Examines the agent's architecture and learning dynamics:

• Agent Types:
  • Foundation Agents: Use pre-trained LLMs/VLMs as policies. They rely on in-context learning and prompting.
  • Specialized Agents: Custom architectures trained via Reinforcement Learning (RL) or Behavioral Cloning (BC).
• Policy Types:
  • Memoryless: Acts only on current input.
  • History-based: Uses a growing sequence of past observations/actions (common in foundation agents).
  • State-based: Compresses history into a fixed internal state (common in specialized agents).
• Learning Strategies:
  • General Pre-training: Broad knowledge acquisition.
  • Environment Learning: Adaptation via RL, BC, or Long-Term Memory (storing successful trajectories).
  • Episodic Improvement: In-context learning (few-shot, Chain-of-Thought) within a single episode.

4. Key Results and Findings

The survey identifies six critical research gaps and current trends:

1. Generalization vs. Specialization: The field is transitioning from specialized agents to foundation-model-based agents. However, agents trained on structured text (HTML) fail to generalize to real-world, messy UIs. Image-based observations are shown to be more robust for generalization.
2. Inefficient Learning: Current learning strategies are costly. RL requires expensive simulations; Behavioral Cloning requires massive labeled datasets. There is a lack of effective, scalable adaptation methods for foundation agents.
3. Limited Planning: Agents struggle with long-horizon, multi-step tasks. Current planning is often implicit or relies on weak Chain-of-Thought prompting rather than robust search or symbolic planning.
4. Benchmark Complexity: Existing benchmarks (e.g., MiniWoB++) prioritize perception realism but lack task complexity. They often feature short, linear tasks that do not test causal dependencies between steps.
5. Non-Standardized Evaluation: There is no consensus on metrics. Studies use different subsets of benchmarks and mix task-level vs. step-level metrics, making cross-paper comparison impossible.
6. Research-Deployment Disconnect: Research assumes static, deterministic environments. Real-world environments are dynamic (notifications, updates) and non-stationary, leading to a disconnect between lab performance and real-world utility.

5. Recommendations and Future Directions

The authors propose six strategic directions to advance the field:

1. Visual Perception: Prioritize image-based observations and low-level control (mouse/keyboard) to enhance generalization across diverse UIs.
2. Adaptive Learning: Move beyond static prompting. Develop cost-effective, self-supervised fine-tuning stages (similar to RLHF) to align foundation models with computer-use contexts before environment learning.
3. Advanced Planning: Integrate robust planning algorithms and leverage reasoning-oriented models (e.g., OpenAI o1) to handle long-horizon tasks and dynamic environments.
4. Complex Benchmarks: Create datasets that combine realistic perception with high task complexity (long, causally dependent action sequences).
5. Standardized Evaluation: Adopt Task Success Rate as the primary metric on complete, established benchmarks. For offline datasets, report Step Success Rate as a proxy but acknowledge its limitations.
6. Real-World Alignment: Design agents that account for dynamic, non-stationary environments and incorporate safety mechanisms (e.g., conditional autonomy, user confirmation for critical actions).

6. Significance

This survey serves as a foundational reference for the ACU community. By establishing a unified taxonomy and identifying specific bottlenecks, it moves the field beyond isolated experiments toward a cohesive research agenda. The paper highlights that while foundation models have accelerated progress, true "general-purpose" computer use requires solving fundamental problems in generalization, planning, and real-world robustness. It provides a roadmap for researchers to build agents that are not just capable in controlled simulations but are reliable and safe for everyday human-computer interaction.