Technical Summary: Understanding User Experiences of Computer Use Agents

Problem Statement

Computer use agents—generative AI systems that automate actions within user interfaces (UIs) based on user commands—are rapidly emerging across desktop, mobile, and web environments. While current research heavily prioritizes training and evaluating the underlying Large Language Models (LLMs) and Multimodal LLMs (MLLMs), the User Experience (UX) design of these agents remains significantly understudied.

Existing design guidelines for agents were largely developed in pre-LLM contexts and do not address the unique challenges introduced by modern generative AI, such as capability uncertainty, output complexity, and the need to integrate perspectives from design, machine learning, and engineering. Consequently, there is a lack of a comprehensive mapping of the agent UX design space and a scarcity of tools to support designers and developers in prototyping these experiences. This gap leaves practitioners without a shared vocabulary for agent design or the necessary scaffolding to explore different interaction paradigms (e.g., varying levels of proactivity, autonomy, and transparency).

Methodology

The authors conducted a three-phase research program to address two primary Research Questions (RQs):

1. RQ1: What are the UX design considerations of computer use agents?
2. RQ2: What support is helpful to prototype UX in computer use agents?

Phase 1: Taxonomy Development (RQ1)

To map the design space, the authors employed a mixed-method approach:

• Literature and Product Review: The team reviewed 11 state-of-the-art computer use agents (announced 2024–2025) across desktop, browser, and mobile platforms.
• Expert Interviews: Eight UX and AI practitioners were interviewed to refine the initial taxonomy, identifying missing themes and extending sub-themes.
• Wizard-of-Oz Validation: A study with 20 end-users involved interacting with a mock agent (controlled by a researcher) to complete tasks in vacation booking and e-commerce domains. This validated the taxonomy's subcategories and ensured they reflected real user concerns.
• Analysis: Thematic analysis and provisional coding were used to synthesize findings into a structured taxonomy.

Phase 2: Requirements Elicitation (RQ2)

To understand the support needed for prototyping, the authors conducted a requirements elicitation study:

• Participants: 12 participants (6 agent experts, 6 non-experts) from a large technology company.
• Protocol: Participants reflected on agent UX factors through interviews and interactions with mock agents. They reviewed examples of agent patterns and discussed challenges in design and development.
• Analysis: Qualitative analysis of transcripts identified key Activities in the prototyping process and derived Desired Capabilities for a prototyping system.

Phase 3: Design Probe and Empirical Study (RQ2)

• Tool Development: The authors built AgentUXlab, a high-fidelity design probe (not a deployable product) that instantiates the identified design requirements and taxonomy. It features a no-code, graphical workflow editor and a browser-based execution environment.
• Empirical User Study: 14 participants with diverse backgrounds (engineers, designers, product managers, ML engineers) used AgentUXlab to prototype agent UX for a coffee ordering website.
• Protocol: Participants engaged in a 30-minute onboarding, a 30-minute prototyping exercise (designing an agent for a real website), and a 30-minute exit interview. They were encouraged to "think aloud" while designing, running, and debugging their agents.

Key Contributions

1. A Taxonomy of Agent UX Design Considerations

The paper presents a taxonomy comprising four main categories and 21 subcategories of UX considerations:

• User Query: Covers levels of expression, timing of queries, input modality, user profiles, ambiguity handling, contextual factors, and safety.
• Explainability of Agent Activities: Includes visibility of actions, retainability of history, descriptions of actions, transparency of reasoning (thoughts), previews of next steps, presentation of plans, and runtime status communication.
• User Control: Encompasses intervention during execution (stop/pause, take-over), handling high-impact scenarios (risk indicators, permissions), intervention on plans, and error recovery/discoverability.
• User Mental Model & Expectations: Addresses agent capabilities, UI context, the scope of the agent (what it can access/control), and risks (privacy, potential harm).

2. Design Requirements for Agent UX Prototyping Systems

Through the requirements elicitation study, the authors identified five Activities and six Desired Capabilities that a prototyping system must support:

• Activities:
  • A1: Design the scope and boundaries of the agent.
  • A2: Design the agent's information display.
  • A3: Design the interactions between the agent and user.
  • A4: Run the agent prototype.
  • A5: Understand the agent's runtime behavior.
• Desired Capabilities:
  • C1: Use no-code interfaces.
  • C2: Help constrain the agent's task space and knowledge of the user.
  • C3: Define the UI of the agent in the chat and the environment.
  • C4: Provide components that enable the agent to invoke different user interactions.
  • C5: Provide an environment to run and control the agent.
  • C6: Help debug the agent's runtime behavior.

3. AgentUXlab

The authors developed AgentUXlab, a design probe that serves as a concrete instantiation of the taxonomy and requirements. Key features include:

• Workflow Tab: A graphical, node-based editor where developers define agent behaviors (nodes) and conditions (edges). It supports bidirectional editing with natural language prompts.
• Prompt Interface: Allows structured prompt engineering, including agent capabilities, user information, and specific instructions.
• Execution Interface: A browser extension that runs the agent on real websites, displaying UI action previews, chat interactions, and runtime status.
• Debug Mode: Provides deep visibility into agent inputs (screenshots, accessibility trees) and outputs (reasoning, tool calls) to aid debugging.

Results and Findings

Findings on RQ1 (Design Space)

The taxonomy successfully validated that agent UX design extends beyond simple task automation. It highlights critical dimensions such as transparency of reasoning and user mental models, which are exacerbated by the autonomy of modern agents. The study confirmed that users require clear indicators of agent actions, the ability to intervene in high-risk scenarios, and mechanisms to manage expectations regarding agent capabilities and scope.

Findings on RQ2 (Prototyping Support)

The empirical study with AgentUXlab yielded several insights into the prototyping process:

• Workflow Notations as Visualization vs. Authoring: While participants valued workflow notations for visualizing agent behavior and debugging logic, many found them difficult to author. The notation was often more effective as a visualization tool than a primary authoring mechanism.
• Divergent Information Needs: A conflict exists between the information needs of end-users (who prefer less information to avoid overload) and prototypers (who require detailed logs, reasoning, and UI state to debug). Effective tools must support testing different personas.
• Scaffolding for Mental Models: Participants struggled to form accurate mental models of the agent's capabilities, particularly regarding available tools and constraints. Providing clear lists of accessible tools and interaction components helped scaffold the design space.
• Debugging Challenges: Participants found it difficult to interpret raw accessibility trees and hallucinated reasoning. They relied heavily on pausing execution to process information, suggesting a need for better controls to slow down or replay agent actions.
• Natural Language vs. Structure: While natural language prompts offered flexibility, participants often lacked the agency to make fine-grained modifications without additional scaffolds.

Significance and Claims

The authors position this work as a foundational step toward understanding and designing the UX of computer use agents.

• Shared Vocabulary: The taxonomy provides a structured vocabulary for designers and researchers to reason about agent UX factors, bridging the gap between early interface agent literature and modern generative AI.
• Prototyping Tooling: By identifying specific activities and capabilities, the paper argues that future agent development tools must move beyond simple workflow builders to include features that support UX-specific concerns like transparency, user control, and mental model alignment.
• Design Probe Utility: AgentUXlab demonstrates that design probes are effective instruments for exploring emerging design spaces where deployable tools do not yet exist. It validates the hypothesis that a no-code, visual approach can help developers explore the trade-offs between autonomy and control.
• Modest Scope: The authors explicitly note that the taxonomy is based on a specific set of agents and may not be exhaustive (e.g., accessibility was not covered). Similarly, AgentUXlab is a research probe for web agents, and the findings may require further validation for desktop or mobile environments. The results are presented as hypotheses for future controlled studies rather than definitive solutions.

In summary, the paper argues that as computer use agents become more autonomous, the focus must shift from purely model performance to the human-centered design of the interaction, necessitating new tools and frameworks that support the iterative exploration of agent UX.