Technical Summary: MYPCBENCH

Problem Statement

Current benchmarks for computer-use agents (CUAs) evaluate models in "impersonal" environments. These evaluations typically rely on empty desktops, generic application states, or minimally seeded databases where agents are explicitly told which applications to open and what workflows to execute. This approach creates a significant gap between research evaluation and real-world deployment, where personal assistants must operate across a user's entire digital life, leveraging context, historical data, and logged-in accounts.

This gap is particularly acute for web tasks. Live-web evaluations often avoid sites requiring login or personal information, yet these are precisely the environments a real personal assistant must navigate. Existing benchmarks like AppWorld seed coherent user identities at the API layer rather than within a lived-in desktop environment, failing to capture the cross-application entanglement of a real user's data (e.g., a single trip generating correlated records in banking, travel, email, and messaging apps). Consequently, current evaluations cannot determine if an agent is truly "personal" or if personalization performance improves with new model releases.

Methodology

The MYPCBENCH Environment

The authors introduce MYPCBENCH, a reproducible, open-source Linux desktop benchmark designed to evaluate agents as personal assistants. The environment is built around a single canonical persona, Michael Scott (from The Office), and is packaged as a Docker container running a QEMU/KVM Ubuntu 24.04 VM with GNOME Shell.

Key environmental features include:

• Cross-App Consistency: The environment seeds 17 simulated real-world web applications (clones of products like Airbnb, Chase, Delta, DoorDash, etc.) and a full desktop stack (LibreOffice, Firefox, file manager). Data is deterministically generated so that a single event (e.g., a trip) leaves correlated records across all relevant applications (banking charges, calendar blocks, email confirmations, chat messages, and browsing history).
• Persona Coherence: The persona is not a generic account but a specific individual with a defined social network, financial profile, routines, and preferences. The data seeding pipeline ensures that friends, co-workers, and preferences are entangled across all applications.
• Real-World Fidelity: The 17 web apps are full Next.js builds (not static mocks) that implement real workflows (e.g., generating QR code boarding passes, handling Zelle transfers). The UI and navigation match real-world analogues.
• Scale: The environment contains 1,812 bank transactions, 2,398 emails, 679 calendar events, 10,746 browser history visits, and thousands of other records.

Task Suite

The benchmark consists of 184 tasks inspired by real requests from the OpenClaw community. The authors manually curated these from 2,749 anonymized use cases, rewriting them to align with Michael Scott's seeded data.

• Task Types: Tasks are categorized into six behavioral types: Bounded Action, Multi-step Orchestration, Cross-source Reconciliation, Aggregation & Reporting, Personal Lookup, and Pattern Inference.
• Complexity: Tasks range from single-app interactions to those spanning up to 19 applications. 68% of tasks are multi-application, and 40% span at least two top-level web domains.
• Evaluation Rubric: Each task includes a natural-language instruction and a detailed rubric (3–13 criteria per task) generated by a coding agent and audited by the authors.

Agent Harness and Evaluation

The authors utilize a harness based on the OSWorld runner, treating the environment as a partially observable Markov decision process (POMDP).

• Observation: Agents receive 1280×800 screenshots of the desktop and a textual history of actions.
• Action Space: To ensure fair comparison, all models are evaluated using a uniform computer + bash tool surface. This includes standard GUI actions (click, type, drag) and shell commands.
• Grading: Evaluation uses an "LLM-as-a-judge" approach (using gemini-3.1-flash-lite-preview) to score trajectories against the rubric. Metrics include:
  • Perfect Rate: The fraction of tasks where every rubric criterion is met.
  • Rubric Score: A weighted sum allowing for partial credit.
  • Trajectory Efficiency: Rubric score per agent step.

Key Contributions

1. Reproducible Personalized Environment: A Docker-based Linux desktop seeded end-to-end with a coherent user identity across 17 web apps and a full desktop stack, enabling the evaluation of agents over persistent identity and cross-app history.
2. 184 Real-World Inspired Tasks: A task suite derived from actual user requests, featuring natural-language instructions and rubric-based grading for partial credit.
3. Comprehensive Benchmarking: Evaluation of six closed- and open-weight models (Claude Opus/Sonnet 4.6, GPT-5.5/5.4 mini, Qwen 3.5 35B/9B) under a uniform tool surface, accompanied by a failure taxonomy and scaling analysis.

Results

The authors evaluated six models on the full 184-task suite.

• Top Performance: Claude Opus 4.6 is the only model to fully solve more than 50% of tasks (55.4% perfect rate), with a rubric score of 81.8%. It significantly outperforms the next best model, Claude Sonnet 4.6 (39.1%).
• Model Variance: GPT-5.5 achieved a 29.3% perfect rate. Open-weight models performed significantly lower, with Qwen 3.5 35B-A3B at 7.6% and Qwen 3.5 9B at 2.7%.
• Complexity Scaling: Performance degrades sharply as task complexity increases.
  • Cross-App Complexity: For tasks touching 7 or more applications, the perfect rate drops to 36% for Opus and 4.5% for GPT-5.5. GPT-5.4 mini, Qwen 35B, and Qwen 9B reach 0% on these complex tasks.
  • Task Types: The largest gaps appear in tasks requiring reasoning over persona history or coordinating writes across multiple apps (Aggregation, Multi-step Orchestration, Pattern Inference).
• Efficiency: Claude Opus 4.6 demonstrates high trajectory efficiency (3.61 rubric points per step), whereas smaller models often loop unproductively or fail to utilize the tool surface effectively.

Failure Analysis

The authors classify failures into five modes, revealing distinct family-specific weaknesses:

1. Premature DONE: The GPT family frequently terminates tasks before completing all rubric criteria (354 total hits, 235 from GPT).
2. Skipped Required App: Agents fail to open necessary applications (323 hits).
3. Surface-Error Abandonment: Agents give up after UI errors (129 hits).
4. Partial Artifact: Agents complete only part of the task (47 hits).
5. Hallucinated Persona Data: Agents invent user data not present in the environment (31 hits), a failure mode dominated by Qwen models.

Additionally, Claude models tend to "shortcut" via bash scripts rather than interacting with the UI, while Qwen 9B collapses under the dual-tool (GUI + bash) schema.

Significance

MYPCBENCH addresses the critical gap between impersonal benchmarking and the reality of personal assistant deployment. The authors argue that current evaluations fail to test whether systems are truly personal or capable of handling the "entangled" nature of user data across applications.

The paper concludes that even the strongest frontier agents currently solve barely more than half of the tasks in a personalized, cross-application environment. The steep performance drop-off on long-horizon, multi-app tasks highlights that personalization remains a significant challenge. By releasing the environment, task set, and evaluation harness, the authors provide a baseline to drive research toward agents that can effectively operate within a coherent, persistent user identity.