Technical Summary: Workspace-Bench 1.0

Problem Statement

Current AI agents face a significant gap between their demonstrated capabilities in isolated tasks and their performance in real-world workplace scenarios. While foundation models and agent harnesses have advanced in tool invocation and GUI interaction, they struggle with Workspace Learning: the ability to identify, reason over, and update explicit and implicit dependencies among heterogeneous files within a complex digital workspace.

Existing benchmarks fail to adequately simulate authentic office workflows. They typically rely on:

• Prompt-Driven or Open-Source-Driven paradigms that bypass the need to process actual digital files.
• Task-File-Driven benchmarks that provide pre-packaged, isolated files, failing to require agents to independently search, filter, and discover information within a messy, hierarchical directory structure.
• Workspace-Relevant benchmarks that often lack persona diversity, cover fewer than 10 file modalities, and treat inter-file synergies as implicit byproducts rather than explicitly evaluating dependency identification (e.g., semantic relations, version lineage).

Consequently, there is an urgent need for a benchmark that evaluates agents on large-scale file dependencies within realistic, heterogeneous workspaces.

Methodology

Benchmark Construction (Workspace-Bench)

The authors constructed Workspace-Bench, a large-scale benchmark designed to simulate realistic digital workspaces and context-grounded office tasks.

1. Workspace Simulation:

   • Personas: Five distinct professional roles were defined: Operations Manager, Logistics Manager, AI Product Manager, Backend Developer, and Researcher.
   • Scale: The benchmark comprises 20,476 files (up to 20GB total) across 74 distinct file types (including .xlsx, .pdf, .md, .java, .yaml, .eml, etc.).
   • Structure: Workspaces were generated using a top-down, two-stage pipeline:
     • Structure Generation: Agents generated role-specific directory hierarchies with controlled structural noise (redundant folders, ambiguous names).
     • Content Population: A hybrid strategy combined semantic-driven agentic crawling of public resources (e.g., arXiv, GitHub) with LLM-synthesized artifacts (emails, meeting notes, reports) grounded in the retrieved data.
   • Complexity: Files are deeply nested (average depth 3.7, max 8) and include historical versions to test lineage tracing.

2. Task Curation:

   • Source: 388 tasks were curated from real-world workflows (sourced from Lark/ByteDance) and manually annotated by domain experts.
   • Dependency Graphs: Each task is paired with a ground-truth file dependency graph specifying the minimal set of essential file paths and their relationships (semantic, aggregative, lineage).
   • Difficulty Levels: Tasks are categorized into Easy (14%), Medium (53%), and Hard (33%) based on the complexity of required abilities (e.g., workspace exploration, lineage tracing, heterogeneous file understanding).
   • Evaluation Rubrics: A total of 7,399 fine-grained rubrics (avg. 19.1 per task) were designed to evaluate not just final output correctness, but also intermediate decisions, file version usage, and dependency adherence.

3. Evaluation Framework:

   • Agent-as-a-Judge: An auxiliary agent evaluates the execution trajectory against rubrics and dependency graphs.
   • Parallel Acceleration: A dual parallel mechanism (workspace-level and task-level) using sandbox pools to handle large-scale evaluation.
   • Metrics: Rubrics Pass Rate, Task Completion Rate (TCR@p), Dependency Graph Recognition Rate (Node/Edge F1), Token Consumption, and Interaction Turns.
   • Workspace-Bench-Lite: A 100-task subset preserving distributional fidelity for rapid evaluation, reducing costs by ~70%.

Experimental Setup

The authors evaluated 28 configurations combining 4 agent harnesses (OpenClaw, ClaudeCode, DeepAgent, Hermes) with 7 foundation models (including Opus-4.7, GLM-5.1, GPT-5.4, Kimi-2.5, etc.). Human experts were also evaluated using a "Human + Tools" baseline.

Key Results

1. Performance Gap: Current agents remain far from reliable workspace learning.

   • The average Rubrics Pass Rate across all configurations is 47.4%.
   • The best-performing configuration (OpenClaw + Opus-4.7) achieves 68.7%.
   • Human experts achieve 80.7%, highlighting a substantial gap between autonomous agents and human capabilities.

2. Difficulty Degradation: Performance degrades consistently as task complexity increases:

   • Easy Tasks: 57.6%
   • Medium Tasks: 49.2%
   • Hard Tasks: 40.5%

3. Capability Bottlenecks:

   • Heterogeneous File Understanding and Lineage Tracing are the primary bottlenecks, with agents performing significantly worse on these dimensions compared to Workspace Exploration.
   • Dependency Recognition: Agents struggle to deduce inter-file dependencies (Edge F1 scores are universally low), often identifying relevant files (Node F1) but failing to understand their relationships.

4. Efficiency vs. Accuracy:

   • High interaction turns and token consumption do not guarantee better performance. Some configurations (e.g., DeepAgent + MiniMax-M2.7) suffer from "cost explosions" (up to 58.1 turns, 0.61M tokens) with low success rates (~45%).
   • Top-tier configurations (e.g., OpenClaw + Opus-4.7) demonstrate high inference efficiency, achieving >65% accuracy with <20 turns.

5. Error Analysis: The majority of failures stem from Missing Content and Reasoning Errors (inaccurate aggregation, statistics, or association), rather than format or process errors.

6. Harness Impact: While foundation model capability is the primary driver, the harness framework significantly impacts performance, particularly for weaker models. For example, DeepAgent + GLM-5.1 shows unique stability across difficulty tiers, while Hermes excels in open-domain semantic interactions for Product Manager personas.

Significance and Contributions

The paper claims the following contributions:

1. Workspace-Bench Benchmark: A novel benchmark shifting evaluation from atomic skills to reasoning over complicated workspace structures. It introduces realistic, large-scale, heterogeneous file ecosystems with explicit dependency annotations, addressing the lack of "workspace-level" evaluation in existing literature.
2. Evaluation Framework: A workspace-grounded framework featuring dual parallel acceleration and an Agent-as-a-Judge paradigm, enabling fine-grained assessment of over 7,000 rubrics covering final correctness, intermediate reasoning, and operational efficiency.
3. Empirical Insights: Comprehensive evaluation revealing that state-of-the-art agents struggle significantly with dependency-aware tasks. The paper identifies specific bottlenecks (heterogeneous file understanding, lineage tracing) and the "Data Association Gap" between current isolated-file processing paradigms and the requirements of real-world workspaces.
4. Theoretical Framework: The proposal of Five Stages of Workspace Learning (from Data Insensitive Execution to Workspace-Native Self-Evolution). This framework characterizes the evolution of agent capabilities and identifies critical transition points, such as the "Orchestration Singularity" (where harnesses become more critical than LLMs) and the "Capability Singularity" (autonomous end-to-end task processing).

The authors conclude that while foundation models have made impressive progress, there is a fundamental and underexplored bottleneck in creating truly reliable productivity agents capable of navigating the complex, relational structures of real-world digital workspaces. Future work must focus on rethinking how agent harnesses discover, represent, and exploit cross-file dependencies.