💬 NLP

WorkSurface-Bench: Benchmarking Enterprise Agents on Multi-Surface Knowledge Routing

This paper introduces WorkSurface-Bench, a new benchmark comprising 1,151 auditable tasks designed to evaluate enterprise agents' ability to route queries across heterogeneous knowledge surfaces (documents, tables, and graphs), revealing that while agents can achieve high routing accuracy, correct surface selection alone is insufficient for high task completion rates.

Original authors: Hao Liang, Meiyi Qiang, Sizhe Qiu, Linzhuang Sun, Wentao Zhang

Published 2026-07-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hao Liang, Meiyi Qiang, Sizhe Qiu, Linzhuang Sun, Wentao Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a super-smart robot assistant to help you run a busy office. You ask it a complicated question, like, "How much did we spend on shipping last month, and which files are causing the delays?" To answer this, the robot can't just guess; it needs to look up information. But here's the catch: the office has three very different kinds of information shelves. There's a Library full of written reports and PDFs (documents), a Spreadsheet Room packed with rows of numbers and calculations (tables), and a Map Room showing how every file connects to every other file (graphs).

For a long time, scientists testing these robots mostly asked, "Did you get the right answer?" If the robot looked in the wrong room, picked the wrong book, or did the math wrong, the test just said, "Fail." But that doesn't tell us why it failed. Did the robot not know it needed to go to the Map Room? Or did it go to the Map Room, find the right map, but then get confused while reading it? This paper steps into that gap. It introduces a new way to test robots by separating the act of choosing the right room from the act of solving the puzzle once inside. It asks: Can the robot figure out which type of knowledge it needs before it even starts looking?

The New Game: WorkSurface-Bench

The researchers built a new testing ground called WorkSurface-Bench. Think of it as a giant, realistic office simulation with 1,151 different tasks. They set up five different "personas" (like a logistics manager or a financial analyst) and gave them access to three distinct surfaces: a document library, a table database, and a file-relationship graph.

The big innovation here is how they grade the robots. Instead of just giving a pass or fail on the final answer, they score two separate things:

Route: Did the robot pick the right "surfaces" (Library, Spreadsheet, or Map)?
Answer: Did it actually solve the problem correctly once it picked them?

They created a "gold standard" for every answer. If a task requires a math calculation, they ran the actual code to get the true answer. If it required reading a document, they checked the exact words in the source. If it needed a file connection, they traced the actual path. This means the "correct" answer isn't just a guess by another AI; it's a verified fact.

The Big Surprise: Knowing Where to Look Isn't Enough

The team tested four of the smartest AI models available (including GPT-4o-mini, DeepSeek-V4-Pro, Gemini-3.1-Pro, and GPT-5.5) in six different scenarios. They wanted to see if making the robot's job easier by telling it exactly which room to go to would fix its mistakes.

Here is what they found, and it's a bit of a plot twist: Knowing where to look does not guarantee you will find the answer.

When the researchers forced the robots to use only the correct surfaces (a "Gold-constrained" setting), the robots became almost perfect at choosing the right room. Their "Route" score jumped to between 98.7% and 99.8%. They knew exactly where to go.

However, their "Answer" score—the actual correctness of the final solution—only reached between 56.1% and 75.3%.

This means that even when the robot was standing in the right room with the right map in hand, it still got the answer wrong more than a quarter of the time. It might have found the right file but misread the numbers, or it might have combined information from the spreadsheet and the document incorrectly. The paper shows that "choosing the right tool" and "using the tool correctly" are two completely different skills, and being good at one doesn't automatically make you good at the other.

Hints vs. Restrictions

The researchers also played a fun game of "give a hint" versus "take away distractions."

The Hint: They told the robots, "You need to look at the Spreadsheet and the Map." This helped the robots choose the right surfaces better, and for some models, it slightly improved their final answers.
The Restriction: They then took away all the other tools the robots didn't need. This made the robots much faster and more efficient, and they chose the right surfaces even better. But, surprisingly, this didn't always make the final answer better. In fact, for some models, removing the extra tools actually made them slightly worse at solving the problem, likely because they lost the ability to double-check their work with a different tool.

What This Means for the Future

The paper concludes that we can't just look at the final score of an AI agent anymore. If an AI gets a question wrong, we need to know if it failed because it didn't know what to look for, or because it looked in the right place but couldn't process what it found.

The authors measured this across 27,624 different attempts by the robots, and they are very sure of these numbers because they audited the results with human checkers and strict rules. They found that while the best robots are getting very good at routing (98%+), they are still struggling with the final step of putting the pieces together correctly (around 56–75%).

In short, the future of smart office robots isn't just about giving them better maps of the office; it's about teaching them how to read the signs and do the math once they arrive. The paper provides a new, detailed scoreboard to help developers track exactly where their robots are stumbling, so they can fix the specific part of the brain that's broken, rather than just hoping the whole thing gets better.

Technical Summary: WorkSurface-Bench

Problem Statement

Enterprise agents frequently encounter questions requiring simultaneous access to heterogeneous knowledge forms: narrative facts in documents, calculations in tables, and file relationships in dependency graphs. Existing benchmarks typically evaluate retrieval (RAG) or tool use in isolation, failing to distinguish between an agent's ability to select the correct knowledge representation (surface routing) and its ability to reason correctly using that representation. Consequently, end-to-end accuracy scores conflate these distinct capabilities, making it difficult to diagnose whether a failure stems from choosing the wrong source, retrieving incomplete evidence, or synthesizing the answer incorrectly.

Methodology

The authors introduce WorkSurface-Bench, a benchmark designed to decouple surface routing from answer generation. The methodology involves three core components:

1. Benchmark Construction

The benchmark is derived from the Workspace-Bench-Lite corpus, projected onto three canonical, routable surfaces:

Document (RAG): 493 canonical UTF-8 markdown documents.
Table: 207 DuckDB views generated from normalized CSV/XLSX sheets.
Graph: 1,438 edges representing file lineage and dependencies (e.g., file-depends-on-file).
Metadata: Standard Operating Procedures (SOPs) are retained as task metadata rather than a routable surface.

The dataset comprises 1,151 atomic tasks across five persona-scoped workspaces. Tasks are categorized by required surfaces: rag_only, table_only, graph_only, and cross_surface (requiring combinations of the three).

2. Verifiable Gold Standard

To ensure auditability, reference answers are not generated by models but derived from deterministic execution:

Table answers: Executed via DuckDB queries.
Document answers: Verified against verbatim spans in source files.
Graph answers: Derived from source dependency annotations.
A rigorous screening pipeline involving deterministic checks, LLM-assisted proposals (GPT-5.5), and independent human/LLM audits (DeepSeek, Gemini) ensures the final set contains no protocol errors and passes strict criteria for answerability and surface necessity.

3. Scoring Protocol

The evaluation framework scores four distinct metrics for each task:

Route F1: Measures the precision and recall of the selected surface set against the gold-required set.
Evidence: Measures the fraction of required artifacts (files, views, nodes) actually accessed in the trace.
Answer: Evaluates final correctness using exact match, relative error tolerance, or set F1, depending on the answer type.
Efficiency: A continuous score penalizing token usage relative to a task-specific budget.

These metrics are aggregated into a single score with weights: Answer (35%), Evidence (30%), Route (25%), and Efficiency (10%).

Experimental Setup

The authors evaluated four backbone models (GPT-4o-mini, DeepSeek-V4-Pro, Gemini-3.1-Pro, and GPT-5.5) under six agent settings:

No-tool: Closed-book baseline.
Always-RAG: Forces a single document search.
Naive-router: Selects exactly one surface upfront.
ReAct-all: Exposes all three surfaces, allowing free interleaving.
Gold-constrained: Agent is given the required surfaces but must select tools within them.
Gold-hint/all: Agent is given the required surfaces but all tools remain available.

Key Results

The experiments, covering 27,624 retained trajectories, yield several critical findings:

Routing $\neq$ Answering: There is only a moderate correlation ( $\rho = 0.62$ ) between routing success and answer accuracy. Even when agents are Gold-constrained (forced to use the correct surfaces), achieving near-perfect Route F1 (98.7–99.8%), their Answer accuracy remains significantly lower (56.1–75.3%).
Post-Routing Bottlenecks: High routing scores do not guarantee correct execution. Under Gold-constrained settings, agents still fail at evidence retrieval (Evidence scores: 66.7–78.5%) and downstream computation/synthesis.
Impact of Hints vs. Restrictions:
- Providing surface hints (S4 $\to$ S6) improves Answer accuracy for three of the four models but has a variable effect on routing.
- Removing irrelevant tools (S6 $\to$ S5) significantly improves Route F1 and Efficiency but does not uniformly improve Answer accuracy; in some cases, it degrades performance.
Surface-Specific Performance: Performance varies by task type. While Table-only tasks generally yield the highest scores (up to 91.0%), Cross-surface tasks remain the most challenging, with scores dropping 13.0–45.7 points below a model's best single-surface performance.

Significance and Claims

The paper claims that WorkSurface-Bench is the first workspace benchmark to score surface routing separately from the Answer metric. Its primary significance lies in:

Diagnostic Clarity: By separating metrics, the benchmark allows researchers to pinpoint whether an agent fails at selection, retrieval, or synthesis, rather than attributing all failures to a monolithic "intelligence" score.
Decoupling Abilities: The results demonstrate that knowing where to look (routing) is a distinct capability from knowing how to use the found information (answering).
Evaluation Rigor: The use of executable, auditable gold answers (SQL, spans, annotations) eliminates reliance on model-generated references, providing a more reliable ground truth for enterprise agent evaluation.

The authors conclude that future agent evaluations must report surface selection separately and explicitly test whether systems can convert retrieved evidence into correct answers, rather than relying solely on end-to-end accuracy.