Technical Summary: Design and Report Benchmarks for Knowledge Work

Problem Statement

The rapid development of Large Language Model (LLM) agents has expanded the scope of AI evaluation into "knowledge work"—labor where knowledge serves as both input and output, including domains like coding, research, healthcare, and administration. However, current benchmark design and evaluation logic largely adhere to traditional Natural Language Processing (NLP) paradigms. These paradigms typically evaluate bounded input-output behaviors (e.g., answer generation, summarization, retrieval) rather than the production of work artifacts within situated workflows.

Consequently, there is a misalignment between benchmark scores and the broader work-capability claims they are used to support. A high score on a component task (e.g., retrieving a document) does not reliably indicate a system's ability to perform the full knowledge work activity (e.g., clinical triage or legal analysis). This inference is fragile because the meaning of a knowledge-work output depends heavily on the role, materials, setting, and downstream workflow in which it is produced. Current benchmarks often fail to specify these contextual constraints, leading to scores that cannot reliably support claims about real-world deployment capabilities.

Methodology

The paper proposes a three-step design and reporting framework to explicitly tie benchmark scores to specific work claims. This approach is grounded in work studies, situated action theory, and validity theory, emphasizing that knowledge work is organized through roles, local materials, and artifacts that must be usable in downstream workflows.

1. Define the Work Activity

The authors argue that benchmark scope should be defined by "work activity" rather than broad domains (e.g., "healthcare") or narrow component tasks (e.g., "summarization"). To operationalize this, the paper derives an inventory of 18 cross-occupation work activities from the O*NET occupational task database (Job Zones 3–5).

• Derivation Process: Starting with 18,796 O*NET task statements, the authors filtered for knowledge-work occupations, removed manual/routine tasks, and retained 12,464 statements. A stricter screen for "atlas inclusion" yielded 8,372 statements, which were rewritten into profession-neutral phrases, embedded, clustered (using UMAP and HDBSCAN), and consolidated via expert panel review.
• Result: 18 distinct work activities (e.g., analysis, administration, design, inspection, troubleshooting, record-keeping, coordination, advising, investigation) were identified. These serve as the reporting unit for benchmark claims.

2. Specify the Tested Setting

The framework requires benchmarks to explicitly define the conditions under which the work activity is evaluated. The score's validity is conditioned by four dimensions:

• Materials: The artifacts available (documents, tools, databases) and whether they include irrelevant items or require the system to find them.
• Tools: The capabilities granted to the system (search, edit, calculate, API calls) and whether tool states persist.
• Role and Scope: The specific role the system plays, its decision boundaries, and escalation protocols.
• Workflow State: The specific stage of the workflow being tested (e.g., initial drafting vs. review vs. handoff) and the required state after completion.

3. Score the Proper Work Product

Scoring must focus on the "work product"—the object left by the system for review, filing, execution, or downstream continuation—rather than just the visible content (e.g., a chat response).

• Rubric Design: Evaluation criteria should be derived from the work product's requirements, checking:
  1. Whether the result used the appropriate materials.
  2. Whether the result adhered to the assigned role and workflow constraints.
  3. Whether the result left necessary information for downstream actors (e.g., traceable citations, handoff notes, state changes).
• Claim Limitation: The "Supported Claim" is defined as the strongest claim a score can support given the specific activity, setting, and product tested.

Key Contributions

1. A Three-Step Reporting Structure: A formalized approach for benchmark design that mandates explicit definitions of the work activity, tested setting, and scored work product to prevent overgeneralized capability claims.
2. O*NET-Based Work Activity Inventory: A preliminary inventory of 18 recurrent work activities derived from O*NET task statements, providing a standardized vocabulary for benchmark designers to map tasks to specific work claims.
3. Case Analysis Framework: The application of this framework to three existing benchmarks to demonstrate how design choices shape the supported claims:
   • GDPVAL: A non-code occupational deliverable benchmark. The analysis shows it supports claims about producing specific compliance deliverables under fixed prompts but lacks evidence for downstream workflow integration.
   • OFFICEQA PRO: A grounded document-analysis benchmark. The analysis reveals it supports claims about grounded numerical analysis but fails to score the analyst's reasoning trace or evidence package, limiting claims about document revision work.
   • APEX-SWE: A software engineering benchmark. The analysis shows it supports claims about completing integration tasks in a specific environment but does not cover production deployment, code review, or long-term maintenance.

Results and Findings

The paper does not present new experimental performance data but rather a structural analysis of existing benchmarks. The case studies demonstrate that:

• Gap Identification: Most benchmarks score a "proxy" (e.g., a final answer or a code patch) rather than the full "work product" (e.g., a revised document with traceable changes or a deployed system with rollback procedures).
• Claim Boundaries: Current benchmarks often support narrow claims (e.g., "can generate a patch that passes tests") but are frequently interpreted as broader claims (e.g., "can perform software engineering"). The proposed framework clarifies these boundaries.
• Inventory Utility: The 18-work-activity inventory successfully maps to diverse benchmarks, revealing that many current evaluations only partially cover specific activities (e.g., using "retrieval" as a proxy for "investigation" without testing evidence selection or reconstruction).

Significance and Claims

The paper modestly claims to provide a minimum reporting structure for keeping knowledge-work benchmark scores tied to the work they actually represent. It does not claim to replace deployment evidence or provide a complete theory of benchmark quality.

• Interpretability: The primary significance is making the evidence provided by a benchmark score explicit. By stating what activity is tested, under what setting, and which product is scored, stakeholders can better interpret what a score implies about a system's real-world capability.
• Prevention of Overclaiming: The framework aims to reduce "overbroad capability claims" by forcing benchmark reports to acknowledge gaps between the tested task and the broader work claim.
• Future Direction: The authors suggest that future benchmarks should move toward scoring work products more directly (e.g., executable state changes, reviewed artifacts) and that the work-activity inventory should be treated as a starting point for revision and expansion based on observed workflows and professional standards.

The paper concludes that while deployment studies remain necessary for claims about productivity and organizational value, benchmarks serve a distinct role as public, repeatable, and comparable pre-deployment evidence. This evidence is most useful when its limitations regarding work activity, setting, and product are explicitly stated.