Technical Summary: EnterpriseClawBench

Problem Statement

Large language models (LLMs) are transitioning from text-only assistants to agents operating within executable workspaces to produce business artifacts. However, existing evaluation frameworks face three critical gaps when applied to enterprise settings:

1. Realism vs. Scalability: Many benchmarks rely on human-authored or simulated tasks that fail to capture the complexity and noise of naturally occurring enterprise demands.
2. Multidimensional Evaluation: Current metrics often collapse performance into a single model score, ignoring the critical coupling between the agent "harness" (the scaffolding enabling tool use) and the base model, as well as the quality of delivered artifacts, cost, and runtime.
3. Skill Evaluation: Existing benchmarks rarely test reusable skills as transfer units. They often evaluate skills at the level of individual task items rather than assessing whether a skill distilled from one task class improves performance on held-out tasks within the same class.

Methodology

The authors introduce EnterpriseClawBench, a benchmark constructed from a proprietary archive of real-world agent sessions from an AI startup with over 100 employees. The data spans March to May 2026 and includes chat turns, uploaded files, tool traces, and persistent workspace states.

Construction Pipeline

The pipeline converts 5,291 raw task instances into 852 reproducible benchmark tasks through a rigorous filtering and transformation process (Figure 2 & 3):

1. Mechanical Gates: Candidates undergo length filtering, fixture recovery (ensuring input files exist), redaction recovery (restoring masked URLs/paths from context), and network reachability checks. This reduces the pool to 3,813 mechanically usable candidates.
2. Self-Containment & Rewriting: Tasks are reviewed for self-containment. Multi-turn, noisy conversations are rewritten into single-turn benchmark prompts with explicit material references and deliverable requirements.
3. Taxonomy Assignment: Tasks are labeled with role classes (e.g., Product, Engineering, HR) and skill subclasses (45 total subclasses across 8 role classes).
4. Rubric Generation: Each task is packaged with hard rules (objective checks for file types, counts, non-emptiness) and semantic rubrics for text and visual judging.
5. Sandbox Preflight: Final tasks undergo sandbox checks to ensure inputs can be uploaded, agents can execute, and artifacts can be downloaded.

Evaluation Framework

The benchmark evaluates harness–model combinations rather than models in isolation.

• Leaderboard: Covers 32 combinations of 5 harnesses (Claude Code, Codex, DeepAgents, Hermes, OpenClaw) and 9 models (including GPT-5.5, Sonnet 4.6, Opus 4.6, Kimi K2.6, etc.).
• Scoring:
  • Hard Rules: Objective checks for file existence, format, and content validity.
  • Semantic Judges: Multi-metric LLM judges score output quality on five dimensions: grounded accuracy, task relevance, substantive depth, practical utility, and communication quality.
  • Routing: Text-extractable outputs are judged by text models; visual artifacts (HTML, PDFs, spreadsheets, slides) are rendered as images for visual judges.
• Skill Evaluation: A specific protocol tests skill transfer by distilling a skill from in-domain tasks (e.g., frontend page generation) and injecting it into a consumer agent to evaluate performance on held-out tasks from the same subclass.

Key Results

Performance on Lite Subset (120 Tasks)

• Unsaturated Performance: The best configuration (Codex with GPT-5.5) achieves a score of only 0.663, indicating significant room for improvement in enterprise artifact delivery.
• Harness-Model Coupling: Performance is highly dependent on the harness. For instance, Claude-family models (Sonnet 4.6, Opus 4.6) perform well under Claude Code, DeepAgents, and OpenClaw (scores 0.62–0.64) but drop significantly under Hermes (0.458). Trace analysis suggests Hermes blocks necessary environment probing and script execution, leading to incomplete artifact generation.
• Cost-Effectiveness: There is a non-linear relationship between cost and score. Moving from low-cost to mid-cost systems yields large gains, but marginal returns diminish at higher cost tiers.
• Role and Artifact Variance:
  • Roles: Marketing and Finance/Ops tasks are the most difficult, likely due to heavy document comprehension requirements and company-specific conventions.
  • Artifacts: GPT-5.5 excels at HTML, code, and JSON, while Opus 4.6 leads in spreadsheets. Notably, visual judges tend to inflate scores for spreadsheets and presentations compared to text-only deliverables.
• Skill Transfer: Skill injection yields high variance. GPT-5.5 acts as a strong skill creator (positive average transfer), while Haiku 4.5 acts as a weak creator (negative average transfer). Crucially, a strong consumer (e.g., DeepAgents/Haiku 4.5) can still benefit from a weak creator, while strong consumers can be degraded by poor creator choices.

Judge Reliability

• Text vs. Visual: Text judges show strong agreement with human raters (Spearman correlation 0.790, MAE 0.134). Visual judges, however, show weak agreement with humans (Spearman -0.259, MAE 0.303), with Sonnet 4.6 being substantially more conservative than human evaluators. This highlights a maturity gap in multimodal artifact evaluation.

Key Contributions

1. Automated Construction Protocol: A reproducible pipeline that converts proprietary, noisy enterprise sessions into standardized benchmark tasks with fixtures, taxonomies, and rubrics.
2. Multidimensional Evaluation Framework: A protocol that jointly reports harness–model performance, artifact delivery quality, text/visual semantic scores, cost, and runtime, moving beyond single-score metrics.
3. Skill Generalization Support: Native infrastructure for evaluating skill transfer at the task-class level using a consumer–creator matrix.

Significance and Limitations

The paper argues that enterprise agent evaluation must evolve to report the harness–model combination and artifact delivery quality rather than collapsing performance into a single model score. The results demonstrate that current systems are far from solving real-world enterprise tasks, particularly those requiring complex document handling and visual output.

Limitations:

• Data Privacy: Due to the proprietary nature of the source data (internal sessions, attachments, business context), the benchmark dataset itself is not released. The contribution is the construction protocol and evaluation framework.
• Generalizability: The benchmark is derived from a single enterprise deployment, which may not represent all organizational structures or workflows.
• Judge Imperfection: LLM judges, particularly for visual artifacts, remain imperfect and require calibration against human raters.

The authors conclude that EnterpriseClawBench provides a necessary foundation for understanding the coupled behavior of agents in realistic, artifact-driven environments, emphasizing that "enterprise agent evaluation must report harness–model combinations, artifact delivery, visual quality, cost, runtime, and skill-transfer behavior."