Technical Summary: WorkstreamBench

Problem Statement

Large Language Model (LLM) agents are increasingly expected to execute end-to-end workflows, producing complete artifacts from high-level instructions. In the financial sector, core workflows such as modeling, forecasting, and scenario analysis are predominantly conducted via spreadsheets. While frontier AI labs have developed agents capable of constructing spreadsheets from scratch, existing benchmarks fail to evaluate this advanced capability. Current evaluations (e.g., SpreadsheetBench) focus on narrow, atomic tasks such as single-cell edits, question answering over tabular data, or generating a single correct formula. These benchmarks rely on exact-matching verification, which is insufficient for professional financial deliverables.

In real-world finance, a spreadsheet is not a one-off calculation but a collaborative artifact reviewed by multiple stakeholders (analysts, managers, VPs, clients). Consequently, the quality of a deliverable extends beyond numerical correctness to include readability, structural clarity, and flexibility for modification. The community lacks a transparent, holistic benchmark to evaluate agents on the end-to-end construction of professional-grade financial spreadsheets.

Methodology

1. Benchmark Construction (WorkstreamBench)

The authors curated a diverse set of end-to-end spreadsheet tasks from three primary sources: the Financial Modeling World Cup (FMWC), ModelOff, and the Wall Street Prep (WSP) training curriculum. These tasks reflect real workplace expectations, requiring the construction of multi-sheet workbooks.

• Scale: Compared to SpreadsheetBench, WorkstreamBench tasks contain approximately 33× more cells (mean) and 93× more function calls (median).
• Difficulty & Types: Tasks are annotated into six difficulty levels (1–5) and categorized by type (e.g., 3-Statement Models, DCF Valuation, Debt Scheduling). Level 2 tasks are feasible without deep financial expertise, while Level 4+ tasks require assistance from seasoned modelers.

2. Evaluation Taxonomy

To capture the multidimensional nature of professional spreadsheet quality, the authors propose a taxonomy comprising three core dimensions, each decomposed into fine-grained sub-dimensions:

• Accuracy (50% weight): Assesses whether the agent performs the desired task (e.g., scenario analysis) and if the underlying computations are sound. Sub-dimensions include Final Calculations, Starting Values, Task Completed, and Sign Consistency.
• Formula (35% weight): Evaluates the robustness and interpretability of cell-level computations. Sub-dimensions include Logic Readability & Size (avoiding monolithic "mega-formulas"), Handling Edge Cases (e.g., #DIV/0!), Avoiding Hardcoding, Range Hygiene, and Absolute References.
• Format (15% weight): Measures clarity and professionalism. Sub-dimensions include Workbook & Sheet Structure, Number Notation (e.g., parentheses for negatives), Readability, Color Standards, and Borders & Shading.

3. Agent Collection and Execution

The study evaluates a wide range of agents:

• GUI Agents: Proprietary products like Claude for Excel and ChatGPT for Excel, accessed via a custom Playwright pipeline that automates browser interactions.
• API Agents: Frontier models (e.g., Claude Opus, GPT-5, Gemini, Grok, OLMo) integrated into a custom agentic framework using the Model Context Protocol (MCP) and OpenPyXL for direct file manipulation.

4. LLM-as-Judge Evaluation

Given that many quality criteria (e.g., logic readability, formatting) cannot be verified via exact string matching, the authors employ an LLM-as-Judge pipeline.

• Process: The judge receives the evaluation rubric, the reference solution, and the agent's output. It outputs a structured JSON verdict (Pass/Fail) with detailed explanations for failures.
• Validation: The judge's reliability was validated against 408 expert annotations (by 2 MBAs and 3 finance professionals, totaling 700+ hours). The judge achieved 0.92 accuracy, 0.88 balanced accuracy, and 0.85 F1 score against human experts. It successfully detected subtle errors missed by exact-match verifiers, such as off-by-one errors in ranges that coincidentally yielded correct numerical results, and hardcoded values that bypassed formula logic.

Key Results

• Performance Gaps: Even the strongest agents fall short of professional finance standards. The top-performing agent, Claude (Web), achieved a composite score of 69.1/100.
• Dimension Breakdown: Claude (Web) outperformed all others across Accuracy, Formula, and Format. However, significant gaps remain; for instance, many agents struggle with Logic Readability, often producing monolithic formulas that are difficult to audit.
• Difficulty Degradation: Performance degrades sharply as task difficulty increases. While agents perform reasonably well on Level 2 tasks, scores drop significantly for Level 4 and 5 tasks (e.g., Claude (Web) drops to 53.4/100 at Level 4).
• GUI vs. API: Proprietary GUI agents (e.g., Claude for Excel) generally underperform their web-based counterparts (e.g., Claude Web), suggesting that the agentic harness and interface play a critical role in performance.
• Common Failure Modes:
  • Hardcoding: Agents frequently hardcode values instead of using formulas, making models uneditable.
  • Formatting Inconsistencies: Issues with font colors, unit labels, and number notation (e.g., using minus signs instead of parentheses for negatives).
  • Incomplete Outputs: On harder tasks, agents often leave sheets blank or fail to construct the full multi-sheet structure.
  • Structural Errors: Failure to use absolute references or handle edge cases like division by zero.

Significance and Claims

The paper claims to be one of the first evaluations of LLM agents on end-to-end spreadsheet tasks in a critical domain (finance). Its primary contributions are:

1. WorkstreamBench: A benchmark that moves beyond atomic tasks to evaluate the construction of complete, multi-sheet financial models.
2. Holistic Taxonomy: A quality framework that operationalizes professional standards (Accuracy, Formula, Format) which are essential for collaborative, high-stakes financial work but previously unmeasured in AI benchmarks.
3. Validated Evaluation Pipeline: A scalable, transparent LLM-as-judge system that aligns closely with human expert judgment, capable of assessing nuanced criteria like "Logic Readability" that are difficult to verify programmatically.

The authors conclude that while state-of-the-art models can handle simpler end-to-end financial tasks, current agents are not yet able to reliably produce professional-quality spreadsheets at the complexity levels demanded by real-world workflows. The significant performance degradation on harder tasks and the prevalence of non-compliance with professional formatting and engineering standards indicate substantial headroom for improvement before these agents can be trusted for enterprise-grade financial modeling.