🤖 AI

BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents

The paper introduces BigFinanceBench, a comprehensive 928-item benchmark designed to evaluate financial-research agents by assessing their full, auditable derivation workflows through detailed rubrics rather than relying solely on final answer accuracy.

Original authors: Alex Wang, Georg Meinhardt, Jacob Katz, Joseph H. Kim, Pratyush K. Chaudhary, Chase Blagden, Eric Xu

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alex Wang, Georg Meinhardt, Jacob Katz, Joseph H. Kim, Pratyush K. Chaudhary, Chase Blagden, Eric Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a brilliant financial detective to solve a complex mystery: "Is this company's reported profit accurate, or are they hiding something?"

In the past, if you asked an AI this question, you would only care about the final number it gave you. If the AI said, "The profit is $10 million," and that was the right number, you'd give it a gold star. But in the real world of finance, a gold star isn't enough. A human expert would ask: "How did you get there? Did you look at the right document? Did you use the right time period? Did you know how to adjust for software costs?" If the AI got the right answer by guessing or hallucinating, it's still a failure because the "why" is broken.

BIGFINANCEBENCH is a new test designed to stop AI from getting gold stars for lucky guesses. Here is how it works, broken down into simple concepts:

1. The "Recipe" vs. The "Cake"

Most old tests for AI are like judging a baker only by how the cake tastes. If the cake is sweet, the baker passes.
BIGFINANCEBENCH is different. It judges the baker on the entire recipe.

Did they pick the right flour (source)?
Did they measure the eggs correctly (data extraction)?
Did they mix the ingredients in the right order (accounting logic)?
Did they bake it at the right temperature (calculation)?

The test gives the AI a "checklist" (called a rubric) with 36,000 tiny steps. Even if the final answer is wrong, the AI can still get points for doing the earlier steps correctly. This is like giving a student partial credit for showing their math work, even if they made a small arithmetic error at the end.

2. The "Expert Panel"

The questions in this test weren't written by computers or simple quizzes. They were written by real financial experts—people who have worked in investment banking and private equity.

The Challenge: They wrote questions that are specifically designed to trip up current AI. They asked things that require digging through multiple documents, making smart assumptions, and doing complex math.
The Audit: Before a question was added to the test, a different expert reviewed it to make sure there was only one correct way to solve it, just like a referee checking a sports play.

3. The "Scorecard"

When the AI takes the test, it doesn't just give an answer. It has to show its work:

Search: It has to find the right financial reports (like 10-K forms).
Extract: It has to pull specific numbers out of those reports.
Adjust: It has to apply accounting rules (like realizing that software development costs should be treated differently).
Calculate: It has to do the math.

Two independent "judge" AIs then grade the detective's work against the expert checklist. They don't just look at the final number; they look at the trail of breadcrumbs the AI left behind.

4. What They Found (The Results)

The researchers tested 10 of the smartest AI models available today. Here is what happened:

The Ceiling is Low: Even the best AI only got about 59% of the total possible points. This means there is still a huge gap between what AI can do and what a human financial analyst can do.
The "Lucky Guess" Problem: If you only looked at the final answers, the AI scores looked slightly better. But when you looked at the steps (the rubric), the scores dropped. This proves that AI often gets the right answer for the wrong reasons, or misses crucial steps along the way.
Specialization: No single AI was the best at everything. One AI was great at finding documents but bad at math; another was great at math but bad at finding the right source. It's like having a team of specialists rather than one "super-hero" who can do it all.

The Bottom Line

This paper argues that to trust AI with money, we can't just ask, "Is the answer right?" We have to ask, "Can you prove how you got there?"

BIGFINANCEBENCH is a tool that forces AI to show its homework. It shows us that while AI is getting better, it still struggles with the messy, step-by-step reality of real-world financial research. It's not just about knowing the answer; it's about knowing the story behind the answer.

Technical Summary: BIGFINANCEBENCH

Problem Statement

Financial research answers are only decision-relevant when their derivation is auditable. Practitioners must verify not just the final number, but the specific sources chosen, accounting definitions applied, assumptions made, and calculation steps performed. Existing financial benchmarks largely evaluate isolated subskills (e.g., retrieval, single-fact lookup) or final answers, leaving the auditable derivation itself under-measured. Furthermore, typical benchmark tasks often pair a query with a single discrete input (e.g., one page or filing), failing to capture the open-ended, multi-source, and assumption-dependent nature of real-world analyst work. This creates a gap where current evaluations do not reflect the "derivation-level resolution" used in expert review.

Methodology

The authors introduce BIGFINANCEBENCH, a workflow-grounded benchmark designed to evaluate financial-research agents on practitioner-inspired question shapes and at the resolution of expert derivation review.

Dataset Construction

Scale and Composition: The dataset contains 928 items authored by 52 financial-research subject-matter experts (predominantly current and former investment bankers and private equity investors).
Quality Control: Each item was audited by a separate panel of 12 reviewers. Questions were rejected if they merely stacked simple lookups; they were accepted only if they required depth of reasoning, niche expertise, or domain-specific judgment.
Objectivity: Questions are time-anchored to specific points in time and require answers fixed by expert consensus under legitimate methodologies. Non-obvious assumptions are explicitly stated in the query.
Rubric Design: Each item pairs a ground-truth reference answer with a point-weighted rubric derived from the analyst workflow.
- Structure: Rubrics decompose the derivation into independently verifiable steps (e.g., entity identification, source selection, line-item retrieval, accounting adjustment, formula construction).
- Criteria: There are 15,656 rubric criteria totaling 36,241 points.
- Authoring Rules: Criteria are binary (yes/no), precise (specifying exact values and periods), atomic (isolating single concepts), self-contained (gradable without other criteria), and weighted (1–10 scale).
Workflow Grounding: The benchmark preserves the open-ended, multi-source structure of analyst work. It is not a frequency-weighted census but a frontier evaluation over realistic workflow families.

Evaluation Setup

Agent Harness: Ten frontier and open-weight models were evaluated in a common open-book ReAct-style harness with a 50-step budget.
Tool Surface: The tool set is minimal and public-source only: web_search, edgar_search (SEC filings), fetch_url, and python_exec (sandboxed calculation). No proprietary financial databases are used.
Judging: Two independent LLM judges (Gemini 3.1 Pro Preview and Claude Opus 4.7) grade each trajectory. They evaluate correctness for every rubric criterion and the final answer.
Metrics:
- Rubric Score: The primary metric, defined as earned rubric points divided by total rubric points (macro-averaged).
- Final-Answer Accuracy: The fraction of final answers judged correct.
- Rubric Criterion Pass Rate: The unweighted fraction of criteria passed.

Key Contributions

Auditable Finance Workflows: The introduction of 928 expert-authored questions requiring source selection, accounting judgment, and calculation over public evidence, moving beyond isolated QA or retrieval tasks.
Derivation-Level Evaluation: A novel grading framework using point-weighted rubrics over independently checkable workflow steps, enabling partial-credit evaluation and localization of failures across the analyst workflow.
Empirical Diagnosis of Current Agents: A comprehensive evaluation of ten models revealing that current systems have substantial headroom and that model capabilities vary non-uniformly across different financial workflows.

Results

Unsaturated Performance: The benchmark remains far from saturated. The best system (Claude Opus 4.7, GPT-5.5, and Claude Sonnet 4.6) reaches only 58.8% on the rubric score and 41.5–44.3% on final-answer accuracy.
Rubric vs. Final Answer: Rubric scores and final-answer accuracy are strongly aligned (Pearson $r = 0.94$ ) but systematically separated. The mean gap is 15.95 percentage points, indicating that rubric grading credits intermediate workflow progress that final-answer grading discards.
Workflow Specialization: The top three closed models are tightly clustered on overall rubric scores but exhibit orthogonal strengths. No single model leads on more than 37.7% of questions; each leads on roughly a third of the benchmark. A simple router using observable features (workflow, source) improves performance by 7.6% relative rubric gain over the best single model.
Failure Localization: Analysis of "clean setup" (perfect Retrieval and Definition credit) reveals that most between-model separation occurs before the calculation stage. Once a model reaches a clean setup, residual calculation differences are small compared to question-to-question variation. This suggests failures accumulate in retrieval and setup rather than arithmetic alone.
Source Sensitivity: Performance varies by source document type; models consistently score 6–9 percentage points lower on 8-K grounded questions compared to 10-K questions, likely due to the less standardized nature of 8-K events.

Significance and Claims

The paper claims that BIGFINANCEBENCH shifts the evaluation paradigm for financial agents from static knowledge tests to economically valuable professional work. By grading the whole derivation rather than just the final output, the benchmark:

Exposes partial progress hidden by final-answer accuracy.
Localizes whether a derivation broke at source selection, metric definition, accounting adjustment, or calculation.
Demonstrates that final-answer accuracy is a useful but "lossy proxy" for workflow correctness.
Suggests that the frontier of financial reasoning is "spiky" rather than uniform, implying that future work may benefit from model routing strategies that select specific models for specific workflow types.

The authors emphasize that the benchmark is a static approximation of analyst work (lacking proprietary databases or live client interaction) and that scores should not be read as deployment validation for financial decisions. Instead, it serves as a high-resolution instrument to measure whether systems can connect the entity, source, period, definition, assumption, and calculation required to make a financial answer reviewable.