🤖 machine learning

GAUGE: Grading Agent-Built Financial Models Without a Golden Answer

The paper introduces GAUGE, a novel benchmark that evaluates AI-built financial models against observed analyst practices rather than a single "golden" answer, revealing that while current agents excel at mechanical model construction, they still significantly lag behind human professionals in valuation judgment.

Original authors: Jiacheng Lu, Sinuo Wang, Wentao Zhao, Rui Sun, Cheng Hua, Tao Song, Hui Cai, Beidi Luan, Zhengze Wu, Lingjing Teng, Yijia He, Jing Li, Daxin Jiang, Zuo Bai, Haibing Guan

Published 2026-07-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiacheng Lu, Sinuo Wang, Wentao Zhao, Rui Sun, Cheng Hua, Tao Song, Hui Cai, Beidi Luan, Zhengze Wu, Lingjing Teng, Yijia He, Jing Li, Daxin Jiang, Zuo Bai, Haibing Guan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge on a cooking show. Usually, the judges have a single "perfect" recipe card. If a contestant's dish tastes even slightly different from that card, they lose points. But what if there are actually ten different ways to make a delicious lasagna? What if the "perfect" recipe is just one person's opinion, and the contestant's version is equally tasty but uses different spices? This is the problem scientists face when testing AI on complex tasks like financial modeling. Financial modeling is like building a detailed prediction machine for a company's future; it mixes hard facts (like past sales) with educated guesses (like how fast the company will grow). For decades, we've tested AI by comparing its guesses to a single expert's answer. But if experts themselves disagree on the best guess, punishing the AI for having a different, yet reasonable, answer doesn't seem fair. This paper asks: How do we grade AI when there isn't just one "right" answer?

The researchers behind this study, titled GAUGE, decided to stop pretending there is only one perfect financial model. They built a new testing ground to see if AI agents can build these complex financial "prediction machines" and, more importantly, if they can make the judgments inside them that human experts make.

First, they tested the old way of grading. They took 108 pairs of financial models built by different human experts for the same companies. When they graded one expert's model against another's using the old "single perfect answer" rule, the results were shocking. The median score was only 0.33 out of 1.0. In fact, 92.6% of the pairs scored below 0.70. This means that even professional humans, who are all experts, often disagree so much that if you treated one as the "truth," you would fail the other. It turns out that in finance, being "different" doesn't mean being "wrong."

To fix this, the team created GAUGE (Grading Agent-Built Financial Models Without a Golden Answer). Instead of a single recipe card, they created a "safety envelope" based on what real experts actually do. Imagine a target where the bullseye isn't a single dot, but a wide ring. If an AI's guess lands anywhere inside that ring of "reasonable expert behavior," it gets credit. The system checks two things:

The Mechanics: Did the math work? Did the spreadsheet balance? (Like checking if the oven was actually turned on).
The Judgment: Did the AI's guesses fall within the range of what real experts consider defensible? (Like checking if the seasoning is within the range of a good chef's taste).

They tested 24 different AI agents on this new system, along with a group of 55 humans ranging from finance students to senior analysts. Here is what they found:

AI is getting good at the math, but bad at the gut feeling. The AI agents were excellent at the mechanical parts. The best AI passed 93% of the mechanical checks (making sure the spreadsheet formulas were correct). However, when it came to the judgment parts (making the actual predictions), the best AI only passed 78%.
The "Gap" is real. On average, the AI agents were 26 points better at building the model than at making the valuation judgments. They can build the car, but they aren't great at driving it yet.
Humans still win. The best AI scored 53.4. This was better than the average finance student (43.2) but worse than every single senior analyst (who averaged 88.3) and most junior analysts. The AI is smart enough to do the homework, but not quite smart enough to be the boss.

The study suggests that while AI is becoming very capable of constructing complex financial models, the hardest part—making the nuanced, defensible judgments that experts use to value a company—remains a significant challenge. The authors didn't just find a score; they proved that the old way of grading (looking for one perfect answer) was unfair to both humans and machines, and they offered a new way to measure progress that respects the messy reality of expert disagreement.

Technical Summary: GAUGE – Grading Agent-Built Financial Models Without a Golden Answer

1. Problem Statement

Financial modeling involves combining public disclosures with analyst assumptions to generate forecasts and valuations. While mechanical components of these models (e.g., historical data accuracy, accounting identity balancing) are objectively verifiable, valuation components (e.g., revenue forecasts, discount rates, target prices) inherently admit multiple reasonable answers.

Existing benchmarks for financial AI agents typically grade outputs against a single expert-authored reference (a "golden answer"). The authors argue this approach is flawed for valuation tasks because it penalizes agents for disagreeing with a specific reference analyst, even when the agent's assumptions are defensible and consistent with professional practice. This design conflates professional disagreement with model error, failing to account for the "rating indeterminacy" inherent in open-ended financial artifacts.

The paper empirically demonstrates this issue by grading independently built analyst models against one another. Across 108 directed pairs covering 65 companies, the median score using a single-reference point-tolerance approach is 0.33, and 92.6% of pairs score below 0.70. No same-vintage pair agrees on implied share price within 10%. This suggests that standard point-tolerance grading penalizes the natural disagreement that exists even among professional analysts.

2. Methodology: The GAUGE Benchmark

To address the limitations of single-reference grading, the authors introduce GAUGE, a benchmark designed to evaluate agent-built valuation models against observed analyst practice rather than a single point estimate.

2.1 Corpus and Task Design

Corpus: The benchmark is built from 1,001 vendor-classified analyst workbooks covering 922 tickers across 25 GICS industries.
Task Format: The evaluation set consists of 196 tasks (with a 48-task core for agent evaluation). Agents receive an "Excel-in / Model-out" task: they are provided with raw historicals, segment skeletons, and guidance, but consensus forecasts are withheld.
Deliverables: Agents must produce a formula-driven model, valuation, sensitivity analysis, assumptions file, and a memo.
Data Integrity: The corpus is de-identified, and a "withheld refresh pool" (~600 workbooks) is reserved for future longitudinal evaluation to prevent contamination.

2.2 Scoring System

GAUGE employs a multi-layered scoring stack that separates mechanical correctness from judgment:

Three-Layer Defensibility Envelope: Instead of a single point, judgment-bearing quantities are evaluated against a range derived from observed practice.
- E-method: Uses the min/max of multi-method values and sensitivity grids within a specific workbook.
- E-industry: Uses P10–P90 distributions of assumptions across the industry.
- E-company: Calibrates bands using the dispersion of multiple analysts covering the same company.
- Scoring: Values inside the tight band score 2; values in the widened "near" band (expanded by P90 disagreement) score 1; values outside score 0.
56 Facets: The system evaluates 56 specific facets categorized into:
- 29 Deterministic (D): Structural checks (e.g., formula linking, accounting identities).
- 23 LLM-Judged (J): Qualitative assessments of assumptions and logic.
- 4 Direct-Rule (R): Specific rule-based checks (e.g., WACC band membership).
Validity Gates (G1–G8): Eight deterministic gates detect structural failures (e.g., unbalanced balance sheets, circular references, look-ahead contamination). If a gate is triggered, it imposes a score cap (ceiling), preventing structurally unusable models from receiving high scores despite plausible local outputs.
Failure-Aware Score ( $\phi_0$ ): The primary metric assigns a score of 0 to non-completions or capability failures, ensuring that agents cannot rank highly by simply completing a subset of tasks while failing others.

3. Experimental Setup

Agents: 24 agents from 12 providers (including Claude, GPT-5 variants, Gemini, Kimi, Qwen, etc.) were evaluated on the 48-task core.
Human Baseline: A 55-participant known-groups study was conducted with three groups: Senior Analysts (12), Junior Analysts (18), and Finance Students (25). Participants worked under the same constraints as agents (no network access, same inputs) but without the strict time budget.
Evaluation Protocol: Agents used a standardized tool-calling harness. Each task was run once per agent to avoid best-case bias.

4. Key Results

4.1 Agent Performance vs. Humans

On the failure-aware score $\phi_0$ :

Senior Analysts: Average 88.3
Junior Analysts: Average 66.0
Finance Students: Average 43.2
Best Agent (Claude Fable 5): Scored 53.4.
- This is above the student mean but below every senior analyst and 15 of the 18 junior analysts.
- 33 of the 55 human participants outscored the best agent.

4.2 Mechanical vs. Judgment Gap

A critical finding is the disparity between an agent's ability to construct models and its ability to exercise valuation judgment:

Mechanical Facets: The best agent passed 93% of mechanical facets (structural/formulaic correctness).
Judgment Facets: The best agent passed only 78% of judgment facets.
Fleet Median Gap: Across all 24 agents, the median gap between mechanical and judgment pass rates is 26 points.
Conclusion: Current agents are substantially stronger at model construction than at valuation judgment.

4.3 Validity and Stability

Known-Groups Ordering: The benchmark successfully orders humans by experience (Senior > Junior > Student) with high statistical significance ( $p < 10^{-6}$ ).
Gate Impact: Removing validity gates reversed 11 of 276 model-pair orderings, confirming that structural checks are necessary to prevent inflated scores for broken models.
Judge Stability: Repeated voting by a single frozen judge (5 draws) yielded a Kendall's $\tau$ of 0.944, indicating high sampling stability.

5. Significance and Claims

The paper claims three primary contributions:

Empirical Examination of Single-Reference Assumption: It demonstrates that grading against a single expert reference is problematic for valuation, as professional analysts frequently disagree on key inputs (WACC, terminal growth, implied price) even when producing internally consistent models.
Introduction of GAUGE: It provides a benchmark that decouples mechanical verification from judgment evaluation, using an "observed-practice envelope" to grade defensible choices rather than identical answers.
Identification of Agent Capabilities: It establishes that while agents are becoming capable of automating the construction of financial models, the judgment required for valuation remains a significant bottleneck.

The authors emphasize that GAUGE does not claim to establish "ground truth" for valuation (as multiple defensible answers exist) but rather provides a framework for evaluating whether an agent's choices fall within the range of observed professional practice. The release includes the methodology, gated data splits, and a versioned longitudinal core to support reproducible and contamination-controlled evaluation.