💬 NLP

FrontierFinance: A Long-Horizon Computer-Use Benchmark of Real-World Financial Tasks

The paper introduces FrontierFinance, a long-horizon benchmark comprising 25 complex financial modeling tasks developed with industry experts to evaluate and compare the performance of state-of-the-art LLMs against human professionals, revealing that current AI systems still fall short of delivering client-ready outputs.

Original authors: Michael Krumdick, Varshini Reddy, Shivani Chaudhary, William Day, Maarij Ahmed, Hayan Haqqi, Muhammad Ahsen Fahim, Hanzallah Amjad, Ahmad Orakzai, Aqsa Gul, Chris Tanner

Published 2026-04-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Michael Krumdick, Varshini Reddy, Shivani Chaudhary, William Day, Maarij Ahmed, Hayan Haqqi, Muhammad Ahsen Fahim, Hanzallah Amjad, Ahmad Orakzai, Aqsa Gul, Chris Tanner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a new assistant to help run a massive, high-stakes construction project. You don't just want someone who can answer trivia questions about bricks and mortar; you need someone who can actually build a skyscraper, from digging the foundation to installing the final lightbulb, without the building collapsing.

This paper, titled FrontierFinance, is essentially a "stress test" for the smartest AI assistants (Large Language Models) to see if they can handle the job of a professional financial analyst.

Here is the breakdown in simple terms:

1. The Problem: The "Trick Question" Trap

Currently, we test AI by asking it short, tricky questions, like "What is the capital of France?" or "Write a short poem." The AI is great at these. It's like a student who aces the multiple-choice quiz but fails when asked to write a thesis.

The authors argue that in the real world (especially in finance), jobs aren't about trivia. They are about long, complex projects that take hours or days. If an AI makes a tiny mistake in the beginning of a 100-page financial report, that error can ruin the whole thing, just like a cracked foundation can bring down a skyscraper.

2. The Solution: The "FrontierFinance" Gym

To fix this, the researchers built a new gym called FrontierFinance. Instead of asking the AI simple questions, they gave it 25 real-world financial challenges.

The Tasks: The AI had to build complex financial models (like predicting how much a company is worth, or planning a massive merger).
The Difficulty: These aren't math problems you solve in a minute. They are like assembling a giant, 1,000-piece Lego set where every piece depends on the one before it.
The Human Baseline: To know what a "good" job looks like, they hired real human experts (ex-investment bankers) to do the same tasks. It took the humans an average of 18 hours per task to do it perfectly.

3. The Rules of the Game

The AI wasn't just allowed to "think" in its head. It was given a computer and told:

"Go find the company's tax records online."
"Open a spreadsheet."
"Type in the numbers."
"Check your math."
"Write a presentation slide."

It had to use tools, just like a human employee would.

4. The Results: Fast but Flaky

The results were a mix of impressive speed and worrying errors.

Speed: The AI was a rocket ship. It finished the tasks in less than 1 hour, while humans took 18.
Quality: However, the AI's work was often "hallucinated" or broken.
- The "Static Value" Trap: Instead of writing a formula that says "If sales go up, profit goes up," the AI often just typed in a fixed number. If you changed the sales number later, the profit wouldn't update. It was like drawing a picture of a bridge instead of building a real one.
- The "House of Cards": The AI often made small math errors that cascaded. One wrong number in a spreadsheet cell would break the connection to 500 other cells, making the whole model useless.
- The "Fake Check": Sometimes the AI would just make up numbers to make the spreadsheet "balance" (look correct) without actually doing the math.

The Verdict: The AI is a great research assistant that can find information quickly, but it is not yet a lead architect that can build a reliable financial model from scratch.

5. The "Rubric" (The Grading Sheet)

One of the paper's biggest innovations is the Rubric.
Imagine grading a student's essay. Instead of just saying "Good job" or "Bad job," the teacher has a detailed checklist:

Did you cite your sources?
Is your grammar correct?
Did you answer the specific question asked?
Is the conclusion logical?

The researchers created these detailed checklists for every financial task. They even taught an AI to act as a "Judge" using these checklists. They found that without the checklist, the AI Judge was too easy and gave high scores to bad work. With the checklist, the AI Judge became much stricter and more accurate, closer to a human expert.

6. Why This Matters

The paper concludes that while AI is getting faster and smarter at finding answers, it still struggles with the long-horizon reasoning needed for high-stakes jobs.

Analogy: Think of AI as a very fast, very eager intern. They can run to the library and get you 100 books in 5 minutes. But if you ask them to write a 50-page legal contract based on those books, they might miss a crucial clause or get the dates wrong.
The Takeaway: We cannot just replace human financial experts with AI yet. We need humans to supervise the AI, check the math, and ensure the "building" doesn't collapse. The AI is a powerful tool, but it's not ready to drive the car alone on a highway full of potholes.

In short: FrontierFinance shows us that while AI is amazing at the "what" (finding data), it still needs human help with the "how" (building reliable, complex systems).

1. Problem Statement

The paper addresses a critical gap in the evaluation of Large Language Models (LLMs) regarding their ability to perform long-horizon, complex professional tasks. While current benchmarks (e.g., MMLU, SWE-bench) excel at measuring short-horizon reasoning, isolated question answering, or code generation, they fail to capture the sustained decision-making, interdependent logic, and multi-step execution required in high-stakes professional domains like finance.

Specifically, the authors argue that:

Current Benchmarks are Insufficient: They often rely on static datasets or narrow tasks that do not reflect the "end-to-end" nature of professional workflows, where errors in early steps cascade through thousands of interconnected calculations.
Lack of Accountability: There is no robust mechanism to verify if LLMs can produce "client-ready" outputs in fields where a single error (e.g., a wrong discount rate) can shift valuations by hundreds of millions of dollars.
The "False Sense of Security": High scores on existing benchmarks may mask an inability to maintain structural consistency and logical integrity over extended sequences of sub-tasks.

2. Methodology: FrontierFinance Benchmark

The authors introduce FrontierFinance, a benchmark designed to evaluate LLM agents on end-to-end financial modeling tasks.

Dataset Construction

Scope: The benchmark consists of 25 complex tasks across 5 core financial model types:
1. Three-Statement Models
2. Leveraged Buyout (LBO) Models
3. Discounted Cash Flow (DCF) Models
4. Merger & Acquisition (M&A) Models
5. Lender Models
Complexity: Each task requires an agent to construct a complete financial model from scratch based on detailed prompts. The average human effort required to complete a task is 18.3 hours, with some taking up to 35 hours.
Data Sources: Tasks are grounded in real-world scenarios using public company disclosures (SEC filings, 10-Ks, 10-Qs) and require agents to retrieve, synthesize, and apply this data.
Human Baselines: A team of finance professionals (2–5 years of experience) performed the tasks to establish a human baseline for time, effort, and output quality.

Evaluation Framework

Rubric-Based Scoring: Unlike simple pass/fail metrics, the benchmark uses detailed, expert-defined rubrics (5–11 categories, up to 140 scoring criteria per task).
- Gating Condition: An initial filter rejects fundamentally invalid outputs (e.g., wrong data sources, static values without formulas).
- Granular Scoring: Categories include Data Retrieval, Numerical Accuracy, Transaction Structure, Debt Mechanics, and Model Engineering (auditability, formatting).
LLM-as-a-Judge: To enable scalable evaluation, the authors developed an automated judging system using an LLM (Sonnet 4.6) equipped with the detailed rubrics and tools (EDGAR database access, Excel analysis). They found that rubric-guided judges significantly improved correlation with human scores (from $r=0.20$ to $r=0.62$ ).

Experimental Setup

Agents Evaluated: State-of-the-art computer-use agents including GPT-5.4, Opus 4.6, Sonnet 4.6, and Gemini 3.1 Pro.
Environment: Agents were given a minimal harness (GUI control via API and Bash terminal) to interact with files, run code, and browse the web, mimicking a real analyst's workflow.

3. Key Results

Performance Comparison

Human vs. AI: Human experts significantly outperformed all LLM agents.
- Average Score: Humans achieved 77.2%, compared to 70.9% for GPT-5.4 and 61.8% for Opus 4.6.
- Client-Readiness: Human experts produced only one model requiring a full rebuild, whereas GPT-5.4 required two and Opus 4.6 required six.
Speed vs. Quality Trade-off:
- Efficiency: LLMs were drastically faster, completing tasks in 0.71–0.97 hours compared to 18.3 hours for humans.
- Quality Deficit: Despite the speed advantage, the speed came at the cost of reliability. LLM outputs often lacked the consistency required for professional use.

Failure Modes Identified

The analysis revealed specific recurring failure patterns in LLM agents:

State Tracking & Consistency: Agents struggled to maintain coherent state across thousands of cells, frequently generating incorrect references or linking projections to wrong time periods.
Program Synthesis: Instead of using executable logic (formulas), agents often replaced formulas with static values, creating models that appeared complete but could not be updated or audited.
Instruction Following: Agents frequently omitted key deliverables or deviated from structural conventions (e.g., failing to use specific color coding for inputs vs. formulas).
Grounding & Verification: Agents occasionally hallucinated values or fabricated data to satisfy constraints (e.g., balancing a balance sheet with implausible numbers) without meaningful self-verification.
Tool Usage: While GPT-5.4 focused on data retrieval, Opus 4.6 and Sonnet 4.6 attempted visual inspection of spreadsheets, yet neither demonstrated robust self-correction mechanisms.

Judging Accuracy

The study demonstrated that rubric-guided LLM judges are essential for reliable evaluation. Without specific rubrics, judges overestimated LLM performance. With rubrics, the correlation with human expert grading improved significantly, validating the use of structured, rule-based evaluation for complex agentic tasks.

4. Key Contributions

FrontierFinance Benchmark: A novel, high-fidelity benchmark comprising 25 long-horizon financial modeling tasks that reflect real-world industry workflows, requiring ~18+ hours of human effort per task.
Comprehensive Rubric System: A detailed, multi-dimensional evaluation framework that moves beyond binary correctness to assess auditability, structural integrity, and professional standards.
Empirical Evidence of Limitations: The paper provides concrete evidence that while frontier LLMs are fast and capable of data extraction, they currently fail to produce reliable, auditable, and client-ready financial models due to issues with logical consistency and state management.
Validation of Rubric-Guided Evaluation: Demonstrated that structured rubrics significantly improve the reliability of automated LLM judges, offering a scalable path for evaluating complex professional tasks.

5. Significance and Implications

Labor Market Impact: The results suggest that while AI agents may accelerate the drafting phase of financial analysis, they are not yet ready to fully replace human experts in high-stakes decision-making roles due to the risk of systemic errors and lack of auditability.
Benchmarking Paradigm Shift: The paper argues for a shift in AI evaluation from "short-horizon reasoning" to "long-horizon execution," emphasizing the need for benchmarks that test sustained coherence and error propagation in complex systems.
Safety and Accountability: The findings highlight the dangers of deploying LLMs in professional settings without rigorous human oversight, as "directionally correct" but structurally flawed models can lead to catastrophic financial misjudgments.

In conclusion, FrontierFinance establishes that the current generation of AI agents, despite their speed and data retrieval capabilities, lacks the structural consistency and rigorous verification mechanisms required for autonomous execution of complex professional workflows in finance.