FinRetrieval: A Benchmark for Financial Data Retrieval by AI Agents

This paper introduces FinRetrieval, a benchmark evaluating AI agents' ability to retrieve specific financial data from structured databases, revealing that tool availability (particularly structured APIs) is the primary driver of performance while highlighting nuanced impacts of reasoning modes and geographic naming conventions.

The Gist

Imagine you are hiring a team of super-smart research assistants to find specific numbers in a massive library of financial reports. You ask them, "How much money did Apple make in the third quarter of 2024?"

Some assistants are brilliant but have to search through the library's public catalog (the internet) to find the answer. Others have a special, high-speed elevator (a structured database) that takes them straight to the exact shelf and page where the number is written.

This paper, FinRetrieval, is a report card on how well these AI "assistants" perform this specific job. The authors (from a company called Daloopa) created a test with 500 questions about real company finances and watched how 14 different AI setups tried to answer them.

Here is the breakdown of what they found, using simple analogies:

1. The "Tool" is More Important Than the "Brain"

The Finding: The biggest difference in performance wasn't which AI model was used (like Claude vs. Google vs. OpenAI); it was whether they had access to the structured database.

• The Analogy: Imagine two detectives. Detective A is a genius but only has a magnifying glass and a dusty old newspaper archive (Web Search). Detective B is slightly less of a genius but has a direct phone line to the police database (Structured API).
• The Result: When Detective A (Claude) had to use the newspaper, they got the answer right only 20% of the time. They would find a clue, get confused, keep searching, and eventually give up. But when Detective B (Claude) got the direct phone line, they got it right 91% of the time.
• The Lesson: Giving an AI a direct connection to a clean database is 3 to 4 times more important than picking the "smartest" AI model.

2. "Thinking Harder" Doesn't Always Help

The Finding: The paper tested "Reasoning Mode" (where the AI thinks for a long time before answering). Surprisingly, the AI that was already good at using tools didn't get much better with extra thinking time. The AI that was bad at using tools in the first place improved the most when told to "think harder."

• The Analogy: Think of it like a student taking a math test.
  • Student A (OpenAI): In the regular test, they forget to look up the formula in the textbook (the tool). When you tell them to "think harder," they finally look up the formula and get a huge grade boost.
  • Student B (Claude): In the regular test, they already know exactly where the formula is and look it up instantly. Telling them to "think harder" just makes them over-analyze a simple step, giving them a tiny grade boost.
• The Lesson: If an AI is already good at using its tools, making it "reason" longer is a waste of time and money. If it's bad at using tools, "reasoning" just helps it figure out how to use the tools better.

3. The First Guess Matters Most

The Finding: When the AI gets the answer right on its very first try, it's usually fast and accurate. When it misses the first time, it tends to get stuck in a loop, asking the same questions over and over, and its accuracy drops.

• The Analogy: It's like playing a game of "Hot or Cold."
  • Hot: You guess the right location immediately. You win in 3 moves.
  • Cold: You guess wrong. Now you start running around the whole house looking for the treasure, getting tired and confused. You might eventually find it, but you're much more likely to get it wrong or give up.
• The Lesson: The key to efficiency isn't how many times the AI searches; it's whether it asks the right question the first time.

4. Geography is a "Naming" Problem, Not a "Brain" Problem

The Finding: The AI was better at answering questions about US companies than non-US companies. But it wasn't because the AI didn't know about Japan or Brazil. It was because of how they name their years.

• The Analogy: Imagine a calendar.
  • US: Everyone starts the year on January 1st.
  • Japan/India: Some companies start their "fiscal year" in April or October.
  • The Confusion: If you ask for "2023," a US company means Jan-Dec 2023. A Japanese company might mean April 2022–March 2023. The AI got confused by the labels, not the math.
• The Lesson: The AI isn't biased; it just needs better instructions on how different countries label their time periods.

Summary: What Should We Do?

The paper concludes that if you want to build a financial AI that works:

1. Don't just buy the smartest brain. Buy the best database connection (the elevator).
2. Don't waste money on "thinking" features if the AI is already good at using its tools.
3. Fix the labels. Make sure the database clearly explains how different countries count their years and quarters.

In short: Tools > Brains. A smart assistant with a map will always beat a genius with a compass.

Technical Summary

Here is a detailed technical summary of the paper "FinRetrieval: A Benchmark for Financial Data Retrieval by AI Agents."

1. Problem Statement

While AI agents are increasingly deployed for financial research, existing benchmarks fail to evaluate their ability to retrieve specific numeric values from structured financial databases.

• Limitations of Current Benchmarks: Existing datasets (e.g., FinQA, TAT-QA, FinanceBench) focus on numerical reasoning over provided documents or open-book QA using web search. They do not test agents' ability to interact with structured APIs (Model Context Protocol) to extract precise data points from large-scale financial databases.
• The Gap: There is no standardized evaluation for how well AI agents can navigate structured financial schemas, handle fiscal period conventions, and utilize tools to retrieve ground-truth data without hallucination.

2. Methodology: The FinRetrieval Benchmark

The authors introduce FinRetrieval, a comprehensive benchmark designed to evaluate AI agents on financial data retrieval tasks.

Dataset Construction

• Source: Data is drawn from Daloopa, a financial platform aggregating structured data from SEC filings, earnings releases, and investor presentations for over 4,000 public companies.
• Scale: 500 unique questions with verified ground-truth answers.
• Stratification:
  • Geography: 54.2% US, 45.8% Non-US (covering 41 countries).
  • Categories: Six financial domains (Balance Sheet, Cash Flow, Operational KPIs, Income Statement, Guidance/Outlook, Segments/Geography).
  • Periods: 59% Quarterly, 33% Annual, 8% Half-year.
• Generation Pipeline: A three-phase process involving stratified sampling, context enrichment (hierarchical data + source document screenshots), and multimodal LLM generation (GPT-5.1) to create natural language questions grounded in visual evidence.
• Quality Control: Exclusion of auto-generated entries and computed values to ensure every answer traces to a specific cell in a source document.

Experimental Setup

• Providers: Three frontier providers: Anthropic (Claude Opus 4.5, Sonnet 4.5), OpenAI (GPT-5.2), and Google (Gemini 3 Pro).
• Configurations: 14 total configurations varying by:
  • Tool Access: Web Search only vs. MCP (Model Context Protocol) access to structured financial APIs (discover_companies, discover_company_series, get_company_fundamentals).
  • Reasoning Mode: Base mode vs. Extended Reasoning (e.g., "Deep Thinking" or "High Effort").
• Evaluation:
  • Scoring: An LLM judge (GPT-5.2) performs binary correctness scoring with strict normalization for units (e.g., millions vs. thousands), signs (parentheses vs. negative), and currency.
  • Traces: Complete execution traces (every tool call, input, and output) are released for behavioral analysis.

3. Key Contributions

1. First Structured API Benchmark: Unlike prior work focusing on document retrieval, FinRetrieval evaluates agents interacting with structured financial databases via MCP.
2. Multi-Provider Comparison: It provides the first controlled comparison of Anthropic, OpenAI, and Google agents on identical financial retrieval tasks.
3. Open Execution Traces: The release of 7,000+ evaluation traces enables deep analysis of agent decision-making, tool utilization, and failure modes, which was previously unavailable.
4. Empirical Findings on Tool vs. Model: The study challenges the assumption that model capability is the primary driver of performance, highlighting the dominance of tool integration.

4. Key Results & Findings

Finding A: Tool Availability Dominates Performance

• Impact: Access to structured data APIs (MCP) is the single most critical factor.
• Data: Claude Opus achieves 90.8% accuracy with MCP tools but drops to 19.8% with Web Search only—a 71 percentage point (pp) gap.
• Comparison: This gap is 3–4× larger than for other providers (e.g., Google drops from 90.6% to 69.2%).
• Cause: The gap is driven by a "gave-up" rate (55% for Claude in Web mode) and infrastructure limitations (Web tools returning snippets rather than full PDF tables).

Finding B: Reasoning Benefits Vary Inversely with Base Capability

• Observation: Extended reasoning modes provide diminishing returns for models with strong base-mode tool utilization.
• Data:
  • OpenAI (GPT-5.2): Gains +9.0pp with reasoning (Base: 80.2% → Reasoning: 89.2%).
  • Claude Opus: Gains only +2.8pp (Base: 88.0% → Reasoning: 90.8%).
• Explanation: OpenAI's base mode makes fewer API exploration calls (1.03 vs. 1.78 for Claude). Reasoning compensates for this by improving series selection, whereas Claude's base mode is already highly efficient at tool use.

Finding C: First-Query Success Drives Efficiency

• Correlation: Correct answers correlate with fewer tool calls, but this is non-causal; it reflects the difficulty of the first query.
• Mechanism: If the first discover_company_series call succeeds (87% rate for Claude), agents follow an optimal 3-call path with 93% accuracy. If it fails, agents enter extended search loops, dropping accuracy to 77%.
• Bottlenecks: 55% of first-call failures are due to keyword mismatches; 45% are due to period filter failures.

Finding D: Geographic Gaps are Data Convention Issues

• Observation: US companies show higher accuracy than Non-US companies (e.g., +5.6pp for Claude).
• Root Cause: This is not a model limitation but a fiscal year naming convention mismatch.
  • US companies typically use calendar year-ends (Dec 31).
  • Non-US companies often use non-standard fiscal years (e.g., March or September).
  • Agents assume "2023FY" means the year ending in 2023, while data sources often label it by the start year (e.g., "2022FY" for April 2022–March 2023).
• Resolution: When controlling for December fiscal year-ends, Non-US accuracy slightly exceeds US accuracy.

5. Error Analysis

• Primary Failure Mode: Period Confusion accounts for 63% of errors in the best-performing configuration (Claude Opus + Reasoning). This includes quarter identity mismatches (Fiscal Q1 ≠ Calendar Q1) and year labeling conventions.
• Secondary Modes: Wrong series selection (19.6%) and field selection errors (15.2%).
• Implication: Most errors stem from undocumented tool conventions rather than model reasoning failures. Better tool documentation could resolve the majority of remaining errors.

6. Significance and Implications

• For Practitioners:
  • Tool Integration > Model Selection: Prioritizing structured data API integration yields far higher ROI than selecting a specific "smarter" model.
  • Reasoning ROI: For models with strong base tool utilization (like Claude), the latency cost of reasoning modes may not justify the marginal accuracy gain.
  • Data Conventions: Financial AI systems must explicitly handle regional fiscal year conventions to avoid systematic errors.
• For Research:
  • The benchmark shifts the focus from "reasoning over text" to "reasoning over structured data retrieval."
  • It highlights that the path to >90% accuracy lies in tool description completeness and API design (e.g., supporting multiple fiscal year formats) rather than just scaling model parameters.

Conclusion: FinRetrieval demonstrates that for financial data retrieval, the quality of the tool interface and the agent's ability to utilize structured APIs are the primary determinants of success, outweighing model architecture or reasoning enhancements. The authors release the dataset, code, and traces to facilitate further research into robust financial AI systems.