💰 quantitative finance

FinTradeBench: A Financial Reasoning Benchmark for LLMs

The paper introduces FinTradeBench, a comprehensive financial reasoning benchmark featuring 1,400 questions that integrate company fundamentals and trading signals to evaluate and reveal the current limitations of large language models in handling complex, multi-modal financial decision-making tasks.

Original authors: Yogesh Agrawal, Aniruddha Dutta, Md Mahadi Hasan, Santu Karmaker, Aritra Dutta

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yogesh Agrawal, Aniruddha Dutta, Md Mahadi Hasan, Santu Karmaker, Aritra Dutta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to decide whether to buy a house. You have two very different types of information to look at:

The House Inspection Report (Fundamentals): This is the boring, legal paperwork. It tells you how many rooms the house has, how old the roof is, and how much the owner owes on the mortgage. It's factual, written in text, and comes from official records.
The Neighborhood Vibe (Trading Signals): This is the energy outside. Is the street crowded with people? Are the "For Sale" signs going up or down? Is the price of the house jumping up and down wildly like a rollercoaster? This is about movement, speed, and feeling.

For a long time, AI models (the "smart computers") have been great at reading the House Inspection Report. They can summarize the paperwork perfectly. But they have been terrible at understanding the Neighborhood Vibe. They get confused when you ask them, "The house looks great on paper, but why is the price dropping every day? Should I buy it?"

Enter FinTradeBench

The researchers at the University of Central Florida built a new "test" called FinTradeBench. Think of it as a giant, 1,400-question pop quiz for AI, designed specifically to see if they can handle both the paperwork and the neighborhood vibe at the same time.

Here is the simple breakdown of what they found:

1. The Test: Three Types of Questions

The quiz had three sections:

The Accountant: "What is the company's profit?" (Pure paperwork).
The Trader: "Is the stock price moving up or down?" (Pure movement).
The Hybrid: "The company is profitable, but the stock price is crashing. Is this a good time to buy?" (This requires mixing the two).

2. The "Magic Glasses" (RAG)

To help the AI answer, the researchers gave them "Magic Glasses" called RAG (Retrieval-Augmented Generation). This is like giving the AI a library card so it can look up the exact facts instead of guessing from memory.

The Good News: When the questions were about the Accountant (paperwork), the Magic Glasses worked wonders. The AI's scores jumped up by about 37%. It could finally read the fine print without making things up.
The Bad News: When the questions were about the Trader (price movements), the Magic Glasses actually hurt the AI. The scores dropped. Why? Because the AI got overwhelmed. It was staring at a wall of raw numbers (like a spreadsheet of stock prices) and didn't know how to turn those numbers into a story. It got distracted by the noise.

3. The "Smartest" AI vs. The "Fastest" AI

The researchers tested 14 different AI models.

The "Reasoners" (like DeepSeek-R1): These models are like detectives who think step-by-step. They were the best at the Hybrid questions. They could look at the profit and the price drop and say, "Ah, the market is scared, but the company is actually healthy. This is a buying opportunity."
The "Summarizers" (like LLaMA): These models are like fast readers who summarize text. When given the Magic Glasses, they got confused. They read the whole document but missed the point. They got "distracted" by the extra information and gave worse answers than if they hadn't looked at the glasses at all.

4. The Big Lesson: "Context is King, but Structure is Queen"

The most surprising discovery was about how the information was presented.

When the AI was given raw data (a messy list of stock prices), it failed.
When the AI was given pre-calculated insights (e.g., "The stock is 15% above its average"), it succeeded.

The Analogy: Imagine asking a chef to make a soup.

No-RAG: You give them a recipe book, but no ingredients. They guess.
Standard RAG: You give them the recipe book and a giant sack of raw, unpeeled, unchopped vegetables. They get overwhelmed trying to chop everything while reading the recipe.
Ideal RAG: You give them the recipe book and a bowl of perfectly chopped, pre-measured ingredients. They make a perfect soup.

Why Does This Matter?

Currently, if you ask an AI, "Should I buy Tesla stock?" it might look at the earnings report (good) and the stock chart (bad) and get confused, giving you a wrong answer.

This paper proves that:

AI needs help with math and numbers. Just giving them more text isn't enough; they need tools to calculate.
The way we feed data matters. If we want AI to be a financial advisor, we can't just dump raw data on them. We need to structure the data so they can "think" about it, not just "read" it.

In short: We built a better test to show that while AI is getting smarter at reading financial reports, it still struggles to understand the chaotic, moving world of the stock market. To fix this, we need to teach AI how to do math, not just how to read.

1. Problem Statement

Real-world financial decision-making requires synthesizing two distinct and often conflicting information sources:

Company Fundamentals: Accounting-based metrics derived from regulatory filings (e.g., SEC 10-K/10-Q), such as profitability, leverage, and valuation ratios.
Trading Signals: Market dynamics derived from historical price and volume data (OHLCV), such as momentum, volatility, and trend reversals.

Existing financial benchmarks (e.g., FinQA, TAT-QA) primarily focus on numerical reasoning over textual financial reports and tables. They rarely evaluate:

Reasoning over trading signals derived from time-series data.
The integration of heterogeneous signals (fundamentals + market dynamics).
Scenarios where market sentiment drives prices independently of current fundamentals (e.g., a stock rallying despite poor earnings).

Consequently, it remains unclear whether current Large Language Models (LLMs) can perform the complex, cross-modal reasoning required for actual financial analysis.

2. Methodology: FinTradeBench Design

The authors introduce FinTradeBench, a benchmark designed to evaluate financial reasoning across these dimensions.

A. Data Scope and Sources

Coverage: 1,400 questions grounded in NASDAQ-100 companies over a 10-year historical window (2015–2025).
Data Sources:
- Regulatory Filings: 10-K and 10-Q forms for fundamental metrics (ROA, ROE, Debt/Equity, etc.).
- Trading Data: Daily OHLCV (Open, High, Low, Close, Volume) for computing technical indicators (RSI, Moving Averages, Momentum, Drawdowns).
Signal Selection: Based on principles of interpretability, empirical relevance, and liquidity, aligning with established asset pricing literature.

B. Question Taxonomy

Questions are categorized into three reasoning types to test specific capabilities:

Fundamentals-Focused (F-type): Reasoning over accounting indicators (e.g., "Is Nvidia's profitability sustainable?").
Trading-Signal-Focused (T-type): Reasoning over market dynamics (e.g., "Did Nvidia's stock pullback in July 2025?").
Hybrid (FT-type): Joint reasoning requiring the integration of both sources (e.g., "Is the pullback a buying opportunity given the valuation?").

C. Construction Pipeline: Calibration-Then-Scaling

To ensure reliability and scalability, the authors employed a three-phase framework:

Seed Generation: 150 expert-authored seed questions (50 per category) with "golden indicators" (key metrics required for the correct answer).
Multi-Model Candidate Generation & Self-Filtering:
- Generated $N=6$ candidate responses per model using diverse prompt templates (TELeR taxonomy).
- Intra-model self-filtering: Each model selected its best response based on factual accuracy and reasoning completeness to avoid cross-model bias.
- Automated Numerical Audit: An independent LLM auditor verified numerical claims against a structured knowledge base.
Human-LLM Judge Alignment:
- Financial experts and an independent LLM judge (Claude Sonnet 4.5) evaluated the self-filtered responses.
- The LLM judge was calibrated against human ratings to achieve a Mean Absolute Error (MAE) of <10%, ensuring the automated evaluation is reliable for scaling to 1,400 questions.

D. Evaluation Setup

Models: 14 LLMs evaluated (ranging from small open-source models like LLaMA 8B to large proprietary models like Gemini 2.5 and DeepSeek-R1).
Settings:
- Zero-Shot (No-RAG): Models answer without external context.
- Retrieval-Augmented Generation (RAG): A dual-track retrieval engine fetches:
  - Track A: Unstructured text (SEC filings) using hierarchical chunking and re-ranking.
  - Track B: Structured time-series data (price chunks) retrieved via temporal queries.
Metrics: Absolute Accuracy, Retrieval Delta ( $\Delta$ ), Golden Indicator F1 (precision/recall of key metrics), Context Integration scores, and Reasoning Depth.

3. Key Contributions

FinTradeBench Benchmark: The first benchmark to explicitly evaluate the joint reasoning of company fundamentals and trading signals, bridging the gap between financial text QA and quantitative finance.
Calibration-Then-Scaling Framework: A robust pipeline combining expert seeds, multi-model generation, self-filtering, and human-aligned LLM judging to create a large-scale, high-quality dataset.
Dual-Track RAG Architecture: A novel retrieval system designed to handle the asymmetric nature of financial data (dense text vs. structured time-series).

4. Key Results and Findings

The evaluation of 14 LLMs revealed significant performance gaps and distinct behaviors regarding RAG:

RAG Benefits Fundamentals (F-type):
- Retrieval significantly improved performance on fundamentals-focused questions (e.g., +37% accuracy for R1-Distill-Qwen, +23.8% for Gemini 2.5 Flash).
- Reasoning: LLMs have strong latent representations of SEC filings (publicly available in pre-training), and RAG acts as a grounding anchor to activate this knowledge.
RAG Harms Trading Signals (T-type):
- Retrieval degraded performance on trading-signal questions for many models (e.g., -19.7% for Gemini 2.5 Flash-Lite, -16.4% for GPT-5-mini).
- Reasoning: Models struggle to parse raw numerical time-series tables retrieved from RAG. They lack the representational framework for quantitative market data and are often "distracted" by the volume of raw numbers, failing to compute intermediate indicators (like momentum) correctly.
Hybrid Reasoning (FT-type):
- Models with latent chain-of-thought (CoT) capabilities (e.g., DeepSeek-R1, R1-Distill-Qwen) significantly outperformed standard instruction-tuned models.
- RAG provided massive gains for hybrid questions (up to +55.1% for R1-Distill-Qwen), as these models could better reconcile conflicting signals (e.g., weak earnings vs. strong price momentum).
The "Distraction Effect":
- While RAG improved surface-level factual grounding (Fundamental Integration +13.4%), it caused a collapse in Golden Indicator F1 (-56.5%) and Reasoning Depth (-10.8%).
- Models tended to summarize retrieved text fluently but failed to isolate the specific, expert-prioritized metrics required for a correct conclusion.
Architecture Sensitivity:
- Qwen/DeepSeek families benefited from RAG.
- LLaMA families (even 70B parameters) showed systematic degradation with RAG, suggesting their architectures are more susceptible to distraction from dense, jargon-heavy financial text.

5. Significance and Future Directions

Benchmarking Standard: FinTradeBench provides a necessary industry-standard dataset for evaluating financial intelligence, moving beyond simple text extraction to complex, multi-modal reasoning.
RAG Limitations: The study highlights that current RAG systems are not a panacea. While effective for textual retrieval, they introduce noise and cognitive overload for quantitative time-series reasoning. Future RAG architectures need to better integrate structured data computation (e.g., code execution) rather than just retrieval.
Research Motivation: The findings motivate future research into:
- Improving LLMs' ability to reason over numerical time-series.
- Developing "Ideal RAG" systems that pre-compute signals rather than retrieving raw data.
- Enhancing model architectures to handle heterogeneous financial signals without distraction.

The paper concludes that while LLMs are becoming proficient at summarizing financial text, they still face fundamental challenges in the numerical and time-series reasoning required for real-world trading decisions.