📊 statistics

FinBench: Time-Gated Calibration and Uncertainty Benchmarking for Agentic Financial Forecasting

This paper introduces FinBench, a novel benchmark designed to evaluate the probabilistic calibration and uncertainty quality of agentic financial forecasting models by enforcing strict time-gating and utilizing proper scoring rules to penalize overconfidence and hallucinated certainty.

Original authors: Rishab Ghosh, Vinay Devarakonda

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC0 1.0

Original authors: Rishab Ghosh, Vinay Devarakonda

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a high-stakes game of "Guess the Weather" where the prize isn't just a gold star, but your entire life savings. In this game, you have a super-smart robot friend who can read millions of news articles and look at complex charts to predict if the sun will shine or if it will rain tomorrow. But here's the catch: the robot doesn't just say "It will rain." It also has to say how sure it is. If the robot says, "I'm 99% sure it will rain," and you bet your house on that, you're going to be in big trouble if it turns out to be sunny. This is the core problem of "calibration": making sure your confidence matches your actual skill. In the world of finance, where computers (called "agents") are starting to make real money decisions for us, a robot that is confidently wrong is far more dangerous than a robot that is unsure. If a robot thinks it's a genius but is actually just guessing, it will bet too big and lose everything.

This is exactly the story behind a new project called FinBench. The researchers, Rishab Ghosh and Vinay Devarakonda, are worried that the super-smart AI models we use today are great at sounding confident but terrible at knowing when they are guessing. They built a special test track to see if these AI agents can honestly admit when they don't know something, rather than just bluffing their way through financial predictions.

The Problem: The Overconfident Gambler

In the past, we tested AI by asking them simple questions like "What is the capital of France?" or "Will this stock go up or down?" and checking if the answer was right or wrong. But in the real financial world, being right 55% of the time isn't enough if you are 100% sure you're right every single time. If you are slightly better than a coin flip but act like a god, you will eventually bet everything on a bad guess and lose it all.

The authors argue that most current tests don't catch this. They don't check if the AI's "confidence meter" is broken. They also don't stop the AI from peeking at the future. Imagine if you were taking a test, but you were allowed to look at the answer key before you started writing. That's called "look-ahead bias," and it's a huge problem in finance. If an AI uses information from the end of the day to predict what happened in the morning, it's cheating.

The Solution: FinBench

To fix this, the team created FinBench, a new kind of test track designed specifically to catch "confident but fragile" AI behavior. Think of it like a driving test for self-driving cars, but instead of checking if the car can stop at a red light, they are checking if the car knows when it's too foggy to drive safely.

How the test works:

Strict Time-Travel Rules: The AI is given a specific time (like 9:30 AM) and must make a prediction. It is strictly forbidden from seeing any data that happens after that time. It's like being locked in a room with only the morning newspaper; you can't read the evening paper to help you guess the weather.
The Two-Part Answer: The AI has to give two things:
- A percentage chance that a stock will go up (e.g., "I think there is a 60% chance").
- A "safety net" range (an 80% prediction interval) where the actual price change will likely land.
The Scoring System: The test uses two special math rules to grade the AI:
- The Brier Score: This punishes the AI if it says "99% sure" and gets it wrong. It rewards the AI for being honest. If the AI says "50/50" when it's actually a coin flip, it gets a good score.
- The Winkler Score: This checks the "safety net." If the AI makes the net too wide (covering everything just in case), it gets a penalty for being a coward. If it makes the net too tight and misses the actual price, it gets a penalty for being reckless.

What They Found (The Pilot Run)

The authors ran a small "pilot" test to see if their system worked. It was like a dress rehearsal before the big show. They picked three popular stocks (Apple, Microsoft, and Nvidia) and asked six different AI models to make predictions for just one trading day.

Here is what the little test revealed:

Accuracy isn't everything: Some models got the direction right (up or down) 100% of the time in this tiny test. But when you looked at their confidence scores, they were all over the place.
The "Confident but Wrong" Trap: Two models (Llama 3.1 and DeepSeek-V3) got a specific prediction wrong. They were only slightly confident (about 55-56%), but because they were wrong, their "Brier Score" was terrible, and they actually performed worse than just flipping a coin. This proves that the test successfully caught models that were overconfident in their mistakes.
Honesty Pays Off: The models that were better at matching their confidence to their actual performance (like GPT-4o) got higher scores, even though they didn't necessarily get every single prediction right.

The researchers found that calibration (honesty about uncertainty) and interval quality (how good the safety net is) are two different skills. A model can have a safety net that catches the right price 80% of the time, but still have terrible confidence numbers. This means you can't just look at one number to see if an AI is safe; you have to look at both.

The Bottom Line

This paper doesn't claim to have found the "perfect" AI for finance yet. In fact, the authors are very careful to say this was just a tiny test with only 33 predictions. They aren't saying one AI is the ultimate winner. Instead, they are saying: "Hey, we built a new ruler that measures something important that other rulers miss."

They showed that it is possible to build a test that stops AI from cheating by looking at the future and forces them to be honest about how sure they are. This is a crucial step because, in the future, if we let AI agents manage our money, we need them to know when to say, "I don't know," rather than betting the farm on a hunch. The paper suggests that without this kind of "calibration check," we might be handing our wallets to overconfident robots that are destined to crash.

Technical Summary: FinBench

Problem Statement

Large Language Models (LLMs) are increasingly deployed as agentic systems capable of observing, planning, and acting. In financial contexts, these systems become decision-relevant when their outputs influence trade sizing or risk allocation. The primary failure mode identified is not merely directional inaccuracy, but the confidence–competence gap: a model that is only slightly better than chance but consistently overconfident will, under standard bet-sizing rules (e.g., Kelly criterion), generate negative long-run growth by over-allocating capital to incorrect views.

Existing benchmarks largely focus on semantic understanding or point accuracy. They fail to address two structural hazards specific to finance:

Non-stationarity: Market regimes change, causing memorized patterns to decay.
Look-ahead bias: Preprocessing can inadvertently leak future information (e.g., using end-of-day variables to forecast intraday outcomes), which is particularly acute in agentic settings where errors compound over time.

Crucially, current evaluations do not test whether a model's stated probabilities reflect true uncertainty under strict temporal constraints.

Methodology: FinBench Specification

The authors introduce FinBench, a benchmark designed to evaluate calibration and uncertainty quality for financial forecasting. The core design principles are:

1. Strict Time-Gating Protocol

To prevent look-ahead bias, the benchmark enforces a protocol where a model producing a forecast for time $T$ is strictly blind to any data (prices, news, derived signals) with timestamps later than $T$ .

Forecast Horizons: Defined as MORNING (09:30 ET), MIDDAY (12:30 ET), and CLOSE (15:30 ET).
Target Time: A fixed evaluation time near the end of the day (15:55 ET), distinct from the forecast horizon.
Inputs: Numerical (OHLCV bars) and textual (headlines, sentiment) data are filtered to ensure only information available at time $T$ is used.

2. Output Schema

For each ticker and horizon, models must output a structured object containing:

$\hat{p} \in [0, 1]$ : The probability that the realized log return is positive ( $Y=1$ ).
$[L, U]$ : An 80% prediction interval for the realized log return $r$ .
(Optional) A natural-language rationale, which is not used for scoring.

3. Evaluation Metrics

FinBench utilizes strictly proper scoring rules to incentivize truthful uncertainty reporting:

Brier Score (BS): Used for probabilistic direction forecasts. It measures the mean squared difference between predicted probability and the actual binary outcome.
- $BS = \frac{1}{N} \sum (\hat{p}_i - Y_i)^2$
- Brier Skill Score (BSS): Reports performance relative to baselines (Coin flip, rolling base rate, momentum heuristic). Positive BSS indicates improvement; negative indicates worse-than-baseline calibration.
Winkler Score: Used for interval estimation. It penalizes both "cowardice" (intervals that are too wide) and "recklessness" (intervals that are too tight and miss the realized return).
- $S_W = (U - L) + \frac{2}{\alpha}(L - r)I(r < L) + \frac{2}{\alpha}(r - U)I(r > U)$ , where $\alpha = 0.2$ .

Key Contributions

Benchmark Design: FinBench is the first benchmark explicitly designed to enforce strict time-gating for agentic financial forecasting while centering strictly proper probabilistic scoring.
Calibration Focus: It shifts the evaluation metric from simple accuracy to calibration, directly penalizing hallucinated confidence which is critical for risk management.
Pilot Implementation: The paper provides a specification and a small-scale pilot run to validate the pipeline mechanics.

Pilot Results

The authors conducted a pilot run on January 15, 2025, covering three liquid tickers (AAPL, MSFT, NVDA) across the three horizons, resulting in 33 total forecasts. Models tested included GPT-4o, GPT-4o Mini, Llama 3.1/3.3, Qwen2.5, and DeepSeek-V3.

Directional Accuracy vs. Calibration: On this specific day, most models achieved high directional accuracy (often 100% for the MORNING horizon). However, calibration-sensitive metrics (Brier Score and BSS) revealed significant variance.
Overconfidence Detection: Two specific cases (MSFT MORNING) involving Llama 3.1 70B and DeepSeek-V3 showed negative skill scores. These models output moderately confident probabilities (0.55 and 0.56) for incorrect outcomes ( $Y=0$ ), resulting in Brier scores > 0.30 and negative BSS values.
Interval Quality: While interval coverage hovered near the nominal 0.8–0.9 range, Winkler scores varied, illustrating the trade-off between interval width and misses.
Composite Performance: GPT-4o achieved the highest composite score, followed by GPT-4o Mini and Llama 3.3, while other models lagged in calibration metrics despite similar accuracy.

Note: The authors explicitly state these results are descriptive and not statistically conclusive due to the small sample size.

Significance and Claims

The paper claims significance in addressing the specific needs of agentic finance, where uncertainty is the primary output required for safe decision-making (position sizing, risk limits, human escalation).

Distinct Axes: The authors demonstrate that calibration and interval quality are distinct axes; a system can achieve target coverage while still reporting poorly calibrated probabilities. A single accuracy metric hides these critical risk behaviors.
Rationale-Probability Mismatch: The benchmark isolates a qualitative pattern where models generate plausible textual explanations but output probabilities that do not reflect those explanations. FinBench scores only the probabilistic components to detect this mismatch.
Safety and Robustness: By using strictly proper scoring rules, FinBench aligns model incentives with honest uncertainty reporting, a prerequisite for safety in high-stakes financial deployment.

The paper concludes modestly, noting that the pilot is too small to claim model superiority or robustness to regime shifts. Future work aims to expand the ticker universe, standardize baseline construction, and introduce stress slices (e.g., macro announcements) to study distribution shifts.