🤖 machine learning

FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecasting

This paper introduces FinTSB, a comprehensive and practical benchmark for financial time series forecasting designed to overcome existing evaluation limitations by addressing data diversity gaps, standardizing assessment protocols, and incorporating real-world market constraints to enable more reliable model comparison and selection.

Original authors: Yifan Hu, Yuante Li, Peiyuan Liu, Yuxia Zhu, Naiqi Li, Tao Dai, Shu-tao Xia, Dawei Cheng, Changjun Jiang

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yifan Hu, Yuante Li, Peiyuan Liu, Yuxia Zhu, Naiqi Li, Tao Dai, Shu-tao Xia, Dawei Cheng, Changjun Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to be a successful stock trader. You want to see if the robot can predict which stocks will go up or down tomorrow.

For a long time, researchers have been building different "brains" (algorithms) for these robots, ranging from simple math formulas to complex Artificial Intelligence. However, there was a big problem: nobody agreed on how to test them.

It was like having a bunch of different cars (the AI models) and testing them on different tracks (datasets) with different rules. One team tested their car on a smooth highway, another on a bumpy dirt road, and they all claimed their car was the "fastest." Because the tests were different, you couldn't really tell which car was actually the best.

This paper introduces FinTSB, a new, super-fair "Driving School" and "Test Track" specifically for financial time series forecasting. Here is how it works, using simple analogies:

1. The Problem: The "Three Leaks" in the Old Tests

The authors say old testing methods had three major holes (leaks) that let bad results slip through:

The Diversity Gap (The "One-Weather" Problem): Old tests often only used data from a few years or just one type of market (like a calm, sunny day). But real markets have storms, hurricanes, and sudden blizzards. If you only train your robot on sunny days, it will crash when a storm hits.
The Standardization Deficit (The "Different Rulers" Problem): Everyone used different measuring tools. One team measured speed in miles per hour, another in kilometers, and another by how much gas they saved. You couldn't compare the results fairly.
The Real-World Mismatch (The "Video Game" Problem): Many tests ignored real-life rules. They forgot about transaction fees (like gas money), trading limits (like speed bumps), or the fact that you can't always sell a stock instantly. It was like playing a video game where you never run out of fuel or hit a wall, making the robot look perfect when it would actually fail in the real world.

2. The Solution: The FinTSB "Super-Track"

To fix this, the authors built FinTSB, a comprehensive benchmark. Think of it as a massive, multi-purpose driving course.

The Four "Weather" Zones: Instead of just one track, FinTSB divides the market into four distinct "weather patterns" based on how stocks move:
1. Uptrends: The market is going up (a smooth highway).
2. Downtrends: The market is going down (a steep downhill slope).
3. Volatility: The market is jumping up and down wildly (a bumpy dirt road).
4. Black Swan Events: Sudden, extreme crashes or spikes (a tornado).
  The researchers took 15 years of real stock data, chopped it up, and sorted it into these four categories to ensure the robots get tested on everything, not just the easy stuff.
The Unified "Rulebook": FinTSB forces every robot to play by the exact same rules.
- The Pipeline: They built a "lightweight" system (like a standardized assembly line) where you just plug in your robot, and it runs the same tests automatically. No more fiddling with different settings.
- The Metrics: They measure performance in three ways:
  1. Ranking: Did the robot correctly guess which stocks were the best?
  2. Portfolio: If you actually bought the stocks the robot picked, would you make money? (This includes calculating fees and risks).
  3. Error: How close was the robot's guess to the actual number?
The "Real-World" Simulation: The system adds real-world friction. It charges a small fee for every trade (transaction costs) and respects rules like "limit-ups" (where a stock can't go up more than a certain amount in a day). This ensures the robot isn't just good at math, but good at trading.

3. The Results: Who Won the Race?

The authors tested dozens of different "brains" (from simple math models to huge AI models) on this new track.

No Single Winner: Just like in sports, there wasn't one robot that won every single event. Some were great at calm markets, others at volatile ones.
The AI Surprise: They found that the biggest, most complex AI models (Large Language Models) didn't automatically win. In fact, sometimes they got worse as they got bigger, until they reached a certain "critical size" where they suddenly got much better. It seems these massive models need a lot of "brain power" to untangle the messy noise of the stock market.
The "Zero-Shot" Magic: The most exciting finding was that robots trained on this diverse FinTSB track could be dropped into a completely new market (the 2024 Chinese stock market) without any extra training, and they still performed very well. This proves the track taught them how to drive in any weather, not just the specific weather they practiced in.

Summary

FinTSB is a new, fair, and realistic testing ground for stock-picking AI. It stops researchers from cherry-picking easy data, forces them to use the same measuring sticks, and makes sure the robots account for real-world costs like fees and trading limits. It's the first time the financial AI community has a standard "Olympics" to truly see which models are ready for the real world.

Technical Summary: FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecasting

1. Problem Statement

Financial Time Series Forecasting (FinTSF) is a critical domain in quantitative finance, yet the field suffers from three systemic limitations that hinder the reliable evaluation and practical deployment of forecasting models:

Diversity Gap: Existing evaluation datasets often fail to capture the full spectrum of stock movement patterns (e.g., uptrends, downtrends, volatility, and black swan events) or cover insufficient temporal ranges. This leads to models that lack generalization capabilities when faced with unseen market regimes. Furthermore, many studies focus on single markets, ignoring distinct structural differences between markets (e.g., the high retail participation in Chinese A-shares vs. the institutional mix in the US).
Standardization Deficit: There is a lack of unified assessment protocols. Discrepancies in evaluation metrics (ranking vs. portfolio vs. error) and the absence of a standardized, user-friendly pipeline make cross-study performance comparisons invalid. Re-running State-of-the-Art (SOTA) models on proprietary settings is often labor-intensive and prone to bias.
Real-World Mismatch: Many evaluations ignore critical market constraints, such as transaction fees, short-selling restrictions (e.g., in Chinese A-shares), and trading halts (limit-ups). This results in inflated performance metrics that do not translate to actual profitability in live trading environments.

2. Methodology: The FinTSB Framework

To address these limitations, the authors propose FinTSB, a comprehensive and practical benchmark designed to standardize evaluation and align with real-world trading conditions.

2.1 Dataset Construction and Tokenization

Data Source: The benchmark utilizes 15 years of historical stock data from multiple financial markets.
Pattern Categorization: Data is segmented into non-overlapping 250-day periods. Based on daily change rates ( $\delta$ $δ$ ), stocks are categorized into four distinct movement patterns:
1. Uptrends: High frequency of positive returns.
2. Downtrends: High frequency of negative returns.
3. Volatility: Frequent fluctuations without a clear directional trend.
4. Black Swan Events (Extreme): Significant, sharp price fluctuations.
Selection Strategy: For each pattern, the top 300 stocks (by positive change rate for uptrends, bottom 300 for downtrends, and remaining for volatility) are selected. Extreme outliers are classified separately.
Preprocessing: Sensitive information is tokenized (anonymized). Normalization is performed at the stock dimension for each trading day to prevent future information leakage.
Structure: The final benchmark consists of 20 datasets (5 per pattern), each containing 300 stocks over 250 days, split into training/validation/test sets (7:1:2).

2.2 Sequence Characteristics Analysis

FinTSB evaluates data quality and intrinsic properties using four sequence characteristics:

Movement Patterns: Defined by the frequency of positive/negative returns and fluctuation magnitude.
Non-Stationarity: Measured via the Augmented Dickey-Fuller (ADF) test to confirm the presence of unit roots, typical in financial data.
Autocorrelation: Measures the influence of past prices on future behavior.
Forecastability: Assessed using frequency domain properties (entropy) to determine the theoretical predictability of a series.

2.3 Unified Evaluation Pipeline

FinTSB introduces a lightweight, user-friendly pipeline comprising four layers:

Data Layer: Handles tokenization, normalization, and dynamic dataloading for specific movement patterns.
Training Layer: Integrates models from six backbone architectures:
- Classic Strategies (e.g., CSM, BLSW)
- Machine Learning (e.g., XGBoost, LightGBM, ARIMA)
- Deep Learning (e.g., LSTM, Transformer, Mamba, GNNs)
- Reinforcement Learning (e.g., PPO, DDPG)
- Generative Models (e.g., Diffusion, VAEs)
- LLM-based Methods (e.g., TimeMoe, Chronos)
Backtesting Layer: Simulates real-world trading using Top-K-Drop strategies. This strategy retains top-performing stocks and only replaces underperformers, reducing turnover. Crucially, it incorporates transaction fees (0.1%) and respects market constraints (e.g., no short selling in specific contexts).
Feedback Layer: Archives logs and provides visualization tools for performance tracking.

2.4 Evaluation Metrics

The benchmark employs 11 metrics across three dimensions to ensure a holistic assessment:

Ranking Metrics: Information Coefficient (IC), IC Information Ratio (ICIR), RankIC, and RankICIR. These measure the alignment between predicted scores and actual returns.
Portfolio-Based Metrics: Annualized Return Ratio (ARR), Annualized Volatility (AVol), Maximum Drawdown (MDD), Annualized Sharpe Ratio (ASR), and Information Ratio (IR). These evaluate profitability and risk-adjusted returns.
Error Metrics: Mean Squared Error (MSE) and Mean Absolute Error (MAE). The authors note that lower error does not guarantee profitability but is included for completeness.

3. Key Contributions

The paper outlines four primary contributions:

Diversity Inclusion: The creation of FinTSB, which tokenizes and pre-processes historical data to capture all major movement patterns across diverse markets, addressing the diversity gap.
Standardization Consistency: The proposal of a unified evaluation framework covering ranking, portfolio, and error metrics, enabling fair and reproducible comparisons.
Real-World Alignment: The design of investment strategies that strictly adhere to market constraints (fees, trading halts, short-selling rules), bridging the gap between theoretical metrics and practical applicability.
In-depth Evaluation: A comprehensive evaluation of a wide range of FinTSF methods, providing critical insights into model performance under varying market conditions.

4. Experimental Results

Extensive experiments were conducted on FinTSB using the unified pipeline:

Performance Heterogeneity: No single method achieved the best performance across all three metric dimensions. Significant heterogeneity exists even within the same backbone category, suggesting that capturing temporal dependencies and cross-sectional relationships is highly method-dependent.
LLM Performance: LLM-based methods exhibited a "paradoxical" scaling pattern. Performance initially deteriorated with model scaling but showed marked improvement at larger scales (e.g., Chronos-boltBase). This suggests that financial time series capabilities may only emerge beyond critical parameter thresholds required to disentangle complex market noise.
Deep Learning vs. Traditional: Contrary to expectations, modern deep learning techniques did not universally outperform traditional quantitative strategies or tree-based models (e.g., XGBoost, LightGBM) in terms of portfolio returns.
Transfer Learning: Models pre-trained on FinTSB demonstrated remarkable consistency when zero-shot transferred to the 2024 CSI 300 market, achieving superior risk-adjusted returns. This validates the benchmark's ability to capture robust, generalizable patterns across different market regimes (bull, bear, and transitional).
Efficiency: The study included an analysis of inference efficiency, balancing ASR performance with computational cost and memory footprint.

5. Significance and Claims

The authors claim that FinTSB provides a novel and comprehensive platform for improving and evaluating FinTSF methods. By addressing the Diversity Gap, Standardization Deficit, and Real-World Mismatch, the benchmark aims to:

Establish a solid foundation for model assessment in the FinTSF community.
Guide researchers in selecting appropriate models under varying market conditions.
Advance the field toward more practical, real-world applicable investment strategies.

The paper concludes that rigorous benchmarking is essential to move beyond fragmented results and that FinTSB serves as a necessary step toward aligning theoretical forecasting models with the complexities of actual financial markets. The code and dataset are made publicly available to facilitate further research.