🤖 machine learning

fev-bench: A Realistic Benchmark for Time Series Forecasting

To address the limitations of existing evaluation standards in time series forecasting, the authors introduce fev-bench, a comprehensive benchmark of 100 tasks across seven domains supported by the fev Python library, which employs statistically rigorous aggregation to provide reliable performance comparisons and identify future research directions.

Original authors: Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen, Lorenzo Stella, Nick Erickson, Pablo Guerron, Michael Bohlke-Schneider, Yuyang Wang

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen, Lorenzo Stella, Nick Erickson, Pablo Guerron, Michael Bohlke-Schneider, Yuyang Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to decide which of ten different weather forecasters is the best. You could just ask them to predict the temperature for tomorrow and see who is right. But what if some of them are also looking at humidity, wind speed, and historical data, while others are just guessing based on the calendar? And what if you only checked their accuracy once, on a single day that happened to be unusually sunny? You might pick the wrong winner just by luck.

This is the problem the paper fev-bench is trying to solve. The authors argue that the current "scorecards" used to test time series forecasting models (AI that predicts future numbers like sales, energy use, or stock prices) are flawed. They are often too narrow, ignore important extra information (called "covariates"), and lack the statistical rigor to tell if a model is actually better or just got lucky.

Here is a breakdown of their solution using simple analogies:

1. The Problem: The "One-Track" Test

Think of existing benchmarks like a driving test that only happens on a straight, empty highway.

Missing the "Traffic": Real-world forecasting is like driving in a busy city. You need to react to traffic lights, pedestrians, and weather. In forecasting, these are covariates (extra data like "it's a holiday" or "we have a sale"). Most current tests ignore these, so the models trained on them never learn to use them.
The "Lucky Guess" Problem: Many tests report a single number (e.g., "Model A is 5% better than Model B"). But is that 5% real, or just random noise? It's like saying a coin flipper is a genius because they got heads 6 times in a row. Without checking if that result holds up over many different scenarios, you can't trust the winner.
The "Black Box" Mess: Some tests are like a locked room where you can't see how the scoring happened. If two people run the same test, they might get different results because they used different rules. This makes it hard to compare models fairly.

2. The Solution: fev-bench (The "City Driving" Simulator)

The authors built a new benchmark called fev-bench (Forecast EValuation benchmark). Think of it as a massive, realistic driving simulator with 100 different "courses" (tasks).

Diverse Terrain: Instead of just one highway, they have 100 tasks across 7 different "cities" (domains like retail, energy, health, and finance).
The "Traffic" is Included: Crucially, 46 of these tasks include covariates. It's like testing the models while they have to deal with traffic lights and rain. This reveals that many current "smart" models are actually failing because they ignore this extra data.
Statistical Confidence: Instead of giving a single score, they use a method called bootstrapping. Imagine running the driving test 1,000 times with slightly different traffic patterns. They then calculate a "confidence interval" (a range of likely scores). If Model A beats Model B in 95% of those 1,000 runs, then you can confidently say A is better. If they are close, the test admits, "We can't tell the difference yet."

3. The Tool: fev (The "Referee's Clipboard")

To make sure everyone plays by the same rules, the authors also released a free software tool called fev.

Think of this as a standardized referee's clipboard. It handles the data loading, the scoring, and the statistical math.
It's lightweight and easy to use, meaning researchers don't have to build their own messy scoring systems from scratch. It ensures that if you and I run the same test, we get the exact same results.

4. What They Found: The "Covariate" Gap

When they ran their new tests on the top AI models available today, they found a surprising gap:

The "Smart" Models: The newest, most advanced AI models (like Chronos-2) performed the best overall.
The Missing Skill: However, the models that could use extra data (covariates) performed significantly better than those that ignored it.
The Reality Check: Many of the current "state-of-the-art" models are essentially ignoring the traffic lights and weather reports, even though those reports are right there. The paper suggests that the next big leap in forecasting won't come from making models bigger, but from teaching them how to use that extra context.

Summary

In short, fev-bench is a more realistic, fair, and statistically rigorous "Olympics" for time series forecasting. It forces models to prove they can handle real-world complexity (like extra data) rather than just memorizing patterns on a simple track. Their main discovery is that the future of accurate forecasting lies in models that know how to use all the available information, not just the past numbers.

Technical Summary: fev-bench

Problem Definition

Time series forecasting is undergoing a transformation driven by pretrained foundation models (TSFMs), which offer zero-shot inference capabilities and often outperform traditional methods. However, the evaluation of these models is hindered by significant shortcomings in existing benchmarks. Current frameworks frequently suffer from limited domain coverage, a lack of statistical rigor in aggregation, and rigid infrastructure that impedes reproducibility.

A critical gap identified is the neglect of covariates (static, past-only, and known dynamic variables), despite their prevalence and importance in real-world applications such as retail demand forecasting. Furthermore, many benchmarks report single-number summaries without confidence intervals, making it difficult to distinguish genuine model improvements from random variation. Finally, the lack of standardized, lightweight evaluation infrastructure often leads to inconsistent implementations and incomparable results across studies.

Methodology

To address these challenges, the authors introduce fev-bench, a comprehensive benchmark comprising 100 forecasting tasks across seven real-world domains (Energy, Nature, Cloud, Mobility, Econ, Health, and Retail). The methodology is built upon three pillars:

1. Benchmark Construction

Task Diversity: The benchmark includes 96 unique datasets sourced from established repositories (Monash, GIFT-Eval, BOOM) and competition data (Kaggle, M5). It features 100 distinct tasks varying in frequency (from minutely to yearly), forecast horizons, and dimensionality.
Covariate Integration: A defining feature is the inclusion of 46 tasks with covariates, covering static, past-only dynamic, and known dynamic variables. This contrasts sharply with prior benchmarks that largely ignore covariates.
Evaluation Protocol: Tasks utilize a rolling-origin protocol with multiple evaluation windows ( $W$ ) to mimic real-world deployment and improve statistical robustness. The number of windows is adjusted based on dataset size (up to 20 for small datasets, 1 for large ones).

2. Evaluation Metrics and Aggregation

The benchmark evaluates both point and probabilistic forecasting using:

Metrics: Mean Absolute Scaled Error (MASE) for point forecasts and Scaled Quantile Loss (SQL) for probabilistic forecasts. SQL is chosen for its scale-independence, analogous to MASE.
Aggregation Strategies:
- Win Rates ( $W_j$ ): The probability that a model outperforms a randomly chosen competitor on a random task.
- Skill Scores ( $S_j$ ): A geometric mean-based metric quantifying error reduction relative to a fixed baseline (Seasonal Naive), clipped to handle extreme values.
- Statistical Significance: To address the reliability of performance differences, the authors employ paired bootstrapping (1,000 samples) to compute 95% confidence intervals for pairwise win rates and skill scores. This determines if performance gaps are robust to variations in benchmark composition.

3. The `fev` Library

Supporting the benchmark is fev, a lightweight Python library designed for reproducibility and integration.

Design: It minimizes dependencies (relying only on Hugging Face datasets and pydantic) and avoids hard-coding specific library versions (e.g., PyTorch, NumPy) to prevent obsolescence.
Functionality: It provides standardized task definitions, data loading, prediction scoring, and result aggregation. It includes adapters for popular forecasting libraries (GluonTS, darts, AutoGluon, etc.) but excludes model implementations to ensure the evaluation infrastructure remains stable over time.

Key Results

The authors evaluated a suite of 21 models, including state-of-the-art pretrained TSFMs (Chronos-2, TiRex, TimesFM-2.5, Moirai-2.0, etc.), statistical baselines, and task-specific models.

Model Performance: Chronos-2 achieved the highest average win rate (91.4% for probabilistic, 87.2% for point) and skill score, followed by TiRex and TimesFM-2.5. Pretrained models generally outperformed statistical and task-specific models in both accuracy and runtime.
Statistical Significance: Pairwise comparisons with bootstrapped confidence intervals revealed a statistically significant gap between Chronos-2 and other models. However, the gap between TiRex and TimesFM-2.5 showed heavy overlap, suggesting no clear winner between them under different benchmark compositions.
Impact of Covariates: A critical finding is that models capable of utilizing covariates (specifically Chronos-2 and TabPFN-TS) showed substantial performance gains when covariates were included (e.g., Chronos-2 skill score increased from 40.9% to 47.0%). Conversely, most existing pretrained models ignore covariates, leaving significant accuracy gains untapped.
Multivariate vs. Univariate: Native multivariate modeling provided only modest gains compared to univariate approaches, contrasting with the large improvements seen from covariate integration.

Significance and Claims

The paper positions fev-bench and the fev library as a practical foundation for the next generation of time series forecasting research. Its significance lies in:

Addressing the Covariate Gap: By explicitly including 46 tasks with covariates, the benchmark highlights a major deficiency in current TSFM development, pointing toward covariate support as a clear direction for future work.
Statistical Rigor: The use of bootstrapped confidence intervals moves the field beyond single-number rankings, providing a more reliable assessment of whether performance differences are genuine or due to random variation.
Reproducibility and Flexibility: The lightweight, modular design of the fev package aims to overcome the rigidity of monolithic benchmark systems, facilitating easier integration into existing workflows and the creation of domain-specific extensions.

The authors conclude that while pretrained models are advancing rapidly, the community must prioritize covariate support and statistically robust evaluation to ensure sustained progress in real-world forecasting applications.