🤖 machine learning

TS-Arena -- A Live Forecast Pre-Registration Platform

TS-Arena is a live forecasting platform that addresses test-set contamination in Time Series Foundation Models by enforcing a strict pre-registration protocol where predictions are submitted before ground-truth data exists, thereby enabling robust, real-time evaluation of generalization capabilities on future data.

Original authors: Marcel Meyer, Sascha Kaltenpoth, Henrik Albers, Kevin Zalipski, Oliver Müller

Published 2026-04-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Marcel Meyer, Sascha Kaltenpoth, Henrik Albers, Kevin Zalipski, Oliver Müller

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The AI "Cheating Trap"

Imagine you are a professional footballer and you want to prove you are the best. But instead of playing against real opponents, you are given a video recording of a match from last season and asked to show how you would have shot the balls. The problem? You have already seen the video! You know exactly where the defender is standing and where the ball is flying. You are not "cheating" intentionally, but you have an unfair advantage because you already know the future of the recording.

This is exactly what is currently happening in the world of Artificial Intelligence (AI) regarding time series forecasting (e.g., electricity consumption or stock prices). Researchers train their models with massive amounts of historical data. When they then test these models with old data, the AIs often "remember" the patterns. They have essentially already seen the answer in their training materials. The result: The AI appears super smart, but in the real world, where the future is still completely unknown, it might be totally terrible.

The Solution: TS-Arena – The "Live Race into the Future"

The researchers have now developed TS-Arena. This is not a static test, but a live platform.

You can imagine it like a real live broadcast:
Think of a quiz show where the questions are asked at exactly this moment, while the participants must answer them. The answer (the "truth") does not yet exist in the physical world! Only after the participants have submitted their prediction do we look at the real data, which arrives minutes or hours later.

This principle is called the "Forecast Pre-Registration Protocol":

The prediction first: The AI must submit its estimate (e.g., "In one hour, 500 megawatts of electricity will be consumed").
Time runs: The AI must do this before the actual measurement has even taken place.
The check: Only after time has run out and the real values are established is the comparison made.

This makes cheating impossible. The AI cannot "look into the future" because the future, at the moment it guesses, simply has not been invented yet.

How does this work technically? (The "Kitchen Machine")

The researchers have built a state-of-the-art "machine" (a microservice architecture). You can imagine it like an automated professional kitchen:

The Supplier (Data Portal): Constantly fetches fresh ingredients (real-time data like wind speed or electricity prices) from the world.
The Head Chef (API Portal): Organizes the rounds, tells the AIs when they must "cook" (forecast), and strictly ensures that no one tastes the ingredients in advance.
The Testers (Reference Models): There are already built-in "standard AIs" that serve as a comparison, so you can immediately see whether a new challenger is truly better or just got lucky.

What did they find out?

They tested the whole thing for a year using electricity data from Europe. The result was exciting:

The "Superstars": Certain modern AI models (like Chronos-2 or TiRex) are indeed extremely good at recognizing patterns, even if they have never seen the data before.
The "Reality Check": With some things (like electricity generation from wind or sun), the AIs struggle more because nature is very unpredictable. Here it becomes clear: An AI that only "memorizes by rote" fails in the real, chaotic world.

Conclusion

TS-Arena is like an incorruptible referee that ensures we only crown the AIs as winners that truly understand the future, and not just those that have memorized the past best. It is the path from the "school exam with the answers on the back" to "real life."

Technical Summary: TS-Arena – A Live Forecast Pre-Registration Platform

1. The Problem: The "Evaluation Crisis" in Time Series Foundation Models (TSFMs)

As the field shifts toward Time Series Foundation Models (TSFMs)—large-scale models capable of zero-shot forecasting—traditional evaluation methods (static historical datasets and retrospective splits) have become unreliable due to two primary forms of information leakage:

Direct Information Leakage (Test-Set Contamination): Because TSFMs are trained on massive, often opaque datasets, there is a high risk that benchmark test sets are inadvertently included in the model's pre-training data, leading to artificially inflated performance.
Indirect Temporal Information Leakage: This occurs when a model is evaluated on a test series that is temporally correlated with a series used in its training data. For example, if a model trains on bus usage data and is tested on metro usage data from the same time period, it can "leak" future patterns from one to the other.

Existing benchmarks (like TSFM-Bench or GIFT-Eval) fail to solve this because they rely on historical data that has already been "captured" by the training corpora of modern foundation models.

2. Methodology: The Forecast Pre-Registration Protocol (FPRP)

The authors propose TS-Arena, a live benchmarking platform that shifts evaluation from the known past to the unknown future. The core innovation is the Forecast Pre-Registration Protocol (FPRP).

How FPRP Works:

Continuous Real-Time Evaluation: Instead of a static dataset, the platform uses live data streams (e.g., energy consumption, prices, and generation).
Strict Temporal Split ( $t_{now}$ ): The platform defines a moving split point, $t_{now}$ . Models are provided with historical context up to $t_{now}$ .
Pre-Registration Requirement: Models must submit their forecasts for a fixed future horizon ( $H$ ) before the ground-truth data for that horizon physically exists.
Window Closure: The submission window closes strictly before the next ground-truth observation ( $X_{t_{now}+1}$ ) is recorded. This makes it physically impossible for the model to access the test set, eliminating both direct and indirect leakage by design.

System Architecture:
TS-Arena utilizes a modular microservice architecture (Docker-based) consisting of:

Data Portal: Ingests and normalizes real-time data from external APIs (e.g., SMARD, Fingrid).
API Portal: Orchestrates challenge rounds, manages authentication, and enforces the registration windows.
Reference Model Service: Hosts containerized, state-of-the-art TSFMs and statistical baselines to ensure every round has a baseline for comparison.
ELO-based Leaderboard: Uses a pairwise comparison system (similar to chess) to rank models. To handle the uncertainty of new entrants, they use a Randomized Replay Bootstrap to calculate 95% confidence intervals.

3. Key Contributions

First Leakage-Free Framework: Introduces the only mechanism (FPRP) that prevents information leakage in TSFMs by evaluating on data that does not yet exist at the time of inference.
Living Benchmark Infrastructure: Moves away from "one-off" competitions toward a continuous, evolving evaluation environment.
Modular Microservice Design: Provides a scalable, reproducible architecture for hosting diverse model types (from lightweight statistical models to GPU-heavy TSFMs).
Public Archive: Provides a quarterly-updated HuggingFace archive to allow researchers to perform offline experiments while maintaining temporal integrity.

4. Results and Analysis

The authors backtested the platform using one year of energy time series data (2025). Key findings include:

TSFMs Outperform Baselines: All top positions on the ELO leaderboard were occupied by TSFMs (e.g., Chronos-2, TiRex, and Moirai-2).
Scaling Laws: Within model families (like Chronos-Bolt), ELO ratings increased monotonically with model size, confirming that parameter scaling benefits zero-shot forecasting.
Task-Specific Strengths: TSFMs performed exceptionally well on electricity consumption (highly predictable) but struggled with electricity generation (highly intermittent due to weather/physical constraints). This suggests that "out-of-the-box" TSFMs still struggle with physically constrained, non-stationary series without auxiliary covariates.
Model Evolution: Newer iterations of models (e.g., Moirai-2 vs. Moirai-base) consistently achieved higher rankings, demonstrating the platform's ability to track rapid research progress.

5. Significance

TS-Arena provides the necessary infrastructure to assess the true generalization capabilities of modern forecasting models. By enforcing a "commitment to the future," it prevents the "evaluation crisis" currently plaguing the field, ensuring that leaderboard rankings reflect actual predictive power rather than the ability to memorize or exploit temporal correlations in historical datasets.