🤖 AI

LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation Models

This paper introduces LiveHouse-TS, the first open-world living benchmark that evaluates Time Series Foundation Models through continuous prequential testing on real future data, revealing that static rankings often fail to reflect long-term robustness and performance under evolving distribution shifts.

Original authors: Haomin Wen, Ziyu Zhou, Qingxiang Liu, Siru Zhong, Yuxuan Liang

Published 2026-08-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haomin Wen, Ziyu Zhou, Qingxiang Liu, Siru Zhong, Yuxuan Liang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Predicting the future is a fundamental human urge, whether we are checking the weather, planning a budget, or trying to understand how a river will flow after a storm. For decades, scientists have built mathematical models to forecast these patterns, but a new generation of tools has recently emerged that promises to do this without needing to be taught the specifics of every single situation. These tools, known as foundation models, are trained on vast amounts of historical data from many different fields at once. They learn the general language of time and change, allowing them to make educated guesses about new, unseen data immediately, a capability called zero-shot forecasting. The hope has been that these powerful systems could replace the need for custom-built models for every specific task, offering a universal solution for predicting everything from stock prices to energy usage.

However, there is a significant gap between how these models are tested in the lab and how they actually perform in the real world. Traditionally, researchers evaluate these systems using static snapshots: they take a fixed chunk of past data, train a model, and then see how well it predicts a fixed chunk of future data that was set aside. This approach is like grading a student on a single practice test that never changes. While it provides a clear score, it fails to capture the messy reality of life, where conditions shift constantly, seasons change, and unexpected events occur. A model might ace a static test by memorizing patterns that no longer exist, only to fail when the world moves on. The question remains: do these models remain reliable when the data they are predicting is constantly evolving, or do they crumble when the ground rules change?

To answer this, a team of researchers has introduced a new way of testing called LiveHouse-TS. Instead of a frozen snapshot, they built a living, breathing benchmark that operates like a continuous performance review. Imagine a stage where models must make predictions in real-time, at the exact moment the data is available, before the true outcome is known. As new data arrives every day, the models are asked to forecast again, and their performance is scored immediately. This process repeats endlessly, creating a stream of evaluations that mimics the actual conditions of deployment. The system covers seventeen different datasets from eleven distinct domains, ranging from weather and ocean waves to financial markets and earthquake counts. It tests models across various time scales, from seconds to years, ensuring that the evaluation captures a wide diversity of real-world behaviors.

The researchers ran this live experiment with several of the most advanced forecasting models available today, alongside traditional statistical methods. The results revealed a dramatic reshuffling of the rankings. Models that had previously dominated the static, snapshot-based leaderboards found themselves struggling in the live environment. In the static tests, a model might appear to be the best because it perfectly matched a specific historical period. But when subjected to the continuous flow of new data, these same models often degraded quickly, unable to adapt to shifting patterns or unexpected changes. Conversely, some models that were not the top performers in static tests proved to be far more robust, maintaining consistent accuracy as the data evolved.

One of the most striking findings was that a model's ability to predict a single point in time does not guarantee it can handle the uncertainty of the future. The study introduced new ways to measure not just how accurate a prediction is, but how stable it remains over time and whether it improves as more information becomes available. They found that while many modern models could produce excellent point forecasts, they often failed to provide reliable probability estimates or to adjust their confidence levels when the data became volatile. For instance, in domains like weather and ocean dynamics, the top-ranked models from static tests performed poorly, while others adapted better to the changing conditions. This suggests that the current standard of testing, which relies on fixed datasets, may be giving a false sense of security about which models are truly ready for real-world use.

The study also highlighted that no single model is a universal winner. Even the most sophisticated systems showed weaknesses in specific areas, such as handling sudden spikes in data or adapting to long-term trends. The live benchmark revealed that performance is not a permanent attribute; a model that is excellent today might not be tomorrow if the underlying data patterns shift. By continuously updating the leaderboard as new data arrives, the researchers demonstrated that rankings are fluid and must be earned repeatedly. This approach exposes flaws that static tests miss, such as a model's tendency to smooth out important details or its failure to react quickly to new information.

Ultimately, this work suggests that the path to truly reliable forecasting requires moving beyond one-time evaluations. The researchers argue that to understand how these powerful tools will behave in practice, we must test them in the environment where they will actually be used: a world that is constantly changing. The live benchmark provides a framework for this, offering a transparent and reproducible way to see which models can truly stand the test of time. It does not declare a single champion but rather provides a dynamic map of strengths and weaknesses, helping scientists and practitioners choose the right tool for the job based on how it performs under real-world pressure. The findings serve as a reminder that in the complex, shifting landscape of time series data, adaptability is just as important as accuracy.

Technical Summary: LiveHouse-TS

Problem Statement

Time Series Foundation Models (TSFMs) have emerged as a promising paradigm for cross-domain zero-shot forecasting. However, current evaluation protocols rely heavily on static benchmarks (e.g., Monash, TFB, GIFT-Eval) that utilize fixed historical train/test windows. The authors argue that these static snapshots fail to capture the operational realities of real-world deployment, where data is non-stationary, subject to concept drift, and influenced by unexpected exogenous events.

In a static benchmark, a model's ranking is immutable once computed, potentially masking performance degradation under seasonal shifts or distribution changes. Furthermore, static evaluations often suffer from potential data leakage, where models might inadvertently access future information during training or evaluation setup. The paper posits that a model's operational superiority is not a permanent attribute but must be continuously tested as the world changes.

Methodology: LiveHouse-TS

To address these limitations, the authors introduce LiveHouse-TS, the first open-world living benchmark infrastructure for TSFMs. The system is designed as a continuous, streaming evaluation platform rather than a one-off leaderboard.

Core Design Principles

Leakage-Resistance: Tasks are constructed from continually arriving public streams. Predictions are made at the current world time before ground truth exists, ensuring no look-ahead bias.
Fairness: Models joining at different times are evaluated only on targets released after their admission, preventing information advantages for early entrants.
Scalability: The infrastructure supports an extensible registry for new data sources and models, allowing researchers to contribute via standardized APIs.

Architecture

The system consists of three decoupled components:

Model Entrance: A unified interface that standardizes heterogeneous forecasters (hosted TSFMs, external endpoints, statistical baselines). It validates inputs and converts diverse outputs (point forecasts, quantiles, samples) into a common scoring representation.
Live Data: Collectors periodically retrieve fresh observations from 17 public datasets across 11 domains (e.g., weather, finance, hydrology, macroeconomy). Data is cleaned, mapped to a shared schema, and windowed into forecasting tasks based on dataset-specific frequencies and horizons.
Evaluation House: Enforces the "future-only" rule. It filters out tasks issued before a model's admission time, dispatches tasks to models, and updates the leaderboard continuously as new ground truth arrives.

Evaluation Metrics

Beyond standard point metrics (RMSE, MAPE) and probabilistic metrics (CRPS), LiveHouse-TS introduces metrics specifically for live evaluation:

Temporal Stability: The standard deviation of a model's performance over time, measuring robustness to drift.
Improvement: A Kendall's $\tau$ metric quantifying whether a model's error consistently decreases as more information becomes available (monotone improvement).
Pairwise Historical Ranking: To handle asynchronous model entry, models are compared only on shared resolved releases. A dataset-balanced win rate and Elo rating are computed to ensure fair comparisons across different evaluation windows.

Key Contributions

New Paradigm: The paper identifies the gap in the "snapshot" evaluation paradigm and proposes a streaming protocol centered on continuous temporal validity to capture model robustness under real-world operational conditions.
New Infrastructure: LiveHouse-TS provides a leakage-resistant, open-world benchmark with a real-time leaderboard, an extensible data registry, and new metrics for temporal stability and monotone improvement.
New Insights: The authors demonstrate that rankings on LiveHouse-TS differ significantly from static benchmarks, suggesting that strong static performance does not necessarily translate to robustness in dynamic environments.

Experimental Results

The authors evaluated six TSFMs (Moirai-2.0, TimesFM-2.5, Chronos-2, TiRex, Toto-1.0, Chronos-Bolt, TabPFN-TS, Sundial) and four classical baselines across 10 stable datasets (selected from a registry of 17 due to missing observations in the others).

Zero-Shot Ability: TSFMs generally outperform classical statistical baselines in zero-shot settings. However, no single model dominates both point and probabilistic forecasting. Moirai-2.0 excels in probabilistic forecasting (CRPS), while TimesFM-2.5 and TiRex perform better in point accuracy (RMSE/MAPE).
Static vs. Live Rankings: A significant ranking inversion occurs between static and live benchmarks. Models like Chronos-2 and TimesFM-2.5, which often top static leaderboards, drop in LiveHouse-TS rankings. Conversely, Moirai-2.0 rises to the top in the live setting, while Toto-1.0 shows mixed results (strong in CRPS but poor in RMSE).
Drift Robustness: The ranking inversion is largely explained by temporal drift. Models that maintain stability under evolving data distributions (like Moirai-2.0) achieve better live performance. Chronos-2 and TimesFM-2.5, despite high static accuracy, show substantial degradation under temporal drift.
Ranking Stability: Even over consecutive weekly snapshots, model rankings continue to evolve. No single model dominates the live benchmark permanently, highlighting the dynamic nature of real-world forecasting.
Case Studies: Visualizations on datasets like Wiki pageviews, Wave heights, and PM2.5 reveal common failure modes in TSFMs under evolving distributions, such as oversmoothing spikes, delayed adaptation, and underestimating distribution shifts.

Significance and Claims

The paper claims that LiveHouse-TS shifts time series benchmarking from snapshot accuracy to continuous temporal validity. It argues that static leaderboards can be misleading, as they fail to capture aspects of robustness that only become apparent under continuous live evaluation.

The authors position LiveHouse-TS not merely as a leaderboard but as a living infrastructure designed to answer vital long-term scientific questions:

Can model rankings be maintained over the long term?
Which models remain genuinely robust under distribution shifts?

The work emphasizes that operational superiority is not a permanent attribute; performance and ranking must be continuously earned as the world changes. The infrastructure is designed to be transparent and reproducible, with code, datasets, and the leaderboard available to the community. The authors modestly note that while the results highlight the strengths of modern TSFMs, they also reveal remaining gaps in robustness and generalization when conditions shift across domains.