A Later Test Set Is Not a New Domain: Pretraining Familiarity Survives a Contamination-Free Hold-Out
This paper demonstrates that temporal hold-out tests fail to fully eliminate the performance advantage of time-series foundation models because their success stems primarily from pretraining familiarity with specific data domains rather than general forecasting capability, necessitating domain-level hold-outs for fair evaluation.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of data science, there is a growing belief that a single, massive computer program can learn to predict the future of almost any repeating pattern, from electricity usage to stock prices, without ever being specifically taught about that particular pattern. These programs, known as foundation models, are trained on enormous libraries of historical data, learning the general "shape" of how time moves. The promise is that once trained, they can look at a new set of numbers and forecast what comes next better than traditional methods, which usually have to be built from scratch for each specific job. However, a shadow has hung over these claims. Because the tests used to measure these models rely on old, public records, it has been impossible to tell if a model is truly good at predicting the future or if it has simply memorized the past. If a model is tested on data that existed before it was built, it might be recalling facts it saw during its training rather than demonstrating a genuine ability to forecast.
To solve this puzzle, researchers built a new kind of test that no model could possibly have seen before. They gathered thirteen different forecasting tools—some old and simple, others new and complex—and asked them to predict data that was published after the newest model had already been released. The data came from five distinct areas of human activity: how many people viewed Wikipedia pages, hourly weather patterns, hourly air quality readings, electricity usage on the Danish power grid, and daily exchange rates between currencies. Every single observation used for the test was recorded in 2026, a time that had not yet arrived when the models were trained. This ensured that no model could have memorized the specific numbers it was being asked to predict. The goal was to see if these powerful, pre-trained giants could still outperform the humble, old-fashioned tools when faced with a truly fresh challenge.
The results revealed a story more nuanced than the headlines suggested. The pre-trained models did not win everywhere. On the daily exchange rates, every method tested, from the most advanced artificial intelligence to the simplest guess, performed exactly the same, and none could beat a basic forecast that assumed tomorrow would look like today. On the hourly electricity grid data, a classic, non-machine-learning method called Theta took the top spot, with the fancy pre-trained models unable to distinguish themselves from it. In these cases, the new technology offered no advantage. However, the models did shine in other areas. They performed significantly better on the Wikipedia pageview data, where they predicted traffic spikes far more accurately than the classical methods. The difference was stark: on weekly Wikipedia views, the best pre-trained model made errors that were 28 percent smaller than the best classical method.
The researchers then asked why the models succeeded in some places but failed in others. They initially suspected that the nature of the data itself was the key. They wondered if the models worked better on data that was very noisy or had complex, repeating cycles that were hard to spot. They measured the strength of these cycles and the amount of randomness in the data, but these factors did not explain the pattern. The models did not do better just because the data was messy or highly seasonal. Instead, the answer lay in the training history of the models themselves. The domain where the models performed best was Wikipedia pageviews. This is the exact same type of data that the creators of one of the leading models, TimesFM, described as making up the bulk of its training library. The model had spent years reading millions of Wikipedia traffic patterns. Even though it had never seen the specific 2026 numbers, it had learned the general behavior of Wikipedia traffic so well that it could predict the future of that specific domain with ease.
This finding suggests that the advantage of these models comes less from a universal ability to forecast anything, and more from a deep familiarity with the specific kinds of data they were raised on. When the researchers compared the different pre-trained models against each other on the same Wikipedia data, the family of models that had been trained on Wikipedia data consistently outperformed the others. On other data types, like weather or air quality, that same advantage disappeared. The models were not magic; they were specialists who had read a specific library of books and could recall the stories within them, even if the chapters were new.
The study also uncovered a hidden flaw in how these models express their confidence. While the pre-trained models were often accurate in their point predictions, they were systematically overconfident. When they provided a range of possible outcomes, they claimed to be 80 percent sure, but their actual predictions only covered the true result about 70 percent of the time. In contrast, the older, classical methods were more cautious, often providing wider ranges that captured the true outcome more frequently. For a person using these forecasts to manage a power grid or a supply chain, this overconfidence could be dangerous, leading to reserves that are too small. The researchers noted that while the pre-trained models were faster and often more accurate, their lack of calibration meant they were not always the safer tool for critical decisions.
Ultimately, the paper concludes that simply moving the test date forward is not enough to prove a model is truly generalizing. A model can pass a test on future dates if it has simply learned the "flavor" of a specific domain during its training. To truly measure a model's ability to generalize, future benchmarks must hold out entire domains that were not part of the training data, not just future dates. For the practitioner, the lesson is clear: before choosing a model, one must ask not just which tool is the most powerful, but whether the data they are trying to predict looks like the data the tool was raised on. If the data is different, the model's apparent brilliance may vanish, leaving the simpler, more cautious tools as the more reliable choice.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.