📊 statistics

Beyond Accuracy: Are Time Series Foundation Models Well-Calibrated?

This paper systematically evaluates the calibration of five recent time series foundation models, finding that they are consistently better calibrated and less prone to systematic overconfidence than baseline models across various forecasting scenarios.

Original authors: Coen Adler, Yuxin Chang, Felix Draxler, Samar Abdi, Padhraic Smyth

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Coen Adler, Yuxin Chang, Felix Draxler, Samar Abdi, Padhraic Smyth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a crystal ball that can predict the future of things like stock prices, weather, or how many people will buy a specific product. In recent years, scientists have built "Foundation Models" for time series data. Think of these as super-smart, all-purpose crystal balls trained on a massive amount of different data from many different worlds. They are incredibly good at guessing the exact number that will happen next (the "point forecast").

But there's a catch: a crystal ball is only truly useful if it tells you how sure it is. If it predicts it will rain tomorrow with 100% confidence, but it's sunny, that's a bad crystal ball. This "honesty" about its own certainty is called calibration.

This paper asks a simple question: Are these new, super-smart time-series crystal balls actually honest about their confidence, or are they just confident bullies?

Here is what the authors found, broken down with some everyday analogies:

1. The "Overconfident Student" vs. The "Honest Expert"

In the world of AI, many deep learning models are like overconfident students who raise their hands and shout the answer even when they have no idea what they are talking about. They are "overconfident."

The researchers tested five of the newest, most advanced time-series foundation models against older, traditional methods (like ARIMA and N-BEATS).

The Old Guard (Baselines): These models were consistently under-confident. Imagine a weather forecaster who says, "It might rain, or it might not, but I'm going to give you a huge umbrella just in case." They are so unsure they make their prediction intervals so wide they are almost useless.
The New Foundation Models: These models were honest. They weren't systematically overconfident (shouting answers they didn't know) nor under-confident (waving their hands in panic). They struck a balance, giving you a prediction interval that actually matched the reality of the data.

2. The "Speedometer" Problem

The paper points out a tricky trap in how we usually measure these models. Imagine you are judging a race car driver.

Old Metric (WQL): This metric is like judging the driver based on how fast they went (sharpness) and how close they got to the finish line (accuracy). If a driver goes super fast but crashes, this metric might still give them a high score because they were "sharp."
New Metric (PCE): The authors used a different tool, called Probabilistic Calibration Error (PCE). This is like a honesty meter. It asks: "When you said there was a 90% chance of rain, did it actually rain 90% of the time?"

The paper found that using the old "speedometer" metric (WQL) sometimes tricked people into thinking the old, under-confident models were actually the best. But when they used the "honesty meter" (PCE), the new Foundation Models clearly won. They were the most honest predictors.

3. The "Swiss Army Knife" Heads

These foundation models are like Swiss Army knives. They have a main body (the brain) that processes the data, but they can attach different "heads" (tools) to make the final prediction.

Some heads predict a specific shape of probability (like a Bell Curve/Gaussian).
Some predict a list of specific chances (Quantiles).
Some predict a mix of shapes (Mixture).

The researchers tried swapping these heads out. They found that the "Gaussian" head (the simple Bell Curve) was like a blunt tool; it made the models under-confident again, acting like the nervous weather forecaster. However, the other heads (Quantiles and Mixture distributions) kept the models honest and well-calibrated. It turns out the "brain" of the model is so good that as long as you don't use the blunt tool, it stays honest.

4. The "Long Walk" Problem

Sometimes you need to predict far into the future, not just the next hour. To do this, the models have to take a "long walk," predicting step-by-step and using their own previous guesses to make the next guess. This is called autoregression.

The Problem: If you take a long walk and you make a tiny mistake in step 1, that mistake gets bigger by step 10, and huge by step 100.
The Fix: The researchers found that the way the model takes these steps matters.
- The "Branching" method: Like asking a group of people to guess the future, then splitting the group into smaller groups for the next step. This tended to make the models overconfident (too sure of themselves) when looking far ahead.
- The "Trajectory" method: Like simulating 100 different possible futures at once and seeing where they all land. This kept the models much more honest and well-calibrated, even for long-term predictions.

The Bottom Line

The paper concludes that these new Time Series Foundation Models are a big step forward. Unlike their predecessors, they don't just guess the number; they are honest about their uncertainty. They aren't the nervous types who cover all bases with huge guesses, nor are the arrogant types who are sure they are right when they aren't. They are the reliable experts who tell you exactly how likely an event is, making them much safer to use for important decisions.

Technical Summary: Beyond Accuracy: Are Time Series Foundation Models Well-Calibrated?

Problem Statement

While Time Series Foundation Models (TSFMs) have achieved state-of-the-art predictive accuracy across various domains, their calibration properties—the alignment between predicted probability distributions and observed data—remain underexplored. In practical applications such as anomaly detection and healthcare decision-making, reliable uncertainty quantification is as critical as point accuracy.

Existing literature often relies on metrics like the Continuous Ranked Probability Score (CRPS), Weighted Quantile Loss (WQL), and Mean Scaled Interval Score (MSIS) to evaluate calibration. However, as noted in prior work (Chung et al., 2021), these metrics conflate calibration with sharpness (the concentration of the predictive distribution). Consequently, they may incorrectly prioritize models that are sharp but poorly calibrated, or fail to distinguish between overconfident and underconfident models. Furthermore, it is unclear if TSFMs, which are often trained with calibration-aware losses (e.g., quantile loss), exhibit the systematic overconfidence common in deep learning models for image and text modalities (Guo et al., 2017; Jiang et al., 2021).

Methodology

The authors conduct a systematic empirical study evaluating five state-of-the-art TSFMs (Chronos-Bolt, TimesFM, Moirai 2.0, TiRex, and YingLong) against two established baselines (N-BEATS and ARIMA).

Experimental Setup

Datasets: Evaluation was performed across six univariate time series datasets spanning diverse domains and granularities: Reviews (hourly), Shopping/M5 (daily), Glucose (5-minute intervals), Heart-Rate (secondly), Crime (daily), and Patents (monthly).
Metrics: To isolate calibration from sharpness and accuracy, the study employs:
- Probabilistic Calibration Error (PCE): Measures the difference between empirical and predicted Cumulative Distribution Functions (CDFs).
- Centered Calibration Error (CCE): Quantifies systematic overconfidence (positive values) or under-confidence (negative values) by comparing observed data coverage against confidence levels.
- Scaled Interval Width (SIW): A proxy for sharpness, measuring the width of prediction intervals.
- Mean Absolute Scaled Error (MASE): Used to measure point accuracy independently of calibration.
Variables Investigated:
1. Prediction Heads: The authors trained alternative projection heads (Gaussian, Student's t, Mixture distributions) on the latent representations of the TSFMs to isolate the impact of the output layer on calibration.
2. Long-Term Forecasting: The study compares different autoregressive (AR) strategies for forecasting beyond the model's native horizon, including naive point-based AR, branching AR (Moirai 2.0), and trajectory-based AR (Toto/TiRex approaches).

Key Results

1. Calibration Superiority of TSFMs

TSFMs are consistently better calibrated than baseline models (N-BEATS and ARIMA).

PCE Performance: TSFMs generally achieve PCE values below 0.05 (5% error), whereas baselines often exhibit significantly higher errors.
Bias Analysis: Unlike deep learning models in other domains that tend to be overconfident, TSFMs do not show systematic over- or under-confidence in short-term forecasting. In contrast, baselines (particularly N-BEATS and ARIMA) are consistently under-confident, producing overly wide prediction intervals (high SIW) with negative CCE values.

2. Impact of Prediction Heads

The form of the prediction head significantly influences calibration:

Gaussian Heads: Consistently result in under-confident forecasts across all model backbones, yielding higher PCE and lower CCE (more negative) compared to other heads.
Quantile, Student's t, and Mixture Heads: These heads demonstrate similar, well-calibrated performance with no significant advantage of one over the others. The limited expressiveness of the Gaussian distribution appears to hinder its ability to model complex conditional distributions effectively.

3. Long-Term Forecasting and Autoregression

For long-term forecasting requiring autoregressive rollouts:

Horizon Length: Increasing the forecast horizon ( $H$ ) reduces calibration error. Models with shorter horizons (e.g., $H=16$ ) exhibit higher overconfidence (positive CCE) when rolled out compared to those with longer horizons ( $H=64, 128$ ).
AR Methods: The trajectory-based AR method (sampling from the predicted distribution) produces better-calibrated forecasts than the branching method (forecasting each quantile independently). While both methods show overconfidence in long-term settings, the trajectory approach mitigates this more effectively.
Native Long-Horizon Models: Models like TiRex and YingLong, which support long horizons natively without AR, are significantly more efficient and maintain better calibration without the systematic overconfidence seen in AR-based rollouts.

4. Metric Discrepancies

The study confirms that WQL and MSIS are highly correlated with point accuracy (MASE) rather than pure calibration. In some cases (e.g., the Glucose dataset), WQL incorrectly suggests that ARIMA is well-calibrated, whereas PCE reveals it is poorly calibrated. This highlights the necessity of using calibration-specific metrics like PCE and CCE.

Significance and Contributions

The paper makes the following primary contributions:

First Systematic Calibration Study: It provides the first comprehensive evaluation of calibration in TSFMs across multiple datasets, architectures, and forecasting strategies.
Metric Clarification: It demonstrates that standard metrics (WQL, MSIS) can be misleading for calibration assessment due to their conflation with sharpness and accuracy, advocating for the use of PCE and CCE.
Empirical Findings:
- TSFMs are generally well-calibrated and do not suffer from the systematic overconfidence typical of other deep learning foundation models.
- The choice of prediction head matters; Gaussian heads are inferior for calibration compared to quantile or mixture heads.
- Long-term forecasting via autoregression introduces overconfidence, which can be mitigated by using longer native horizons or trajectory-based AR methods.

The authors conclude that while TSFMs offer competitive point accuracy, their true value lies in their ability to provide reliable uncertainty quantification, making them suitable for high-stakes decision-making tasks where calibration is paramount. Future work is suggested to explore calibration under distribution shifts, fine-tuning scenarios, and multivariate settings.