📊 statistics

Model selection with proper scoring rules on data sets of time series

This paper investigates how score distribution skewness causes conflicting model selection outcomes among mean, median, and rank-based statistics on time series data, demonstrating that while these criteria converge with large test sets, the mean score is uniquely reliable for identifying the true model on short test sets, as evidenced by analysis of intermittent time series including the M5 competition.

Original authors: Giorgio Corani, Stefano Damato, Dario Azzimonti, Lorenzo Zambon

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Giorgio Corani, Stefano Damato, Dario Azzimonti, Lorenzo Zambon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a coach trying to pick the best player for your team. You have a huge roster of players (time series), and you want to see who performs best on average. To judge them, you use a "scorecard" (a proper scoring rule) that gives a penalty number for every mistake they make. The lower the number, the better the player.

This paper is about how to read that scorecard when you have many players and many games to look at. The authors discovered that the way we usually tally the scores can sometimes trick us into picking the wrong player, especially when the games are short or the mistakes are rare but huge.

Here is the breakdown using simple analogies:

1. The Problem: Two Ways to Count

When you have a team of players, you can't just look at one game. You have to combine their results. The paper says there are two main ways coaches usually do this:

Method A: The "Average Penalty" (Mean Scaled Score). You take every single penalty point a player got, add them all up, and divide by the number of games. This tells you the average cost of their mistakes.
Method B: The "Win-Loss Record" (Mean Rank). You don't look at the points. Instead, you just ask: "In how many games did Player A beat Player B?" You count the wins. If Player A won more games, they get the top spot.

2. The Trap: The "Big Mistake" Skew

The authors found that for certain types of predictions (like predicting rare events or high values), the "scorecard" is skewed.

The Analogy:
Imagine a game where you usually make small mistakes (like tripping over your shoelaces), but occasionally, you make one massive, catastrophic mistake (like falling off a cliff).

The "Average Penalty" (Method A) sees the cliff fall. It adds up all the small trips and the one huge fall, giving a high average penalty. It knows you are dangerous.
The "Win-Loss Record" (Method B) looks at the games individually. In 99 out of 100 games, you only tripped on your shoelaces, which is a tiny penalty. In the other 1 game, you fell off a cliff.
- If you are playing against a rival who never falls off a cliff but trips slightly more often, Method B might say, "Hey, you won 99 games! You are the champion!"
- But Method A says, "Wait, that one cliff fall cost you the whole season. Your average penalty is actually worse."

The paper argues that Method B (Mean Rank) is dangerous because it ignores the size of the mistakes. It only cares about who won the most individual rounds. If the "cliff falls" (huge errors) are rare, Method B often misses them, especially if you haven't watched enough games (short test sets).

3. The Solution: Watch More Games

The authors ran simulations to see what happens as you watch more games (increase the test set size).

Short Test Sets (Few Games): Method B is unreliable. It often picks the "lucky" player who avoids the big mistakes just by chance, even if their average performance is worse.
Long Test Sets (Many Games): As you watch more and more games, the "cliff falls" eventually happen enough times that Method B starts to see the truth. The two methods start to agree.

The Key Finding: If you only have a short history of data (a short test set), Method A (Average Penalty) is the only one that consistently picks the truly best model. Method B is too easily fooled by the "luck" of avoiding rare, huge errors.

4. The Real-World Test: The M5 Competition

The authors tested this on real data from the famous "M5 Forecasting Competition," which involves predicting sales for thousands of products (some of which are "intermittent," meaning they sell very rarely).

They compared two mathematical models:

The Poisson Model: A simpler model.
The Negative Binomial Model: A more complex model known to be better at handling "spikes" in demand.

What happened?

When using Method B (Mean Rank) on short data, it often picked the Poisson model. It thought the simpler model was better because it "won" more individual rounds.
When using Method A (Average Penalty), it consistently picked the Negative Binomial model, correctly identifying it as the one that handles the big spikes better.

The authors concluded that the "Average Penalty" was right. The "Win-Loss Record" was fooled because the test sets were too short to capture the rare, massive errors that the simpler model failed to predict.

5. Does it matter how we measure the penalty?

The paper also checked if changing the "ruler" (scaling factor) used to measure the penalties changed the results.

Good News: The "Average Penalty" method is very robust. Whether you measure the penalty in dollars, in percentage, or against a baseline, it almost always picks the same winner.
Bad News: The "Win-Loss Record" method is fragile. It can flip-flop depending on how you measure things and how much data you have.

The Bottom Line

If you are trying to pick the best forecasting model:

Don't just count wins. (Don't rely solely on Mean Rank).
Look at the average cost of mistakes. (Use the Mean Scaled Score).
Be careful with short data. If you don't have enough history, the "Win-Loss" method will likely trick you into picking a model that looks good on paper but fails when a rare, big event happens.

The paper essentially says: "Don't let a player who gets lucky avoiding one huge disaster fool you into thinking they are the best player. Look at their total average performance."

Technical Summary: Model Selection with Proper Scoring Rules on Time Series Data Sets

Problem Statement
The paper addresses the challenge of selecting the best probabilistic model across a dataset containing multiple time series. While model selection on a single time series is straightforward (selecting the model with the lowest score), aggregating performance across $n$ time series requires summary statistics. The two most common approaches are the mean scaled score (averaging scores after scaling to ensure comparability) and the mean rank (averaging the rank of models across series). Previous literature has noted that these two criteria can yield conflicting decisions, yet the underlying cause and the reliability of each criterion remain unclear. Furthermore, the sensitivity of the mean scaled score to the choice of scaling factor is an open question.

Methodology
The authors investigate these issues through a combination of theoretical analysis, simulation, and empirical evaluation on real-world intermittent time series data.

Theoretical Analysis of Skewness: The paper analyzes the distribution of pointwise scores for common proper scoring rules (Quantile Score, Interval Score, CRPS). It demonstrates that these distributions are inherently skewed, particularly for high quantiles ( $p > 0.5$ ). The skewness arises because underestimation errors (rare events for high quantiles) are penalized much more heavily than overestimation errors.
Simulation Study: A simulation was conducted comparing a true model ( $N(0, 1)$ ) against a misspecified model ( $N(0, 0.85^2)$ ) across varying test set lengths ( $n_{te}$ ). The study evaluates how the mean score (estimating the expected value) and the mean rank (estimating the median difference) behave as $n_{te}$ increases.
Empirical Evaluation: The authors applied these criteria to three datasets of intermittent demand: the bottom-level series of the M5 competition (30,490 series), Auto (3,000 series), and RAF (5,000 series). They compared the Poisson and Negative Binomial distributions (the latter theoretically superior for intermittent demand due to overdispersion) using various scoring rules (CRPS, QS at different quantiles) and scaling factors (MAE naive, in-sample mean, in-sample score of the empirical distribution).

Key Contributions and Findings

Source of Conflict: The paper establishes that conflicting decisions between mean rank and mean scaled score are caused by the skewness of the score distribution.
- The mean scaled score estimates the expected value of the scoring rule.
- The mean rank (and related statistics like win rate or median score) effectively estimates the median difference between models.
- When the distribution of score differences is skewed (common with small $n_{te}$ or high quantiles), the mean and median can point to different "best" models. Specifically, a misspecified model may frequently achieve a lower score on short test sets by avoiding rare, heavily penalized underestimation errors, leading the mean rank to incorrectly favor the misspecified model.
Convergence with Test Set Size: As the test set length ( $n_{te}$ ) increases, the distribution of scores becomes more symmetric (due to the Central Limit Theorem), and the mean rank and mean scaled score progressively converge to the same conclusion. However, for short test sets, only the mean score reliably identifies the true model.
Robustness of Mean Scaled Score: The empirical results on the M5, Auto, and RAF datasets show that the decision based on the mean scaled score is robust. It consistently selects the theoretically superior Negative Binomial model across all test set lengths and all tested scaling factors.
Scaling Factor Sensitivity: Contrary to the perceived risk that model selection depends heavily on the scaling factor, the study finds that the choice of scaling factor (e.g., MAE naive vs. in-sample mean) has a minor impact on the final model selection decision when using the mean scaled score. The decision remains stable across different scaling methods.
Reliability of Mean Rank: The mean rank is shown to be an unreliable criterion for short test sets, particularly when evaluating high quantiles. In the M5 dataset experiments, the mean rank favored the inferior Poisson model for high quantiles (QS0.975, QS0.995) even with a substantial number of series, whereas the mean scaled score correctly identified the Negative Binomial.

Significance and Recommendations
The paper concludes that model selection based on proper scoring rules fundamentally requires estimating the expected value of the score. Because non-mean statistics (mean rank, median, win rate) do not estimate the expected value, they can lead to incorrect model selection when score distributions are skewed, a condition prevalent in time series with short test sets or high quantile forecasts.

The authors recommend:

Prioritizing the mean scaled score over the mean rank for model selection, as it is a direct estimator of the expected score.
Exercising caution when relying on mean ranks, especially for high quantile levels or small $n_{te}$ .
Validating conclusions by checking results across multiple scaling factors, though the study suggests the specific choice of factor is less critical than previously assumed.
Acknowledging that determining the exact threshold for "large enough" $n_{te}$ to ensure convergence remains difficult and depends on the specific scoring rule and data distribution.

The paper does not propose new scoring rules or specific future applications but provides a critical theoretical and empirical justification for preferring mean-based aggregation over rank-based aggregation in probabilistic time series model selection.