🤖 machine learning

Beyond Model Ranking: Predictability-Aligned Evaluation for Time Series Forecasting

This paper introduces a novel predictability-aligned evaluation framework based on spectral coherence, featuring the Spectral Coherence Predictability (SCP) and Linear Utilization Ratio (LUR) metrics, to address the limitations of standard benchmarks by quantifying data difficulty and revealing that model effectiveness depends on the inherent predictability of the forecasting task.

Original authors: Wanjin Feng, Yuan Yuan, Jingtao Ding, Yong Li

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wanjin Feng, Yuan Yuan, Jingtao Ding, Yong Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a coach trying to figure out which of your athletes is the best runner. Usually, you just look at their race times. If Runner A finishes in 10 seconds and Runner B in 12, you assume A is better.

But what if Runner A was running on a flat, smooth track with a tailwind, while Runner B was running up a steep, rocky hill in the rain? The time alone doesn't tell the whole story. You can't fairly compare them without knowing how hard the track was.

This is exactly the problem the paper "Beyond Model Ranking" identifies in the world of AI time-series forecasting.

The Problem: The "Leaderboard" Trap

Right now, AI researchers judge forecasting models (which predict things like electricity usage or stock prices) using a simple scoreboard: How close was the prediction to the actual result? They use a metric called "Mean Squared Error" (MSE).

The paper argues this is misleading. It mixes up two things:

The Model's Skill: How good the AI is.
The Task's Difficulty: How predictable the data actually is.

Sometimes a simple, "dumb" model looks great just because the data it was tested on was easy to guess (like predicting the sun will rise tomorrow). Other times, a super-smart model looks bad because it was tested on chaotic, noisy data that is nearly impossible to predict. The current "leaderboards" hide this difference, making it hard to know who is actually the best athlete.

The Solution: A New Diagnostic Tool

The authors propose a new way to evaluate models that separates the runner from the track. They introduce two main tools:

1. SCP: The "Track Difficulty" Meter

Spectral Coherence Predictability (SCP) is a way to measure how easy or hard a specific piece of data is to predict before you even run the model.

The Analogy: Think of a radio signal. Sometimes the signal is clear and strong (easy to predict). Sometimes it's full of static and noise (hard to predict).
How it works: The authors look at the data in the "frequency domain" (like tuning a radio to different stations). They calculate how much of the future signal can be explained by the past signal using simple linear math.
The Result: They get a score (0 to 1) that tells you the "theoretical limit" of how well any model could possibly do on that specific data. If the score is low, the data is inherently messy. If it's high, the data is very predictable.
Why it matters: It gives you a "lower bound" (a floor). If a model's error is close to this floor, it's doing its best. If the error is way higher, the model is underperforming.

2. LUR: The "Efficiency" Gauge

Linear Utilization Ratio (LUR) measures how well a specific model uses the information that is available.

The Analogy: Imagine you have a bucket of water (the predictable information).
- LUR < 1: The model is spilling water. It's failing to catch the simple patterns that were right there to be found.
- LUR ≈ 1: The model is catching every drop of the simple, linear water. It has hit the ceiling of what simple math can do.
- LUR > 1: The model is doing something magical. It's finding patterns that simple math couldn't see (non-linear patterns), effectively getting more out of the bucket than the "difficulty meter" said was possible.

What They Discovered

By using these tools, the authors found some surprising things:

Difficulty Drifts: The "difficulty" of a task isn't static. Just like the weather changes, the predictability of a time series changes over time. A dataset might be easy to predict for a week, then suddenly become chaotic. Standard averages hide this; their tool sees it.
Different Models for Different Jobs:
- Simple Models (Linear): These are like sprinters. They are incredibly fast and efficient at catching the "easy" signals (low-frequency, stable patterns). When the data is predictable, they often beat the complex models.
- Complex Models (Transformers/Deep Learning): These are like marathon runners with high-tech gear. They struggle a bit more with the simple stuff but shine when the data gets messy and non-linear. They are better at finding the "hidden" patterns when the simple math fails.
Fairer Comparisons: Instead of just saying "Model A is better than Model B," this framework says, "Model A is better at predicting easy data, while Model B is better at handling hard data."

The Bottom Line

The paper doesn't just want to rank models; it wants to diagnose them.

They are asking us to stop looking at a single number on a leaderboard and start asking: "Was this model bad, or was the data just impossible?" and "Did this model waste the easy opportunities, or did it find something new?"

By using their "Spectral Coherence Predictability" (the difficulty meter) and "Linear Utilization Ratio" (the efficiency gauge), we can finally have a fair conversation about which AI models are actually doing the best work.

Technical Summary: Beyond Model Ranking: Predictability-Aligned Evaluation for Time Series Forecasting

1. Problem Statement

The field of time series forecasting faces a critical evaluation dilemma: standard aggregate metrics (e.g., Mean Squared Error, MSE) conflate model limitations with the intrinsic difficulty of the data instances being evaluated. Current leaderboards often rank models based on marginal improvements in aggregate error, obscuring why errors occur. A sophisticated model may appear inferior to a baseline simply because the test sequence is regular and easy to predict, or conversely, a model may appear successful on a dataset that is inherently easy.

Existing tools for quantifying predictability, such as entropy-rate estimators and Lempel-Ziv complexity, are ill-suited for modern deep learning forecasting because they:

Paradigm Mismatch: They were designed for symbolic dynamics and single-step classification (0–1 loss) rather than multi-horizon regression under squared error.
Computational Cost: They often entail quadratic-to-cubic complexity, making them prohibitive for large-scale, high-dimensional time series.
Lack of Diagnostic Insight: They typically yield a single global score, failing to reveal where a model fails across time, channels, or frequency bands.

The paper argues that the field requires a diagnostic framework that quantifies instance difficulty in alignment with forecasting objectives to enable stratified evaluation and reveal where models under-utilize available information.

2. Methodology

The authors propose a diagnostic framework grounded in Spectral Coherence, consisting of two core components: Spectral Coherence Predictability (SCP) and Linear Utilization Ratio (LUR).

2.1 Spectral Coherence Predictability (SCP)

SCP serves as a computationally efficient, task-aligned surrogate for the theoretical Bayes risk (intrinsic predictability). It quantifies the amount of linearly exploitable information available for forecasting a specific instance.

Mechanism: Operating in the frequency domain via Welch's method, SCP computes the squared coherence $\gamma^2_{xy}(f)$ between the history $x$ and the future $y$ .
Linear Lower Bound: It derives a conservative linear Mean Squared Error lower bound ( $MSE_{lb}$ ) by aggregating the residual spectrum (unexplained power) and adding a mean-shift term to account for boundary mean mismatches:
$MSE_{lb} = \Delta^2 + \sum_{f} \hat{S}_{e}(f)$
where $\hat{S}_{e}(f) = \hat{S}_{yy}(f)(1 - \gamma^2_{xy}(f))$ .
Predictability Score: The predictability $P_{xy}$ is defined as the normalized reduction of uncertainty:
$P_{xy} = \max\left(0, 1 - \frac{MSE_{lb}}{\widehat{Var}(y)}\right)$
Efficiency: The method operates in $O(N \log N)$ time per sample using Fast Fourier Transforms (FFT), significantly faster than matching-based complexity estimators.

2.2 Linear Utilization Ratio (LUR)

While SCP measures instance difficulty, LUR diagnoses how effectively a specific model exploits the linearly predictable structure.

Mechanism: LUR compares the power-weighted coherence between the model's prediction $\hat{y}$ and the target $y$ against the coherence between the history $x$ and the target $y$ .
Definition:
$LUR = \frac{P_{model}}{P_{linear}} = \frac{\sum \gamma^2_{y\hat{y}}(f) \hat{S}_{yy}(f)}{\sum \gamma^2_{yx}(f) \hat{S}_{yy}(f)}$
Interpretation:
- $LUR < 1$: Indicates under-utilization (the model fails to capture linear correlations present in the history).
- $LUR \approx 1$ : Indicates the model has saturated the linear information ceiling.
- $LUR > 1$: Suggests the model captures additional predictive gains via non-linear dynamics or global inductive biases.
Stratification: The framework allows for band-wise analysis (low/mid/high frequencies), enabling fine-grained assessment of model behavior across spectral bands.

3. Key Contributions

Predictability-Aware Diagnostic Framework: The paper introduces a systematic approach to separate model performance from instance difficulty, addressing the ambiguity in current evaluation protocols.
SCP and LUR Metrics:
- SCP: A computationally efficient ( $O(N \log N)$ ), task-aligned instance-difficulty reference that provides a linear MSE lower bound.
- LUR: A frequency-resolved diagnostic quantifying model efficiency in exploiting linear structures.
Empirical Validation: Extensive experiments on synthetic and real-world benchmarks (ETTh1/2, ETTm1/2, ECL, Weather, Traffic, ILI) demonstrate that SCP is well-calibrated and strongly correlates with realized forecasting errors across diverse state-of-the-art (SOTA) models.

4. Experimental Results

Calibration: In synthetic settings, SCP increases monotonically with Signal-to-Noise Ratio (SNR), and $MSE_{lb}$ tightly tracks the error of optimal linear forecasters. On real-world datasets, SCP shows a strong inverse correlation with normalized MSE (NMSE) across models (Pearson $R \ge 0.8$ ).
Predictability Drift: The analysis reveals that task difficulty is not static. "Predictability drift" occurs where the amount of exploitable information fluctuates significantly over time and across variables. Aggregate test-set statistics can be misleading because they average over these varying difficulty regimes.
Stratified Evaluation:
- Frequency Bands: Linear models (e.g., DLinear) often show higher LUR in high-frequency bands, while Transformer-based models (e.g., iTransformer, PatchTST) excel in low-frequency, high-energy bands.
- Difficulty Regimes: Non-linear models tend to outperform linear baselines in low-predictability (hard) regimes, whereas linear models are highly competitive in high-predictability (easy) regimes.
Robustness: The framework remains stable across different Welch parameters (window length, overlap) and history window lengths. It also extends to multivariate settings and nonlinear feature spaces (via kernel maps).

5. Significance and Claims

The paper advocates for a paradigm shift from "leaderboard-style ranking" to "predictability-aware diagnostics." The authors claim that:

Fairer Comparisons: By stratifying evaluation based on instance difficulty (SCP), researchers can uncover complementary architectural strengths that aggregate scores obscure.
Actionable Insights: The framework identifies specific failure modes (e.g., under-utilization of linear signals vs. inability to model non-linear drifts), guiding the development of more robust forecasting models.
Efficiency: The proposed metrics provide a rigorous, low-cost alternative to complex entropy-based methods, making instance-level difficulty assessment feasible for large-scale benchmarks.

The authors conclude that moving beyond simple error minimization toward understanding the predictability of the data instances is essential for scientific iteration and understanding the true capabilities of time series forecasting models.