🤖 machine learning

Selective Time Series Forecasting via Metalearning

This paper proposes a metalearning-based selective forecasting framework that uses structural lag characteristics to model error percentiles, enabling domain-agnostic abstention from difficult predictions and significantly improving accuracy across diverse time series.

Original authors: Ricardo Inácio, Vitor Cerqueira, Marília Barandas, Carlos Soares

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ricardo Inácio, Vitor Cerqueira, Marília Barandas, Carlos Soares

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a weather forecaster. You have a super-smart computer model that predicts the weather for the next week. Usually, it's great. But sometimes, the weather is just chaotic—sudden storms, weird temperature swings—and the model gets it wrong.

The problem is, the model doesn't know when it's about to mess up. It just spits out a prediction every time, whether it's confident or not. If you could tell the model, "Hey, don't give me a prediction for Tuesday because the conditions look too messy," you would avoid being misled by bad data.

This paper proposes a way to teach a computer to say, "I'm not sure about this one," before it even makes the prediction.

The Old Way: Guessing Based on the Answer

Currently, some systems try to decide if a prediction is risky by looking at the prediction itself.

The Analogy: Imagine a student taking a test. After they write an answer, they look at it and think, "Hmm, this answer looks shaky, maybe I should erase it."
The Flaw: This only works if the student has seen similar tests before. If you take that same student and put them in a completely different subject (like switching from History to Physics), their ability to judge their own shaky answers might fail because the "style" of the questions is different.

The New Way: The "Pre-Flight" Inspector

The authors propose a new method called Selective Forecasting via Metalearning. Instead of looking at the answer to see if it's risky, they look at the conditions leading up to the question.

Think of it like a pre-flight inspection for a plane:

The Forecast Model: This is the pilot flying the plane (predicting the future).
The Meta-Model (The Inspector): This is a separate, smaller AI that looks at the runway, the fuel, and the weather before the plane takes off.
The Decision: The inspector doesn't know where the plane is going yet. It just looks at the "structural features" of the recent past (like trends, seasons, or how chaotic the data has been).
- If the runway looks bumpy and the fuel gauge is flickering (high structural risk), the inspector says, "Do not fly today." (Reject the prediction).
- If the runway is smooth, they say, "Go ahead." (Accept the prediction).

Why is this special?

The paper claims this method is universal (or "scale-invariant").

The Analogy: Imagine you have a mechanic who is an expert at fixing Toyotas. Usually, if you bring them a Ford, they might not know what to do. But this specific mechanic learned to recognize the sound of a failing engine, regardless of the car brand.
The Paper's Claim: Because this "Inspector" (Meta-Model) looks at the shape of the data patterns (like "is it getting chaotic?") rather than the specific numbers, it can learn from one set of data (e.g., electricity usage in Portugal) and apply that knowledge to a totally different set (e.g., tourism numbers in Brazil) without needing to relearn everything from scratch.

How They Tested It

They trained this "Inspector" on historical data. They taught it: "When the recent data looks like this pattern, the main model usually makes a big mistake."

Then, they tested it in two ways:

Same Domain: Using it on the same type of data it learned from.
Transfer Learning: Using it on completely different data (like switching from monthly sales data to quarterly tourism data).

The Results:

The "Inspector" was very good at spotting the "bad days" where the main model would fail.
When they let the system skip the bad predictions (the "rejected" ones), the average accuracy of the remaining predictions went up significantly.
It worked better than the old methods (which looked at the prediction itself) because it didn't get confused when the data changed from one type to another.

The Bottom Line

This paper introduces a "gatekeeper" for time-series predictions. Instead of blindly trusting a prediction, this gatekeeper checks the recent history. If the history looks too messy or unpredictable, the gatekeeper blocks the prediction. This ensures that you only get predictions when the system is likely to be right, saving you from making decisions based on bad guesses.

What the paper does NOT claim:

It does not claim this works for medical diagnoses or clinical uses.
It does not claim this is a magic bullet that fixes all bad data; it just helps you avoid the worst errors.
It does not claim the system is perfect; it still makes mistakes, but it makes fewer mistakes than the alternatives when you are willing to skip some predictions.

Technical Summary: Selective Time Series Forecasting via Metalearning

Problem Statement

Deep learning methods have achieved state-of-the-art performance in time series forecasting; however, their accuracy is not uniform across all samples. Certain instances within a time series are inherently difficult to predict, leading to substantial variations in forecasting errors. While "reject option" mechanisms—allowing models to abstain from high-risk predictions—are well-established in classification and regression, they remain underexplored in forecasting.

Existing abstention strategies in forecasting typically rely on proxy heuristics derived from the forecast itself, such as prediction interval widths, nonconformity scores, or learned confidence estimates. The paper argues that these approaches are inherently tied to the specific training domain and model architecture. Consequently, they struggle to generalize to new domains or time series with different characteristics, limiting their utility in transfer learning settings.

Methodology

The authors propose a selective forecasting framework based on metalearning that decouples the rejection decision from the forecast output. The core innovation is modeling the empirical percentile of forecasting errors (a scale-invariant statistic) based on structural characteristics extracted from recent lags, rather than relying on forecast-derived signals.

The methodology consists of two primary stages:

1. Performance Estimation (Source Domain)

Using a rolling origin evaluation procedure (time series cross-validation), the authors train a global forecasting model (e.g., KAN or NHITS) on a source domain. For every forecast origin $t$ , the model predicts a horizon $h$ , and the error $e_{i,t}$ is computed against the ground truth.

2. Metalearning

Instead of using raw error values as targets, the framework transforms errors into empirical percentile ranks computed within each individual time series:
$u_{i,t} = \frac{\#\{j \in T_i : j \neq t, e_{i,j} < e_{i,t}\}}{|T_i| + 1}$
This normalization maps errors to a bounded $(0, 1)$ range, creating a scale-invariant target that allows the meta-model to learn transferable rejection criteria across heterogeneous datasets.

Meta-Predictors: The input features for the meta-model are extracted from the lag window preceding each forecast origin using the TSFEL library. These features summarize structural characteristics such as trend, seasonality, and complexity, rather than absolute values.

Meta-Model: A CatBoostRegressor is trained to map these structural features to the predicted error percentiles.

Inference and Rejection

At inference, the meta-model scores each forecast origin before the forecast is issued.

Zero-shot: The source-trained meta-model is applied directly to a new domain.
Domain-adapted: The meta-model is fine-tuned on a small subset of labeled data from the target domain.
Decision Rule: Forecasts with a predicted percentile score $\hat{u}_{i,t} \geq q$ (where $q$ is a threshold corresponding to the desired coverage level) are rejected.

Key Contributions

Metalearning for Error Estimation: The paper introduces a metalearning approach that estimates large forecasting errors using structural properties of recent lags and scale-invariant targets (error percentiles).
Domain-Agnostic Transfer: By decoupling the rejection decision from the forecast and grounding it in domain-agnostic features, the framework enables effective abstention transfer across heterogeneous time series.
Empirical Validation: The authors provide evidence that this approach outperforms forecast-derived heuristics (such as prediction interval widths and residual variances) in both in-domain and transfer learning settings.

Experimental Results

The framework was evaluated on the M3, M1, and Tourism univariate time series collections (monthly and quarterly frequencies) using KAN and NHITS as base forecasters.

Meta-level Evaluation (Ranking Quality)

In-domain: The meta-model achieved high Spearman correlation ( $\rho$ between 0.71 and 0.90) between predicted scores and actual error ranks, consistently outperforming baselines like PI-width and residual variance.
Transfer Learning: While zero-shot transfer showed some degradation due to domain shift, the meta-model still outperformed most baselines. Crucially, domain adaptation (fine-tuning on 30% of target data) substantially recovered performance, often matching or exceeding the source-domain performance.
Baselines: Forecast-derived baselines (PI-width, Residual-scale) generally failed to generalize in transfer settings, often performing worse than random rejection in zero-shot scenarios.

Base-level Evaluation (Forecasting Accuracy)

Selective forecasting using the meta-model consistently improved the average sMAPE of accepted forecasts across all coverage levels compared to "keep all" and other baselines.
The method achieved the lowest average gap to the "Oracle" (the theoretical best performance if true errors were known) across all rejection fractions.
The approach proved model-agnostic, yielding similar improvements regardless of whether KAN or NHITS was used as the underlying forecaster.

Significance and Claims

The paper claims that forecastability can be treated as a structural property of recent observations, enabling risk-aware decisions before the forecast horizon is realized.

Pre-forecast Triage: Unlike uncertainty quantification methods that require the forecast to be generated first, this method operates ex ante, using only lag descriptors to screen out difficult instances.
Generalization: The scale-invariant design allows the mechanism to transfer effectively across domains without retraining the base forecaster, addressing a key limitation of existing selective prediction methods.
Practical Utility: The results suggest that practitioners can selectively avoid unreliable predictions, significantly reducing average error in critical applications where high-risk forecasts are detrimental.

The authors acknowledge limitations, noting that the method relies on the representativeness of meta-training data and estimates relative risk rather than providing calibrated probabilistic guarantees. They suggest future work could integrate this mechanism with conformal prediction for formal uncertainty quantification.