🤖 machine learning

CombinationTS: A Modular Framework for Understanding Time-Series Forecasting Models

The paper introduces CombinationTS, a modular probabilistic framework that decomposes time-series forecasting models to reveal that well-designed data embeddings and input transformations often yield superior performance and stability compared to complex encoder architectures, challenging the necessity of increasing model complexity.

Original authors: Xiaorui Wang, Fanda Fan, Chenxi Wang, Yuxuan Yang, Rui Tang, Kuoyu Gao, Simiao Pang, Yuanfeng Shang, Zhipeng Liu, Wanling Gao, Lei Wang, Jianfeng Zhan

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaorui Wang, Fanda Fan, Chenxi Wang, Yuxuan Yang, Rui Tang, Kuoyu Gao, Simiao Pang, Yuanfeng Shang, Zhipeng Liu, Wanling Gao, Lei Wang, Jianfeng Zhan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of time-series forecasting (predicting future trends like stock prices, weather, or energy usage) as a high-stakes cooking competition. For years, chefs (researchers) have been trying to win by building increasingly complex, multi-layered ovens (AI models). They claim their new "Super-Oven" with 50 knobs and a digital brain is the best because it produced the tastiest cake in one specific trial.

However, the authors of this paper, CombinationTS, argue that the competition is rigged. They believe many of these "winning" ovens aren't actually better at cooking; they just got lucky with the ingredients or the specific temperature settings used in that one trial.

Here is the paper's core message, broken down into simple concepts and analogies:

1. The Problem: The "Black Box" and the "Lucky Shot"

Currently, scientists treat these forecasting models like black boxes. You put data in one side, and a prediction comes out the other. If the result is good, they say, "Our new complex engine is amazing!"

But the authors say this is misleading. It's like saying a car is fast because of its new engine, when in reality, the car was just driven on a downhill slope with a tailwind.

The Attribution Gap: We don't know if the success comes from the engine (the complex math inside) or the fuel (how the data was prepared and viewed).
The Benchmarking Crisis: Most researchers only report their single "best" result. This is like a golfer only showing the one hole where they got a "hole-in-one" and ignoring the 17 other holes where they struggled. This creates a false sense of superiority.

2. The Solution: The "Lego" Framework

To fix this, the authors built CombinationTS. Think of this as a Lego disassembly kit for forecasting models.

Instead of looking at the whole model as one giant, unchangeable brick, they break every model down into five distinct, interchangeable Lego blocks:

Input Transformation: How you wash and chop the ingredients before cooking.
Embedding: How you arrange the ingredients on the plate (the "data view").
Encoder: The actual cooking process (the complex reasoning).
Decoder: Plating the dish.
Output Transformation: Adding the final garnish or sauce.

The framework allows researchers to swap these blocks around. You can take the "chopping" from Model A, the "plating" from Model B, and the "cooking" from Model C to see which part actually does the heavy lifting.

3. The New Way to Judge: "The Weather Report" vs. "The Single Day"

Instead of judging a model on one single day (one specific setting), CombinationTS treats the evaluation like a weather forecast.

They don't just ask, "Did it rain today?" (Point Estimate).
They ask, "What is the average chance of rain over the next month, and how much does that chance fluctuate?" (Distribution).

They test every model under hundreds of different "weather conditions" (different settings, data sizes, and random seeds) to calculate two numbers:

Effectiveness ( $\mu$ ): How good is it on average?
Stability ( $\sigma$ ): How reliable is it? Does it crash if you change the settings slightly?

4. The Big Surprises (The "Paradoxes")

After running thousands of these "Lego" experiments, they found three things that challenge what everyone thought was true:

A. The "Identity" Paradox (The Simplest Tool Wins)

The Claim: Everyone thought you needed a super-complex "brain" (like a Transformer) to understand time series.
The Reality: Once you arrange the ingredients correctly (the Embedding), a parameter-free "Identity" encoder (which is basically just a mirror that passes the data through without changing it) often performs just as well, or even better, than the complex brains.
The Analogy: It turns out that if you slice your vegetables perfectly (good data view), you don't need a fancy robotic chef to cook them; a simple spoon works just fine. The complexity of the "chef" was mostly just adding noise.

B. The "Cycle" Secret (Preparation Matters More than Cooking)

The Claim: Adding complex math to the cooking process helps.
The Reality: It's better to add structural priors during the preparation phase. For example, if you know the data has a daily cycle (like traffic or electricity), explicitly telling the model "this repeats every 24 hours" works better than trying to teach a complex model to figure it out from scratch.
The Analogy: It's better to pre-chop the onions for the cook than to give the cook a whole onion and a very complex knife and hope they do a good job.

C. The Frequency Illusion

The Claim: Looking at data through a "frequency lens" (like a prism splitting light) is always more stable.
The Reality: While looking at frequencies sometimes makes the average result slightly better, it doesn't make the model more stable. It's a trade-off, not a magic bullet.

5. The Conclusion: Stop Chasing the "Best" Score

The paper concludes that the field has been chasing "State-of-the-Art" (SOTA) titles based on fragile, lucky results.

The New Rule: If you want to claim your new model is better, you can't just show one high score. You must prove that your new component works better on average and is more stable than a simple, parameter-free baseline (the "Identity" model) across many different conditions.

In short: Don't build a bigger, more complex engine if a simple mirror works just as well. And stop bragging about your one lucky shot; show us your average performance over a whole season.

Technical Summary: CombinationTS

1. Problem Statement

Recent advancements in Time-Series Forecasting (TSF) have led to a rapid increase in architectural complexity, yet reported State-of-the-Art (SOTA) gains are often statistically fragile or misattributed. The field faces two fundamental methodological deficits:

The Attribution Gap: Existing evaluations treat models as indivisible "black boxes," conflating the contribution of the data view (Embedding) with that of the modeling module (Encoder). It remains unclear whether performance improvements stem from how data is presented or how it is processed.
The Benchmarking Crisis: Standard protocols rely on point estimates (e.g., a single random seed or a fixed configuration), falling into the "Fairness Trap" (underestimating potential by using suboptimal settings) or the "Best Trap" (cherry-picking peak results). These point estimates fail to capture the expected performance ( $\mu$ ) and stability ( $\sigma$ ) of a design under the epistemic uncertainty of real-world hyperparameters.

Consequently, it is difficult to determine if modern TSF gains reflect genuine architectural reasoning or are merely driven by improved data views and favorable evaluation conditions.

2. Methodology: The CombinationTS Framework

To shift from "Model Selection" to "Modular Attribution," the authors propose CombinationTS, a self-contained probabilistic evaluation framework.

2.1 Modular Decomposition

The framework decomposes the forecasting pipeline into five orthogonal, interchangeable stages, allowing for the isolation and recombination of architectural components:

Input Transformation ( $T_{in}$ ): Injects structural priors (e.g., normalization, detrending, decomposition) into the raw signal.
Embedding ( $E$ ): Defines the data view (tokenization strategy), such as point-wise, patch-wise, or variate-wise embeddings.
Encoder ( $\Phi$ ): The modeling module (e.g., Transformer, MLP, Identity) responsible for capturing interactions.
Decoder ( $D$ ): Projects latent features to the forecasting horizon.
Output Transformation ( $T_{out}^{-1}$ ): The inverse operation of $T_{in}$ (e.g., re-adding trends).

2.2 Probabilistic Evaluation Protocol

Grounded in the principles of Evaluatology, the framework treats architectural analysis as a signal–noise disentanglement problem.

Evaluated Object (EO): The specific module variant under investigation (e.g., a specific Encoder).
Evaluation Condition (EC): A stochastic environment encompassing complementary modules, training hyperparameters, and sources of randomness.
Shared Condition Space ( $\Omega$ ): Instead of fixing a single configuration, the framework samples a broad distribution of conditions.
Stratified Monte Carlo Estimation: By performing paired evaluations across $K$ $K$ shared samples from $\Omega$ $Ω$ , the framework quantifies two distributional statistics for each component:
- Marginalized Performance ( $\hat{\mu}$ ): The expected performance across conditions.
- Stability ( $\hat{\sigma}$ ): The sensitivity of performance to condition variations.

This approach moves beyond fragile point estimates to robust distributional statistics, enabling fair comparisons of architectural necessity.

3. Key Contributions and Empirical Findings

Through large-scale paired evaluations across six benchmarks (Weather, Electricity, and four ETT subsets), the authors present three empirical findings that challenge prevailing intuitions:

3.1 The "Identity" Paradox

Finding: Once the data view (Embedding) is well-designed, a parameter-free Identity Encoder frequently achieves comparable or even superior effectiveness ( $\hat{\mu}$ ) and stability ( $\hat{\sigma}$ ) compared to complex backbones like Transformers and Mixers.
Implication: On standard benchmarks, deep modeling modules often act as over-parameterized noise generators rather than essential feature extractors. The primary performance bottleneck lies in data view formulation, not the modeling module.
Evidence: Statistical significance tests (Mann–Whitney U) confirm Identity encoders achieve significantly lower MSE than Transformers and MLPs across aggregated datasets.

3.2 Conditional Utility of Structural Priors

Finding: Input Transformations are not universally superior. Explicit periodic inductive biases (e.g., learnable cycles) provide consistent gains. However, naive decomposition strategies (e.g., multi-scale splitting) often degrade performance unless coupled with sophisticated cross-component interaction.
Implication: Architectural synergy is necessary; decomposition alone is insufficient without mechanisms to enforce deep mixing between components.

3.3 Effectiveness Gain of Spectral Modeling

Finding: Frequency-domain modeling (e.g., Fourier/Wavelet) yields higher effectiveness ( $\hat{\mu}$ ) than temporal baselines but provides no significant reduction in instability ( $\sigma$ ).
Implication: The spectral advantage stems from better signal representation rather than increased robustness to hyperparameter variation. Furthermore, spectral processing fails to surpass the parameter-free Identity baseline on certain datasets, suggesting its gains over attention mechanisms are merely due to reduced signal distortion.

4. Significance and Claims

The paper claims to offer a standardized protocol for architectural auditing rather than proposing a new predictive architecture. Its significance lies in:

Reframing the Burden of Proof: The authors propose a new evaluation principle where future innovations must demonstrate significant gains in Marginalized Performance ( $\hat{\mu}$ ) and Stability ( $\hat{\sigma}$ ) over a robust Identity baseline. Merely reporting fragile leaderboard peaks is deemed scientifically untenable.
Distinguishing Reasoning from Illusion: CombinationTS serves as a lens to distinguish genuine reasoning capabilities from the "illusion of progress" driven by data view optimization or favorable hyperparameter alignment.
Open Science: The authors release a modular library and full experimental logs to facilitate reproducible, component-level research, addressing the "reproducibility crisis" in time-series forecasting.

The work advocates for a shift in the TSF community from chasing SOTA on fixed leaderboards to rigorously auditing the specific mechanisms that drive performance, ensuring that architectural complexity is justified by genuine, robust improvements.