🤖 machine learning

Beyond Holistic Models: Systematic Component-level Benchmarking of Deep Multivariate Time-Series Forecasting

This paper introduces TSCOMP, the first large-scale benchmark that systematically deconstructs deep multivariate time-series forecasting methods into fine-grained components to demonstrate that data-driven, automated component selection consistently outperforms manually designed complex holistic architectures.

Original authors: Shuang Liang, Chaochuan Hou, Xu Yao, Shiping Wang, Hailiang Huang, Songqiao Han, Minqi Jiang

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shuang Liang, Chaochuan Hou, Xu Yao, Shiping Wang, Hailiang Huang, Songqiao Han, Minqi Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to bake the perfect cake. For years, bakers (data scientists) have been obsessed with building massive, complex ovens (deep learning models) hoping that the sheer size and complexity of the machine would guarantee a delicious result. They would buy a new oven, bake a cake, taste it, and say, "This oven is the best!" But they never really stopped to ask: Was it the oven? Was it the flour? Was it the temperature?

The paper you provided, "Beyond Holistic Models," argues that we've been looking at the wrong thing. Instead of judging the whole oven, we need to take it apart, piece by piece, to see which specific ingredient or setting actually makes the cake taste good.

Here is the breakdown of their approach, using simple analogies:

1. The Problem: The "Black Box" Mystery

In the world of forecasting (predicting the future based on past data, like stock prices or weather), researchers have been building huge, complex "black box" models. They treat these models as indivisible units. If a model works well, they assume the whole thing is great. If it fails, they assume the whole thing is bad.

The Analogy: It's like judging a car solely by its brand logo. You might say, "This Ferrari is fast!" but you don't know if it's the engine, the tires, the aerodynamics, or the driver that's doing the work. You can't improve the car if you don't know which part to fix.

2. The Solution: TSCOMP (The "Lego" Approach)

The authors created a new system called TSCOMP. Instead of treating models as black boxes, they broke them down into Lego bricks.

They realized that every forecasting model goes through four main stages, like an assembly line:

Preparation (Preprocessing): Cleaning the data (like washing vegetables before cooking).
Encoding: Turning data into a format the computer understands (like chopping vegetables into specific shapes).
Architecture (The Brain): The actual math model doing the thinking (the chef).
Optimization: How the model learns and improves (the tasting and adjusting process).

They took apart dozens of the world's most advanced models and sorted their "bricks" into a giant pool. They found 49 different types of bricks (like specific ways to wash data, specific types of attention mechanisms, or different loss functions).

3. The Experiment: The "Recipe Tester"

Instead of baking one giant cake, they ran a massive experiment. They used a special statistical method (called Orthogonal Design) to mix and match these bricks.

The Analogy: Imagine a chef who wants to find the perfect soup. Instead of making 1,000 different soups from scratch, they take a base broth and swap out one ingredient at a time.

"What happens if I use salt instead of soy sauce?"
"What if I use a slow-cook method instead of a boil?"
"Does this work better with beef or chicken?"

They tested over 20,000 different combinations across 13 different real-world datasets (like electricity usage, traffic, and weather). This created a massive "Performance Corpus"—a giant database of what works and what doesn't.

4. The Big Discoveries

Here is what they found when they looked at the data:

The "Prep" is More Important than the "Chef":
Surprisingly, the most important part of the process wasn't the complex brain of the model (the architecture). It was the Preprocessing (cleaning and normalizing the data).
- Analogy: It turns out that washing and chopping your vegetables perfectly matters more than whether you use a Michelin-star chef or a home cook. If your ingredients are bad, the best chef in the world can't save the dish.
- Result: The way they handled the data explained 66% of the success, while the actual model architecture only explained about 8%.
One Size Does NOT Fit All:
A brick that works great for one type of data might be terrible for another.
- Analogy: A tire that is perfect for a race car on a dry track might be useless for a truck in the snow. You can't just copy-paste a "winning" model from one dataset to another.
Simple is Often Better:
They found that you don't need a massive, complex "Transformer" or "Large Language Model" to get great results. Often, a simple MLP (a basic neural network) combined with the right preprocessing bricks performed better than the most complex models.
- Analogy: You don't need a nuclear-powered toaster to make toast. Sometimes, a simple toaster with the right bread and butter works best.

5. The Magic Tool: The "Smart Assistant"

Because they built this massive database of 20,000+ experiments, they trained a Meta-Predictor.

The Analogy: Think of this as a smart shopping assistant.

Before: You had a new dataset (a new recipe). You had to spend weeks trying different ovens, flours, and temperatures to find what worked.
Now: You show the assistant your new dataset. The assistant looks at its giant database of "what worked for similar ingredients" and instantly says: "For this specific data, use Brick A for cleaning, Brick B for encoding, and Brick C for the brain. Don't use the others."

This allows for Zero-Shot Construction. You don't need to train a new model from scratch. The system instantly builds the perfect, lightweight model for your specific needs based on what it learned from the past.

The Bottom Line

The paper proves that systematic component selection beats complex manual design.

Instead of trying to build a bigger, scarier, more complex "monster model," we should focus on picking the right, simple building blocks for the specific job at hand. The authors released all their code and their giant database of results so anyone can use this "smart assistant" to build better forecasting models without needing to be a PhD-level expert.

In short: Stop trying to build a better engine; start learning how to wash the car better. It turns out that's what actually makes the car go faster.

Technical Summary: Beyond Holistic Models: Systematic Component-level Benchmarking of Deep Multivariate Time-Series Forecasting

Problem Statement

Multivariate time series forecasting (MTSF) has seen a proliferation of deep learning models, ranging from RNNs and Transformers to Large Language Models (LLMs) and Time Series Foundation Models (TSFMs). However, existing research predominantly focuses on developing complex, holistic models and evaluating them as indivisible "black boxes." This approach obscures the specific contributions of internal mechanisms, making it difficult to determine which design choices (e.g., normalization, tokenization, attention mechanisms) are genuinely effective versus those that are merely artifacts of specific model architectures. Furthermore, current benchmarks often fail to analyze the multi-level hierarchy of the MTSF pipeline, hindering the combination of effective components into superior solutions and limiting the ability to adapt models to specific data characteristics.

Methodology

The authors propose TSCOMP, a comprehensive framework designed to systematically deconstruct and benchmark deep MTSF methods at a granular, component level. The methodology consists of three primary phases:

1. Hierarchical Deconstruction

TSCOMP breaks down the standard MTSF workflow into a structured design space across three levels:

Pipeline Level: Four stages: Series Preprocessing, Series Encoding, Network Architecture, and Network Optimization.
Dimension Level: 11 specific component dimensions (e.g., Series Normalization, Channel Independence, Tokenization, Attention Mechanisms, Loss Functions).
Component Level: 49 concrete implementations extracted from State-of-the-Art (SOTA) models (e.g., RevIN, PatchTST, iTransformer, GPT4TS, TimeMixer).

2. Constrained Orthogonal Experimental Design

To manage the combinatorial explosion of potential configurations (over $10^6$ theoretical combinations), TSCOMP employs a constrained orthogonal experimental protocol.

Constraint Filtering: Invalid architectural combinations (e.g., pairing inverted encoding with channel independence) are strictly excluded to ensure soundness.
Pairwise Coverage: A greedy algorithm selects a subset of approximately 136 models per horizon that ensures every valid pairwise interaction between components is covered. This balances interaction analysis with computational tractability.
Evaluation: The selected configurations are evaluated across 13 diverse real-world datasets (including ETT, Traffic, Weather, and financial datasets) using standardized metrics (MSE, MAE, SMAPE, MASE, OWA).

3. Statistical Analysis and Automated Construction

Multi-View Analysis: The authors utilize Generalized Linear Mixed Models (GLMM) to isolate marginal contributions, ANOVA to quantify variance explained by dimensions, and Cohen's $d$ to measure effect sizes across data characteristics.
Automated Model Construction: The resulting performance corpus (over 20,000 evaluations) trains a Meta-Predictor. This predictor maps dataset meta-features (extracted via TabPFN) and component embeddings to predicted performance rankings. This enables zero-shot automated model construction, where optimal components are selected for new datasets without extensive retraining or experimentation.

Key Contributions

Comprehensive Benchmark via Hierarchical Deconstruction: TSCOMP is the first large-scale benchmark to systematically deconstruct deep MTSF methods into atomic components, moving beyond holistic model evaluation to isolate core mechanisms driving performance.
Multi-View Analysis and Insights: The study provides rigorous experimental evidence regarding component efficacy across different backbones (MLP, RNN, Transformer, LLM, TSFM), diverse data domains, and specific data characteristics (e.g., stationarity, correlation). It reveals complex interaction effects and validates or refutes community claims.
Open-Sourced Corpus and Automated Construction: The authors release a fine-grained performance corpus and demonstrate its utility in automating model design. The corpus-driven approach allows for the adaptive assembly of optimal pipelines tailored to specific datasets, achieving results that surpass manually designed SOTA architectures.

Experimental Results

Component Dominance: The analysis reveals that Series Preprocessing (specifically Series Normalization) is the most influential phase, accounting for 66.6% of the total performance variance. This significantly outweighs the contribution of Network Architecture (8.0%) and Network Optimization (7.1%).
Architecture-Specific Sensitivities:
- MLPs: Rely heavily on Series Preprocessing and benefit from Channel Independence.
- RNNs: Show disproportionate gains from Channel Independence to prevent error propagation but suffer from standard decomposition methods.
- Transformers: Performance is heavily dependent on Loss Function design and Tokenization strategies (e.g., Orthogonal Encoding), while Series Decomposition often degrades performance.
- LLMs/TSFMs: Exhibit unique sensitivities; for instance, TSFMs are highly sensitive to Tokenization, where Patching (used in pre-training) can degrade performance if not decoupled from Channel Independence.
Data Characteristics: Component effectiveness is context-dependent. For example, Channel Independence degrades performance on highly correlated datasets, and Stationary Normalization is crucial for datasets with high distribution shifts.
Automated Construction Performance: The zero-shot automated construction using the TSCOMP corpus consistently outperforms SOTA deep MTSF models, AutoML baselines (e.g., AutoGluon, TimeFuse), and Large Time-Series Models. Notably, the system achieves these results using simple MLP-based configurations, demonstrating that precise component selection is more critical than architectural complexity. In long-term forecasting, TSCOMP achieved SOTA results on 10 out of 14 tasks.

Significance and Claims

The paper claims that the field of MTSF has reached a point where increasing model complexity yields diminishing returns compared to the systematic selection of appropriate components. The authors assert that:

Systematic Component Selection > Complex Architectures: Manually designed complex architectures are often suboptimal compared to pipelines constructed by adaptively selecting components based on data characteristics.
Preprocessing is Foundational: Handling non-stationarity and distribution shifts through proper preprocessing is more fundamental to forecasting success than the choice of the underlying neural network backbone.
Zero-Shot Adaptation: A data-driven, corpus-based approach can effectively guide model construction for new datasets without the need for exhaustive local experimentation or heavy fine-tuning of large models.

The work advocates for a paradigm shift from "holistic model design" to "granular component engineering," providing a robust foundation for future automated and adaptive time series forecasting systems.