⚡ electrical engineering

Benchmarking M-LTSF: Frequency and Noise-Based Evaluation of Multivariate Long Time Series Forecasting Models

This paper introduces a simulation-based evaluation framework using parameterizable synthetic datasets to systematically benchmark multivariate long-term time series forecasting models, revealing how their performance varies based on signal frequency patterns, noise types, and signal-to-noise ratios to provide concrete guidance for model selection.

Original authors: Nick Janssen, Melanie Schaller, Bodo Rosenhahn

Published 2026-03-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nick Janssen, Melanie Schaller, Bodo Rosenhahn

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a group of four different students how to predict the weather for the next week.

In the real world, you'd give them a notebook full of messy, confusing data: "It rained yesterday, but the wind was weird, and the temperature sensor was broken." The problem is, you don't know exactly what's wrong with the data. Is the rain real? Is the broken sensor the problem? Because the data is messy, you can't tell if a student is actually good at predicting weather or just lucky.

This paper is about building a "Weather Simulator" to test these students fairly.

Instead of using messy real-world data, the authors created a computer program that generates perfect, fake weather data. They can control every single variable:

The Signal: They can make the weather follow a smooth sine wave (like a gentle breeze), a jagged sawtooth wave (like a sudden power surge), or a square wave (like a light switch flipping on and off).
The Noise: They can add different types of "static" to the data. Sometimes it's random static (White Noise), sometimes it's a slow drift (Brownian Noise), or even sudden spikes (Impulse Noise).
The Difficulty: They can make the signal very clear (High Signal-to-Noise Ratio) or make it almost impossible to hear the signal over the noise (Low Signal-to-Noise Ratio).

The Four Students (The Models)

The authors tested four popular "AI students" (deep learning models) to see who is best at this task:

R-Linear: The "Simple Calculator." It's very fast and efficient but a bit dumb. It tries to draw a straight line through the data. It works okay for simple, smooth patterns but gets confused by complex, multi-variable weather.
Autoformer: The "Deconstructionist." It breaks the weather down into "Trend" (long-term changes) and "Seasonality" (repeating patterns). It's great at handling sharp, jagged patterns (like sawtooth waves) but can be finicky and inconsistent.
iTransformer: The "Pattern Spotter." It uses a complex attention mechanism to look at how different variables relate to each other. It loves smooth, gentle waves (sine waves) but struggles with sharp, sudden changes.
S-Mamba: The "State-Space Wizard." It's a newer, very efficient model that remembers past states well. It is the overall champion, performing best on jagged patterns and handling most noise well, though it gets confused if the data has a weird long-term drift.

The Big Discoveries

1. The "Window" Problem
Imagine trying to guess the next step in a dance routine. If you only look at the last 2 seconds of a dance that takes 10 seconds to complete, you can't guess the next move.
The paper found that if the "lookback window" (how much past data the model sees) is too short to see a full cycle of the pattern, all models fail miserably. They start thinking a repeating pattern is a long-term trend, and their predictions go off the rails.

2. The Noise Trap
Real-world data is noisy. The study showed that:

White Noise (random static) hurts everyone, but the models get better as the signal gets louder.
Brownian Noise (slow, drifting errors) is the worst enemy. It confuses the models the most, especially when the signal is weak.
Specific Weaknesses: S-Mamba gets confused by "Trend Noise" (if the data slowly drifts up or down). iTransformer gets confused by "Seasonal Noise" (if the repeating pattern itself is messy).

3. The Frequency Secret
The authors looked at the data through a "spectral lens" (like looking at a rainbow of frequencies). They found that even the best models struggle to perfectly recreate the "sound" of the data. They often add fake frequencies that shouldn't be there. However, S-Mamba and iTransformer were the best at keeping the original "sound" of the data clean.

The Takeaway: How to Choose Your Student

The paper gives you a cheat sheet for picking the right model based on your data:

Need speed and have simple data? Pick R-Linear.
Have sharp, jagged patterns (like power grids)? Pick Autoformer or S-Mamba.
Have smooth, gentle waves (like temperature)? Pick iTransformer.
Have a complex, noisy, multi-variable mess? S-Mamba is generally the safest bet, but be careful if your data has weird long-term drifts.

In summary: This paper built a controlled "lab" to test AI forecasters. It proved that you can't just throw data at a model and hope for the best. You need to know your data's shape (smooth vs. jagged) and its noise level to pick the right tool. Otherwise, you might be hiring a calculator to do a wizard's job!

1. Problem Statement

Multivariate Long Time Series Forecasting (M-LTSF) is critical for domains like energy, meteorology, and traffic modeling. However, evaluating the robustness of deep learning models in this field faces significant challenges:

Ambiguity in Real-World Data: Standard benchmark datasets (e.g., ETT, Traffic) contain unknown combinations of signal components, noise types, and temporal patterns. This makes it impossible to isolate whether a model's performance is due to learning true underlying signals or overfitting to specific noise characteristics.
Lack of Controlled Variables: Existing evaluations rely on metrics like Mean Squared Error (MSE) without controlling for signal-to-noise ratios (SNR), frequency bands, or specific noise types (e.g., Brownian vs. White noise).
Limited Insight: Current benchmarks fail to reveal why models fail or succeed under specific conditions, such as when the lookback window is smaller than the seasonal cycle or under high-noise regimes.

2. Methodology

The authors propose a simulation-based evaluation framework using a parameterizable synthetic dataset generator to create controlled, reproducible M-LTSF scenarios.

A. Synthetic Dataset Framework

The framework generates multivariate time series by hierarchically combining predefined signal and noise components:

Signal Components:
- Trends: Modeled using monomial functions (linear, non-linear).
- Seasonality: Three distinct waveform types are generated: Sinusoidal (smooth), Smoothed Sawtooth (gradual rise, sharp drop), and Smoothed Square (flattened oscillations).
Noise Components:
- Signal-Independent: White noise (Gaussian), Brownian noise (cumulative sum of white noise), and Impulse noise (sporadic spikes).
- Signal-Dependent: Trend noise (amplitude scales with trend magnitude) and Seasonal noise (amplitude scales with seasonal magnitude).
Generation Process: Components are weighted and aggregated. The final time series is constructed by scaling the aggregated signal and noise to achieve a specific Signal-to-Noise Ratio (SNR).
Experimental Design: The study systematically varies:
- Signal Types: Sine, Square, Sawtooth.
- Frequency Bands: Seven distinct index ranges (from "very low" to "very high") relative to the lookback window.
- Noise Levels: SNR values ranging from $\infty$ (noise-free) to $1$ (signal and noise equal magnitude).
- Variate Count: $V=800$ variates to simulate complex multivariate dependencies.

B. Evaluated Models

Four representative M-LTSF architectures were benchmarked:

S-Mamba: A State Space Model (SSM) using bidirectional Mamba blocks for efficient long-sequence processing.
iTransformer: An attention-based model that applies self-attention across variates (treating features as tokens) rather than time steps.
R-Linear: A linear method using channel-independent modeling with instance normalization.
Autoformer: A decomposition-based model separating trend and seasonal components using autocorrelation.

C. Evaluation Protocol

Training: Models are trained on noisy data.
Testing: Models are evaluated on the corresponding noise-free ground truth signal. This tests the model's ability to recover the latent signal structure rather than memorizing noise.
Metrics: Primary metric is MSE, supplemented by Spectral Analysis (FFT) to evaluate frequency domain reconstruction.

3. Key Contributions

Parameterizable Synthetic Framework: A hierarchical generation approach allowing precise control over signal characteristics, noise types, and SNR, providing a reproducible benchmark for systematic evaluation.
Granular Model Characterization: Identification of specific architectural strengths and weaknesses regarding signal types (e.g., sawtooth vs. sine) and noise sensitivities (e.g., trend vs. seasonal noise).
Frequency and Spectral Insights: Demonstration that model performance is heavily dependent on whether the lookback window captures full seasonal cycles and an analysis of how well models reconstruct frequency spectra.
Open-Source Codebase: Full release of the synthetic dataset generation framework and experimental configurations.

4. Key Results

A. Impact of Frequency and Lookback Windows

Critical Threshold: When the lookback window ( $T=96$ ) is smaller than the shortest period in the signal, performance degrades sharply for all models.
Performance Gap: Average MSE increased by ~0.575 when moving from optimal frequency ranges (where cycles are fully captured) to suboptimal low-frequency ranges.
Signal Type Preferences:
- S-Mamba & Autoformer: Perform best on Sawtooth patterns. S-Mamba's time-varying state transitions excel at modeling sharp transitions; Autoformer's decomposition isolates sawtooth waves effectively.
- iTransformer & R-Linear: Perform best on Sinusoidal patterns. The self-attention mechanism in iTransformer captures gradual phase relationships better than sharp transitions.

B. Noise Robustness

General Trend: Performance deteriorates as SNR decreases. Brownian noise causes the most severe degradation at low SNR ($SNR=1$) due to its non-stationary nature.
Architectural Vulnerabilities:
- S-Mamba: Highly sensitive to Trend Noise (noise correlated with long-term drift).
- iTransformer: More vulnerable to Seasonal Noise and Impulse noise.
- R-Linear: Reaches performance saturation quickly; struggles with complex cross-variate dependencies in high-dimensional settings ( $V=800$ ).
- Autoformer: Shows high variability but is robust against White, Trend, and Seasonal noise when processing Sawtooth signals.
Persistence of Preferences: Signal type preferences (e.g., S-Mamba liking sawtooth) persist even under noisy conditions, indicating these are fundamental architectural traits.

C. Spectral Analysis

Universal Failure: None of the models achieved a "clean" spectral representation (perfect frequency reconstruction) even in noise-free conditions, suggesting deep learning optimizers converge to local minima introducing spurious frequencies.
Best Performers: S-Mamba and iTransformer demonstrated superior frequency reconstruction capabilities compared to Autoformer and R-Linear under clean conditions.
Correlation: Spectral error correlates strongly with MSE performance; models that better preserve the spectrum generally yield lower MSE.

D. Efficiency vs. Performance

R-Linear: Fastest inference but poor performance on complex multivariate data.
S-Mamba & iTransformer: Good performance but slower inference speeds.
Autoformer: Fastest training time but largest parameter count and high sensitivity to hyperparameters.

5. Significance and Implications

Model Selection Guidance: The paper provides a decision matrix for practitioners. For example, if data exhibits sawtooth patterns and trend noise, S-Mamba might be risky, whereas iTransformer could be preferred. If inference speed is paramount and data is simple, R-Linear is viable.
Beyond Real-World Benchmarks: The study proves that real-world benchmarks often mask specific failure modes. Controlled synthetic evaluation reveals that "state-of-the-art" performance on standard datasets may not translate to robustness under specific noise or frequency conditions.
Future Research Directions: Highlights the need for architectures that can better handle non-stationary noise (like Brownian noise) and improve spectral reconstruction to avoid spurious frequency components. It suggests that future models should be evaluated not just on MSE, but on their ability to reconstruct the underlying frequency spectrum.

In conclusion, this work shifts the paradigm of M-LTSF evaluation from "black-box" benchmarking on ambiguous real-world data to a white-box, systematic analysis of how specific signal and noise characteristics interact with model architectures.