Multi-Scale Convolution with Optimal Transport Attention Effect on Multivariate Time Series

Technical Summary: Multi-Scale Convolution with Optimal Transport Attention on Multivariate Time Series

Problem Statement
Multivariate Time Series (MTS) forecasting faces significant challenges in capturing multi-granularity structural patterns while effectively suppressing noise. While Transformer-based models have become dominant in sequence modeling, their application to time series often relies on embedding multiple variates at the same timestamp into channels, focusing primarily on temporal dependencies. Recent advancements, such as iTransformer, introduced an "inverted embedding" paradigm where each variate's entire time series is treated as a single token to better capture cross-variate dependencies. However, the authors identify a critical limitation in this approach: by compressing a variate's temporal information into a fixed-dimensional token, fine-grained local temporal patterns (e.g., short-term fluctuations, periodicities) are lost. Furthermore, standard attention mechanisms are susceptible to outliers and noise, which can lead to skewed attention distributions and suboptimal predictions.

Methodology: MSC-OT
To address these limitations, the paper proposes MSC-OT (Multi-Scale Convolution with Optimal Transport Attention), a novel architecture designed to enhance the inverted embedding framework. The method consists of three core components:

1. Multi-Scale Convolution Enhancement:

   • Objective: To recover and model local structural patterns within the attention score matrix that are lost during the inverted embedding process.
   • Mechanism: The model applies parallel 2D convolutions with kernel sizes of $3 \times 3$ and $5 \times 5$ directly to the attention score matrix ($S \in \mathbb{R}^{B \times H \times N \times N}$).
   • Function: These convolutions capture local structural patterns at different granularities within the variate-interaction space. The resulting multi-scale features are aggregated via simple averaging to ensure numerical stability and preserve complementary information.

2. Sinkhorn Optimal Transport (OT) Regularization:

   • Objective: To formulate attention computation as an optimal transport problem to ensure balanced information flow and robustness against noise.
   • Mechanism: The attention score matrix is treated as a cost matrix for an entropy-regularized optimal transport problem. The Sinkhorn algorithm is employed to iteratively normalize rows and columns, producing a doubly-stochastic attention distribution.
   • Function: The entropy regularization term ($\lambda H(P)$) naturally smooths attention weights, preventing any single variate (even one with an anomalously high score) from dominating the attention distribution. This effectively suppresses the impact of outliers and measurement noise common in real-world MTS data.

3. Adaptive Fusion Strategy:

   • Objective: To dynamically combine the base attention scores, convolution-enhanced scores, and OT-regularized scores.
   • Mechanism: The final attention score is computed as a weighted sum: $S_{final} = \alpha \cdot S + \beta \cdot S_{conv} + \gamma \cdot P$.
   • Function: The weights ($\alpha, \beta, \gamma$) are learnable parameters initialized to 0.7, 0.2, and 0.1, respectively. They are normalized via softmax to ensure stable training. This allows the model to adaptively determine the optimal contribution of each component based on the specific characteristics of the dataset.

Key Contributions

• Enhanced Inverted Embedding: The paper proposes a method to retain the benefits of iTransformer's cross-variate modeling while compensating for the loss of fine-grained temporal dependencies through multi-scale convolution on attention scores.
• Noise-Robust Attention: By integrating Sinkhorn Optimal Transport, the model introduces a mechanism that inherently suppresses noise and prevents outlier dominance, addressing a key weakness in standard softmax-based attention for MTS.
• Synergistic Architecture: The work demonstrates that multi-scale convolution (local pattern capture) and OT regularization (global balance and noise suppression) are complementary. The adaptive fusion strategy allows these components to work synergistically rather than independently.
• State-of-the-Art Performance: The authors provide extensive empirical validation showing that MSC-OT outperforms recent state-of-the-art models (including iTransformer, PatchTST, Crossformer, and TimesNet) across multiple benchmark datasets.

Experimental Results
The authors evaluated MSC-OT on five widely used benchmark datasets: ECL (Electricity), ETT (Electricity Transformer Temperature), Exchange-Rate, Traffic, and Weather.

• Performance: MSC-OT achieved the best Mean Squared Error (MSE) results on ECL, Traffic, and Weather, and the best Mean Absolute Error (MAE) on four out of five datasets.
• Comparative Gains: The paper claims significant improvements over specific baselines, citing improvements of 27% over Crossformer, 35% over TimesNet, and 19% over Fedformer in specific contexts.
• Ablation Studies: Experiments confirmed that removing either the multi-scale convolution or the OT regularization leads to performance degradation. The studies also showed that the adaptive fusion weights learn to balance the components effectively (e.g., in ECL, the convolution and OT components combined accounted for nearly 60% of the weight, indicating their critical role).
• Hyperparameters: The optimal entropy regularization coefficient was found to be $\lambda=5.0$, and two Sinkhorn iterations were deemed sufficient for convergence without excessive computational cost.

Significance and Claims
The paper positions MSC-OT as a practical and effective solution for real-world multivariate time series forecasting. The authors claim that the method successfully optimizes the attention mechanism by addressing the specific trade-offs between capturing local temporal structures and maintaining robust cross-variate modeling in the presence of noise. By leveraging optimal transport, the approach establishes a new methodological path for attention optimization that is less sensitive to outliers and better suited for the complex, multi-scale nature of real-world time series data. The code is made available to facilitate further research and application.