Technical Summary: TFGformer

Problem Statement

Multivariate time series forecasting (MTSF) requires models to capture complex temporal dynamics and intricate inter-variable dependencies. However, existing approaches face two primary challenges:

1. Noisy Correlations: In practice, many variables exhibit weak or spurious correlations that introduce interference. Current models often struggle to distinguish meaningful relationships from noise, particularly when dependencies are sparse or evolve over time.
2. Underutilized Covariates: Crucial contextual information, such as calendar features, weather conditions, and external events, is frequently underutilized. Most existing models incorporate these via simple concatenation or by treating them as extra channels, which fails to account for their inherent structure and often introduces noise rather than enhancing prediction accuracy.

Methodology

The paper proposes TFGformer, a unified framework integrating Time-Frequency Graph (TFG) learning with Covariate-Aware Representation Fusion. The architecture consists of three core components:

1. Time-Frequency Graph (TFG) Module

This module is designed to learn dynamic, sparse inter-variable correlations by operating in the joint time-frequency domain.

• Spectral Feature Extraction: It employs the Short-Time Fourier Transform (STFT) on each variable to generate magnitude spectrograms, capturing complementary temporal and spectral patterns.
• Adaptive Correlation Learning: Instead of assuming static associations, the module learns pairwise relational probabilities using a learnable Mahalanobis distance metric. This metric captures the covariance structure of the time-frequency feature space, moving beyond simple Euclidean distances.
• Sparse Graph Construction: To derive a discrete, sparse graph suitable for attention mechanisms, the continuous probability matrix is converted into a binary adjacency matrix using Gumbel-Softmax sampling. This process filters out irrelevant dependencies, ensuring the model focuses only on significant inter-variable paths.
• Attention Masking: The resulting binary channel mask is combined with a causal time mask via a Kronecker product to form the final attention mask, guiding the Transformer backbone to model relevant dependencies while maintaining causality.

2. Covariate Feature Fusion (CFM) Module

To effectively integrate auxiliary information without losing original data characteristics, the model utilizes an MLP-based fusion strategy divided into two stages:

• Input Covariate Fusion (I-CFM): Integrates historical covariates with the original time series patch embeddings before entering the Transformer. This allows the model to exploit prior trend knowledge.
• Output Covariate Fusion (O-CFM): Incorporates future covariates after Transformer processing to refine hidden representations. This module includes a preprocessing step (forward-filling for continuous missing values, zero-padding for categorical) to ensure robustness against missing future data.
• Residual Connections: Both fusion steps employ residual connections to ensure the original data characteristics are preserved while modulating the representation with contextual signals.

3. Transformer Backbone

The enriched sequence, containing relational information from the TFG and covariate-aware features from the CFM, is processed by a standard Transformer backbone. The TFG-generated attention mask is applied during the self-attention computation to explicitly enforce the learned sparse dependencies.

Key Contributions

The authors identify three main contributions:

1. Novel Time-Frequency Graph Learning: A module leveraging STFT and Mahalanobis distance to capture dynamic, nonlinear dependencies. It adaptively constructs an attention mask in the joint time-frequency domain, enabling more accurate modeling of complex interactions than static or purely time-domain approaches.
2. MLP-Based Covariate Fusion: A dual-stage fusion mechanism (I-CFM and O-CFM) that integrates both historical and future covariates via residual connections. This design allows the model to fully exploit prior trend knowledge and external dynamic context without the noise associated with simple concatenation.
3. Comprehensive Empirical Validation: Extensive experiments on seven benchmark datasets across electricity, traffic, and weather domains demonstrate the framework's effectiveness.

Experimental Results

The model was evaluated on seven widely used benchmark datasets (comprising four subsets of ETT, plus ECL, Traffic, and Weather) across multiple forecasting horizons (96, 192, 336, 720 time steps).

• Performance: TFGformer achieved state-of-the-art (SOTA) results on 6 out of 7 datasets (ETTh2, ETTm1, ETTm2, Weather, Electricity, and Traffic) in terms of Mean Squared Error (MSE) and Mean Absolute Error (MAE). On the ETTh1 dataset, TFGformer remained highly competitive but did not achieve the best overall performance, as FEDformer demonstrated lower MSE at specific horizons (e.g., 96, 192, and 336).
• Comparative Gains: On average across all datasets, TFGformer reduced MSE by 3.6% compared to iTransformer, 6.4% compared to PatchTST, 5.2% compared to TiDE, and 16.4% compared to FEDformer.
• Ablation Studies: Removing the TFG module resulted in a 6.0% average MSE increase, while removing the CFM module caused a 3.0% increase, confirming the necessity of both components.
• Interpretability: Visualization of the learned probability matrix on the ETTh1 dataset showed that the model successfully assigned higher probabilities to highly correlated variable pairs while suppressing noisy connections.

Significance and Claims

The paper claims that TFGformer addresses the critical limitations of existing Transformer-based models by:

• Selective Interaction Modeling: Moving beyond the assumption that all variable pairs share equal correlation strength. By learning sparse, dynamic graphs in the time-frequency domain, the model effectively suppresses interference from noisy correlations.
• Effective Contextual Integration: Providing a structured approach to fuse heterogeneous covariates (both historical and future) that respects their temporal patterns and statistical properties, rather than treating them as simple extra channels.
• Robustness: Demonstrating consistent improvements across diverse domains (energy, traffic, meteorology), validating its ability to handle non-stationary and time-varying correlation structures common in real-world applications.

The authors conclude that this framework offers a promising direction for multivariate forecasting, with future work planned to extend the architecture to anomaly detection, imputation, and large-scale foundation models.