Technical Summary: Cross-Variable Loss for Multivariate Time Series Forecasting

1. Problem Definition and Motivation

Multivariate Time Series Forecasting (MTSF) aims to predict future values of multiple interrelated variables. While recent advances have focused heavily on modeling input-side channel dependencies (extracting representations from historical features), this paper argues that output-side dependencies—the intrinsic correlations among future predictions—have been largely overlooked.

Current mainstream approaches predominantly follow the Direct Forecasting (DF) paradigm. In this paradigm, a model learns a mapping $f_\theta: \mathbb{R}^{H \times D} \to \mathbb{R}^{T \times D}$ to generate $T$-step forecasts for $D$ variables simultaneously. However, standard DF training relies on point-wise loss functions (e.g., Mean Squared Error, MSE) that treat each time-step and variable entry as an independent scalar prediction.

The authors identify a critical limitation in this approach:

1. Physical Mismatch: Real-world multivariate systems (e.g., financial markets, traffic networks) exhibit shared system dynamics where variables co-evolve. Blindly decoupling future predictions violates these physical laws, potentially leading to forecasts that are numerically accurate point-wise but structurally contradictory (e.g., violating market rules or physical constraints).
2. The Objective Gap: From a probabilistic perspective, standard point-wise MSE optimization assumes that prediction residuals follow a completely independent, isotropic Gaussian distribution (diagonal covariance matrix). However, due to cross-variable and lagged dependencies, the true spatiotemporal covariance matrix of residuals contains significant non-zero off-diagonal elements. This creates a theoretical "Objective Gap" ($\Delta \neq 0$) between the standard loss and the true joint distribution of the data. Consequently, minimizing point-wise MSE fails to capture the joint geometric topology of the original data.

2. Methodology: Cross-Variable Loss (CvLoss)

To address the Objective Gap, the authors propose Cross-Variable Loss (CvLoss), a plug-in structural regularizer that explicitly constrains the consistency of forecast residuals across variables.

2.1 Theoretical Foundation

The authors formalize the problem using a graph-structured perspective. They define the prediction error vector $e$ and posit that the true residuals follow a multivariate Gaussian distribution with a structured precision matrix $\Sigma_{ST}^{-1}$ containing non-zero off-diagonal elements.

• Proposition 3.2 establishes that minimizing the negative log-likelihood of a Gaussian with a structured precision matrix is equivalent to minimizing the standard MSE plus a squared graph total variation (TV) penalty over a cross-variable graph.
• While the theory suggests an $\ell_2$ (squared) penalty, the authors empirically choose an $\ell_1$ norm for the penalty term. This choice is motivated by the need to be less sensitive to localized anomalies or variable-specific shocks that might dominate a squared penalty, while still promoting graph-level residual consistency.

2.2 Loss Formulation

CvLoss operates on patch nodes. The forecast horizon $T$ is divided into $P$ non-overlapping patches of length $L$. Each node $v$ represents a patch for a specific variable.

• Graph Construction: A cross-variable graph is constructed where edges connect variables. The authors define three edge types:
  1. Synchronous edges: Connect variables at the same time step (capturing concurrent interactions).
  2. Asynchronous edges: Connect variables across different time steps (capturing lagged propagation).
  3. Fully connected edges: The default setting, including both synchronous and asynchronous interactions to maximize coverage without domain-specific priors.
• Loss Calculation: For each edge $(i, j)$ in the graph, the loss penalizes the discrepancy between the structural differences of the predictions and the ground truth.
  $$L_{cv} = \frac{1}{|E| \times L} \sum_{(i,j) \in E} \| (\hat{z}_i - \hat{z}_j) - (z_i - z_j) \|_1$$
  Using the node-wise residuals $e_i = \hat{z}_i - z_i$, this simplifies to the total variation of the residuals across the graph:
  $$L_{cv} = \frac{1}{|E| \times L} \sum_{(i,j) \in E} \| e_i - e_j \|_1 = \frac{1}{|E| \times L} \| \tilde{A} \text{vec}(E) \|_1$$
  where $\tilde{A}$ is the lifted incidence matrix of the graph.

2.3 Training Objective

The final training objective combines the standard point-wise DF loss ($L_{df}$) with the structural regularizer ($L_{cv}$):
$$L_\alpha = (1 - \alpha) \cdot L_{df} + \alpha \cdot L_{cv}$$
where $\alpha \in (0, 1)$ is a fixed hyperparameter tuned on the validation set. Crucially, CvLoss is model-agnostic and plug-and-play, integrating seamlessly with various backbones (Transformers, MLPs, etc.) without altering the inference procedure.

3. Key Contributions

1. Identification of the Objective Gap: The paper demonstrates that standard point-wise learning objectives neglect inherent inter-channel error dependencies among future predictions, creating a theoretical mismatch with real-world spatiotemporal correlations.
2. CvLoss Proposal: A novel structural regularizer that explicitly constrains inter-channel relationships via a residual consistency formulation on a cross-variable graph. It encourages consistency in both synchronous and asynchronous interactions.
3. Empirical Validation: Extensive experiments show that CvLoss consistently improves state-of-the-art (SOTA) models across diverse datasets (ETT, ECL, Traffic, Weather, PEMS) and architectures (iTransformer, TimeFilter, TQNet, etc.), achieving superior predictive performance with minimal computational overhead.

4. Experimental Results

The authors evaluated CvLoss on 12 standard benchmarks using various backbones. Key findings include:

• Performance Gains: Adding CvLoss consistently reduced forecasting errors (MSE and MAE). For example, on the Traffic dataset, MAE decreased by 0.014. Across 114 backbone-controlled comparisons, CvLoss improved results in 111 cases, tied in 2, and lost in 1, with an average relative error reduction of 3.39% in MSE and 3.61% in MAE.
• Structural Recovery: Visualizations of cross-variable correlation matrices show that models trained with CvLoss recover the ground-truth correlation structure significantly better than baselines. On the ECL dataset, CvLoss improved the structural error for 84.83% of cross-variable entries compared to a plain iTransformer baseline.
• Comparison with Other Objectives: CvLoss outperformed 11 other learning objectives, including shape-alignment losses (Soft-DTW), likelihood maximization (QDF, Time-o1), and distribution balancing methods.
• Ablation Studies:
  • Topology: The fully connected graph (combining synchronous and asynchronous edges) yielded the best performance, confirming the necessity of modeling both types of interactions.
  • Hyperparameters: The method is robust to the choice of $\alpha$, with optimal performance typically found around $\alpha \approx 0.5$. It is also robust to patch length variations.
• Efficiency:
  • Inference: CvLoss imposes zero overhead during inference as it is removed from the computation graph after training.
  • Training: While the full graph introduces quadratic complexity $O(P^2D^2)$, the authors demonstrate that random edge sampling (limiting edges to 1000 per batch) effectively reduces training time to a constant level without sacrificing predictive accuracy.

5. Significance and Limitations

Significance:
The paper argues that modeling the structural geometry of future variables is a simple yet effective direction for improving MTSF. By shifting focus from purely input-side dependencies to output-side structural consistency, CvLoss bridges the gap between the mathematical assumptions of standard loss functions and the physical realities of co-evolving systems. It offers a universal, lightweight mechanism to enhance existing forecasting models without requiring architectural changes.

Limitations:

• Graph Construction: The default approach uses a fully connected graph, which may include uninformative interactions in very high-dimensional systems. While random sampling addresses scalability, learning adaptive graph structures remains a future direction.
• Scope: The work focuses on deterministic forecasting on regular benchmark datasets. Extensions to probabilistic forecasting, irregular sampling, and missing-value scenarios are not addressed.
• Theoretical vs. Empirical: The authors acknowledge that while the theoretical derivation (Proposition 3.2) supports an $\ell_2$ (squared) penalty, the deployed method uses an $\ell_1$ norm based on empirical evidence. The theory motivates the need for a relation-aware constraint but does not strictly derive the specific $\ell_1$ form used.

In conclusion, CvLoss provides a principled way to regularize multivariate forecasts, ensuring that the predicted future variables evolve in a manner consistent with their underlying physical and statistical dependencies.