VFEM: Visual Feature Empowered Multivariate Time Series Forecasting with Cross-Modal Fusion

Technical Summary: VFEM – Visual Feature Empowered Multivariate Time Series Forecasting with Cross-Modal Fusion

Problem Statement
Multivariate time series forecasting faces two primary limitations in current large-scale foundation models. First, the prevalent channel-independent architecture treats each variable separately to handle varying data dimensions. While this facilitates zero-shot generalization, it ignores crucial cross-channel dependencies and correlations that could enhance forecasting accuracy. Conversely, channel-dependent models often require full-parameter training on massive datasets, incurring high computational costs and potentially failing to capture domain-specific patterns when flattening multivariate series.

Second, existing cross-modal methods for time series predominantly rely on textual modalities (e.g., repurposing LLMs with text prompts). These approaches underutilize the spatial pattern recognition capabilities of vision models. Multivariate time series, when rendered as 2D images, exhibit visual patterns such as periodicity, lead-lag relationships, and anomalous events that are naturally suited for recognition by pre-trained vision models, yet this potential remains largely unexplored in cross-modal integration.

Methodology: VFEM Architecture
The authors propose VFEM (Visual Feature Empowered Multivariate Time Series Forecasting), a cross-modal framework designed to capture complex cross-variable patterns by leveraging pre-trained Large Vision Models (LVMs). The architecture consists of three main stages:

1. Visual Modality Processing:

   • Transformation: The multivariate time series $X \in \mathbb{R}^{L \times M}$ (where $L$ is the look-back window and $M$ is the number of variables) is transposed to $X^\top \in \mathbb{R}^{M \times L}$. This matrix is treated as a single-channel intensity map and replicated across three channels ($c=3$) to form a 3-channel RGB image input $X_{vs}$. In this representation, rows correspond to variables and columns to time steps.
   • Feature Extraction: A pre-trained vision encoder (specifically SigLIP-2-base-NaFlex) processes the image to extract a global visual embedding. Crucially, the vision encoder is frozen during training.
   • Expansion & Projection: Since the vision encoder outputs a single global embedding, it is replicated across all $M$ variables to provide shared visual context. A learnable projection network (with residual connections) then bridges the domain gap between visual features and time series representations, mapping them to a unified hidden dimension.

2. Temporal Modality Processing:

   • The raw time series undergoes Reversible Instance Normalization (RevIN) to remove distribution shifts while preserving temporal patterns.
   • The normalized series is processed by a temporal branch using spatiotemporal self-attention. The temporal dimension is partitioned into segments, and the tensor is rearranged to allow attention mechanisms to capture dependencies across both variables and time segments simultaneously.

3. Cross-Modal Fusion:

   • The visual hidden states ($H_{vs}$) and temporal hidden states ($H_{ts}$) are concatenated along the feature dimension.
   • A fusion encoder (using self-attention layers) processes the combined features to model cross-modal interactions and cross-variable dependencies jointly.
   • Finally, a linear prediction layer generates the forecast, followed by reverse instance normalization to restore the original data scale.

Key Contributions

• Cross-Modal Framework: The paper introduces a novel approach that transforms multivariate time series into visual representations, enabling pre-trained vision models to perceive spatial relationships and cross-variable patterns that channel-independent models miss.
• Parameter Efficiency: By freezing the large vision encoder, VFEM trains only 7.45% of its total parameters (approximately 30M trainable parameters out of 375M total). This strategy leverages pre-trained visual knowledge while avoiding overfitting on limited time series data.
• Performance: Through the fusion of visual and temporal branches, VFEM achieves competitive forecasting performance across multiple benchmarks, demonstrating that visual pattern recognition is a viable and effective modality for time series analysis.

Experimental Results
The authors evaluated VFEM on seven benchmark datasets: ETTh1, ETTh2, ETTm1, ETTm2, Electricity, Weather, and Traffic. The model was compared against:

• Zero-Shot Time Series Foundation Models: Chronos (Base/Large), Moirai (Base/Large).
• Task-Specific Supervised Models: GPT4TS, UniTST, TimesNet, PatchTST.

Key Findings:

• Superior Accuracy: VFEM achieved the lowest Mean Squared Error (MSE) and Mean Absolute Error (MAE) across all datasets and prediction horizons ($F \in \{96, 192, 336, 720\}$), outperforming both zero-shot foundation models and supervised task-specific baselines.
• Long-Sequence Stability: The visual modality enhanced the model's ability to recognize long-term patterns, resulting in slower performance degradation as the forecasting horizon increased (e.g., minimal MSE increase from $F=336$ to $F=720$).
• Ablation Studies:
  • Removing either the visual or temporal branch degraded performance, with the visual branch having a more significant impact on long-term forecasting.
  • The projection layer was essential for aligning modalities.
  • Freezing Strategy: Keeping the vision encoder fully frozen yielded the best results compared to unfreezing the last 2 or 4 layers or full fine-tuning, suggesting that preserving the pre-trained visual representations is critical when dealing with heterogeneous modalities.
  • Representation Quality: t-SNE visualizations and Silhouette Score analysis confirmed that the trained projection layers successfully adapted pre-trained visual features to the time series domain, producing embeddings with significantly better clustering quality than raw frozen encoder outputs or hand-crafted statistical features.

Significance and Claims
The paper claims that VFEM offers a new perspective on multivariate time series forecasting by validating the utility of visual modalities. It demonstrates that:

1. Multivariate time series contain visually recognizable patterns (periodicity, lead-lag, anomalies) that can be effectively captured by pre-trained vision models.
2. Cross-modal fusion of visual and temporal features can effectively model complex inter-variable dependencies without the computational burden of full-parameter training on large-scale time series corpora.
3. Freezing a large vision model and training a small fraction of parameters is a highly efficient strategy that achieves state-of-the-art performance, providing a viable alternative to existing channel-independent or fully fine-tuned channel-dependent approaches.