🤖 machine learning

Causal Semantic Alignment for LLM-based Time Series Forecasting

The paper proposes CVAformer, a novel framework that improves LLM-based time series forecasting by explicitly disentangling invariant and dynamic components to mitigate spurious correlations through causal intervention and non-causal attention mechanisms.

Original authors: Kexuan Zhang, Xiaobei Zou, Cesare Alippi, Gary G. Yen, Yang Tang

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kexuan Zhang, Xiaobei Zou, Cesare Alippi, Gary G. Yen, Yang Tang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Language Expert to Read Numbers

Imagine you have a brilliant librarian (a Large Language Model, or LLM) who has read every book in the world. This librarian is amazing at understanding stories, metaphors, and the deep meaning of words. Now, you want to hire this librarian to predict the weather or stock prices based on a spreadsheet of numbers.

The problem? The librarian speaks "English," but the data speaks "Math." To make them work together, you have to translate the numbers into words.

The Old Way (The Problem):
Previous methods tried to translate the numbers by looking at them one second at a time, like reading a book one letter at a time. They tried to force the librarian to treat a temperature reading at 9:00 AM as a "word" and the reading at 9:01 AM as the next "word."

The paper argues this is a mistake for two reasons:

The "Noise" Problem: A temperature reading isn't just the "true" temperature (the meaning); it's also the "true" temperature plus a sudden gust of wind or a sensor glitch (the noise). If you translate the whole thing, the librarian gets confused by the noise, thinking a sudden gust of wind is a new "word" in the story.
The "Order" Problem: In a story, word #1 comes before word #2. But in a spreadsheet of weather data, the temperature, humidity, and wind speed all happen at the same time. Forcing the librarian to read them in a strict line (Temperature -> Humidity -> Wind) is like forcing a group of friends to speak one by one when they are actually all talking at once. It creates artificial rules that don't exist in reality.

The Solution: CVAformer (The "Smart Translator")

The authors propose a new system called CVAformer. Think of it as a smart translator that helps the librarian understand the data without getting confused. It does this in three clever steps:

1. The "Filter" (Separating the Signal from the Noise)

Imagine you are trying to describe a person's personality (their invariant self) while they are running a marathon (the dynamic situation).

The Old Way: You describe them as "sweaty, panting, and red-faced." This description changes every second depending on how hard they are running.
CVAformer's Way: It uses a special filter to separate the person's core personality from the sweat and panting. It says, "Okay, the person is calm and determined (Invariant), but the sweat is just because they are running (Dynamic)."
Why it matters: The paper claims that by removing the "sweat" (dynamic fluctuations) before translating, the librarian gets a much clearer picture of who the person actually is, rather than being distracted by temporary changes.

2. The "Group Huddle" (Non-Causal Attention)

Once the data is filtered, the system needs to understand how the different variables relate to each other.

The Old Way: The librarian reads the data like a sentence: "First the temperature, then the humidity, then the wind." This is called "causal attention."
CVAformer's Way: It lets the librarian look at all the variables at once, like a group of friends having a huddle. The temperature, humidity, and wind are all allowed to "talk" to each other simultaneously.
Why it matters: In the real world, weather variables influence each other instantly. They don't wait for their turn to speak. This "huddle" approach captures the true relationships between variables much better.

3. The "Causal Check" (Fixing the Confusion)

The paper uses a concept called Causal Intervention.

The Analogy: Imagine you see a person running and sweating. You might think, "They are sweating because they are running." But what if they are sweating because it's hot outside? If you don't account for the heat, you might make the wrong prediction.
CVAformer's Way: It acts like a detective. It asks, "Is this change in the data caused by the variable's true nature, or is it just a temporary glitch?" It mathematically "blocks" the influence of the temporary glitches so the librarian only learns from the true, stable patterns.

What Did They Find?

The authors tested this new "Smart Translator" against many other models (including other AI that uses language models) on various tasks:

Long-term predictions: Guessing what the weather will be like weeks from now.
Short-term predictions: Guessing the next few minutes.
Few-shot/Zero-shot: Learning from very little data or data from a completely different place (like learning from New York weather to predict London weather).

The Results:

CVAformer was generally the most accurate, beating the previous best models.
It worked especially well when the data was messy or when there wasn't much data to learn from.
The "filtering" (separating noise) and the "group huddle" (non-causal attention) were proven to be the key reasons for its success.

In Summary

The paper argues that to make AI good at predicting numbers, we shouldn't just force it to read numbers like words. Instead, we need to:

Clean the data first (remove the temporary noise).
Let the variables talk to each other freely (don't force a strict order).
Use logic to ensure the AI isn't tricked by coincidences.

By doing this, the "Librarian" (the LLM) can finally understand the "Math" (the time series) and make much better predictions.

Technical Summary: Causal Semantic Alignment for LLM-based Time Series Forecasting (CVAformer)

1. Problem Statement

Recent advancements in Large Language Models (LLMs) have enabled time series forecasting by aligning temporal patterns with pretrained word embeddings. However, existing LLM-based methods suffer from a fundamental limitation: they often overlook the heterogeneous nature of time series, where invariant semantics (the intrinsic identity of a variable) and dynamic components (externally induced, high-frequency fluctuations) are entangled.

From a causal perspective, this entanglement creates a confounding effect. The dynamic component acts as a confounder ( $C$ ) that simultaneously influences the estimated invariant representation ( $I$ ) and the final aligned embedding ( $A$ ). This leads to spurious correlations and semantic anchor drifts, where the alignment target shifts over time due to dynamic variations, resulting in unstable and biased alignments. Furthermore, most existing approaches perform alignment at the temporal level (treating time steps or patches as tokens) or use causal attention mechanisms that impose artificial autoregressive orderings on variables. This is suboptimal because variables in multivariate time series do not possess an inherent temporal order, and treating them as such introduces inductive biases that do not reflect true inter-variable relationships.

2. Methodology: CVAformer

To address these issues, the authors propose CVAformer (Causal Variable-level Alignment Transformer), a framework that conducts alignment at the variable level while explicitly disentangling invariant and dynamic components. The architecture consists of three main stages:

A. Invariant-Dynamic Decomposition

Before alignment, CVAformer decomposes each variable's time embedding ( $E_{time}$ ) into two distinct components:

Invariant Component ( $E_{invariant}$ ): Represents the stable semantic identity of the variable.
Dynamic Component ( $E_{dynamic}$ ): Captures short-term fluctuations and external variations.

This decomposition is achieved using a CausalEncoder and a learnable MLP. To ensure effective disentanglement, a contrastive loss (inspired by Momentum Contrast) is employed. This loss encourages the invariant representation to remain stable across temporal augmentations while allowing the dynamic component to vary, thereby suppressing entanglement.

B. Causal Alignment (Textual Branch)

The textual branch performs variable-level alignment between the decomposed invariant representation and pretrained word embeddings. To mitigate the confounding effect of the dynamic component, CVAformer applies a backdoor adjustment via causal intervention:

Confounder Summary: A compact representation of the dynamic context ( $E_{causal}$ ) is extracted using global statistics (mean, covariance) and fed into a CausalEncoder.
Causal Intervention: The alignment process conditions the query on both the invariant representation and the confounder summary ( $Q = (E_{invariant} + E_{causal})W_q$ ). This approximates the interventional distribution $P(A | do(I))$, effectively blocking the backdoor path $I \leftarrow C \rightarrow A$ .
Soft Gating: A variable-wise gating mechanism dynamically balances the contribution of the aligned semantic features and the dynamic fluctuations, ensuring the final representation is robust to environmental changes.

C. Temporal Branch (Non-Causal Attention)

To capture inter-variable dependencies without imposing artificial ordering, the temporal branch replaces standard causal attention with Non-Causal Attention Blocks (NCBlocks).

Unlike standard LLMs that use causal masking to enforce autoregression, NCBlocks allow each variable to attend to all other variables at the same time step.
This design captures unordered inter-variable interactions and ensures permutation equivariance, making the model robust to the ordering of input channels.

D. Optimization

The model is trained using a composite loss function:

Supervised Loss ( $L_{sup}$ ): Ensures forecasting accuracy on the temporal branch.
Invariant Loss ( $L_{invariant}$ ): Enforces the disentanglement of invariant and dynamic components via contrastive learning.
Consistency Loss ( $L_{consistency}$ ): Encourages semantic agreement between the textual and temporal branches.
The backbone LLM is kept frozen, with only lightweight adapters (LoRA, normalization layers, positional embeddings) updated to prevent overfitting and reduce computational cost.

3. Key Contributions

The paper outlines four primary contributions:

Variable-Level Alignment Paradigm: A shift from tokenizing time steps to treating entire variables as tokens, enabling LLMs to capture semantic information without artificial temporal orderings.
Causally Grounded Disentanglement: A principled methodology that separates invariant semantics from dynamic fluctuations using a backdoor-inspired mechanism to eliminate confounding effects during cross-modal alignment.
Unified Non-Causal Architecture: The integration of non-causal attention blocks to model unordered inter-variable dependencies, which can be readily incorporated into existing LLM-based predictors.
Comprehensive Empirical Validation: Extensive experiments demonstrating consistent improvements over state-of-the-art models across long-term, short-term, few-shot, and zero-shot forecasting settings.

4. Experimental Results

CVAformer was evaluated on multiple datasets (Weather, Traffic, Electricity, ETT, M4) across four tasks:

Long-term Forecasting: CVAformer consistently outperformed baselines (including TimeLLM, GPT4TS, PatchTST, and iTransformer) on most datasets, achieving relative reductions of 10.07% in MSE and 8.21% in MAE compared to PatchTST. It showed particular strength in datasets with rich cross-variable interactions.
Short-term Forecasting: On the M4 dataset, CVAformer ranked first in 11 out of 15 evaluation cases, achieving an average improvement of 2.8% over PatchTST and 2.2% over GPT4TS.
Few-shot and Zero-shot Forecasting: In data-constrained settings (10% training data) and cross-domain transfer tasks, CVAformer demonstrated superior generalization, reducing errors by 13% compared to PatchTST in few-shot settings and 17.7% in zero-shot settings.
Ablation Studies: Removing the causal disentanglement module led to significant performance drops (e.g., +3.2% MSE on ETTh1), validating the necessity of addressing confounding effects. Similarly, replacing non-causal attention with causal attention degraded performance, confirming the benefit of modeling unordered variable interactions.
Efficiency: Despite its complexity, CVAformer maintained a competitive training time (13.74s/epoch) and memory footprint (709.15 MB) compared to other large models like TimeLLM and FEDformer.

5. Significance and Claims

The paper claims that CVAformer offers a new perspective for LLM-based time series tasks by addressing the root cause of semantic drift: the entanglement of dynamic and invariant components. By explicitly modeling the causal structure of time series data, the framework achieves more interpretable representations and improved generalization across diverse scenarios.

The authors emphasize that their approach moves beyond simple cross-modal alignment to a causally guided process that respects the intrinsic properties of multivariate time series (i.e., the lack of inherent variable ordering and the presence of confounding dynamics). While the model shows robustness across different LLM backbones (GPT-2, BERT, DeepSeek), the authors modestly note that future work will focus on deeper causal analysis of the alignment process and further enhancing model interpretability for real-world applications.