Factorize to Generalize: Retrieval-Guided Invariant-Dynamic Decomposition for Time Series Forecasting

Technical Summary: Factorize to Generalize (RIDDE)

Problem Statement
Time Series Foundation Models (TSFMs) have demonstrated strong zero-shot forecasting capabilities through large-scale pretraining and retrieval-augmented prediction. However, the authors identify a non-trivial limitation in existing retrieval-based approaches: retrieval tends to induce more oscillatory predictions. While this improves performance on highly fluctuating series, it degrades accuracy on smoother, trend-dominated series. Empirical analysis reveals that retrieval reduces errors on high-frequency (seasonal) components but increases errors on low-frequency (trend) components. This suggests that retrieved information is fused into the prediction without explicitly distinguishing between stable temporal structures (invariant across environments) and instance-specific variations (dynamic). Consequently, standard models learn entangled representations that reduce robustness under distribution shifts.

Methodology: RIDDE Framework
The authors propose RIDDE (Retrieval-guided Invariant-Dynamic DEcomposition), a framework that shifts the role of retrieval from a direct predictive context to a structural signal for representation decomposition. The core intuition is that retrieved sequences act as implicit samples from related environments; while they share underlying mechanisms with the query, their dynamic variations differ.

The framework operates through the following stages:

1. Retrieval-Guided Representation Learning:

   • The input query sequence is partitioned into patches and encoded.
   • Top-$K$ similar sequences are retrieved from a knowledge base based on cosine similarity in the latent space.
   • Instead of directly reusing retrieved contexts, the future horizons of these retrieved samples are projected into a shared latent space and aggregated via an attention mechanism. This yields a consensus representation ($h_{ret}$) that emphasizes patterns consistently aligned with the query, effectively reducing the variance of sample-specific fluctuations.
   • A learnable gating mechanism fuses the intrinsic query representation ($q$) with the retrieval consensus ($h_{ret}$) to form a fused representation ($h$).

2. Invariant–Dynamic Decomposition:

   • A retrieval-guided routing mechanism decomposes the fused representation $h$ into two components:
     • Invariant Component ($z_{inv}$): Encouraged to preserve stable structures supported across retrieved samples.
     • Dynamic Component ($z_{dyn}$): Encouraged to capture query-specific deviations and residual variations.
   • This decomposition is guided by a routing gate conditioned on both the fused representation and the retrieval consensus, biasing the split toward the structural information provided by the retrieval.

3. Mechanism-Aware Forecasting:

   • The invariant and dynamic components are processed by separate predictors ($f_{inv}$ and $f_{dyn}$).
   • The final prediction is obtained by fusing the outputs of these separate pathways, allowing the model to balance transferable patterns with adaptability to evolving dynamics.

4. Training Objectives:

   • The model is trained using a prediction loss ($L_{pred}$) to minimize forecasting error.
   • An orthogonality-based disentanglement regularizer ($L_{dis}$) is introduced to reduce overlap between the invariant and dynamic factors, acting as a weak geometric constraint to encourage structured decomposition.

Theoretical Insight
The paper provides a theoretical perspective based on variance reduction. By aggregating multiple retrieved samples, the consensus representation approximates the shared latent structure ($\mu$) while reducing the variance of sample-specific noise ($\epsilon_k$). This aggregation serves as a low-variance anchor, motivating its use as a structural guide for separating invariant and dynamic factors without requiring explicit environment supervision.

Key Contributions

• Identification of Limitation: The authors identify that retrieval-augmented forecasting introduces a structural bias toward high-frequency fluctuations, weakening the modeling of stable trends.
• RIDDE Framework: Proposes a novel framework that uses retrieval as a structural signal for representation decomposition rather than merely as auxiliary predictive context.
• Routing Mechanism: Develops a routing-based mechanism that decomposes retrieval-aware representations into invariant and dynamic components, supported by a regularizer to encourage branch separation.
• Empirical Validation: Demonstrates consistent improvements in robustness and zero-shot forecasting performance under distribution shifts across diverse benchmarks.

Experimental Results
Extensive experiments were conducted on seven benchmark datasets (ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity, Exchange) using a zero-shot setting.

• Performance: RIDDE achieved the best average MSE and MAE across all datasets, outperforming seven existing TSFMs (including Chronos, MOMENT, TimesFM) and three RAG-based baselines (RAF, TS-RAG, Cross-RAG).
• Ablation Studies: Removing the Invariant-Dynamic Decomposition (IDD) or the disentanglement loss ($L_{dis}$) resulted in significant performance degradation, confirming the necessity of both components.
• Efficiency: RIDDE demonstrated lower average step times and training time per epoch compared to RAG-based baselines, with comparable peak GPU memory usage, attributed to its lightweight gating and decomposition design.
• Sensitivity: The model showed robust performance across a range of hyperparameters, particularly regarding the number of retrieved sequences ($K$) and the scaling parameter ($\rho$).

Significance
The paper argues that the effectiveness of retrieval in time series forecasting is not uniform; it can inadvertently bias models toward oscillatory patterns. By reframing retrieval as a tool for representation decomposition, RIDDE enables models to preserve transferable temporal structures while remaining adaptive to specific dynamics. This approach offers a structured alternative to standard fusion methods, improving robustness under distribution shifts without the need for task-specific fine-tuning. The work suggests that disentangling stable and dynamic factors is crucial for generalization in non-stationary environments.