Technical Summary: TS-Memory

Problem Statement

Time Series Foundation Models (TSFMs) have demonstrated strong zero-shot forecasting capabilities through large-scale pre-training. However, their deployment in real-world scenarios is hindered by distribution shifts (e.g., scale changes, seasonal discrepancies, sensor variations), which severely degrade the performance of frozen backbones. Existing adaptation strategies face a fundamental trade-off:

1. Parametric Adaptation: Methods like full or parameter-efficient fine-tuning (e.g., LoRA) adapt weights to the target domain but require maintaining distinct model copies for each domain (high storage/maintenance costs) and risk catastrophic forgetting of general pre-training knowledge.
2. Non-Parametric Retrieval: Methods like Retrieval-Augmented Generation (RAG) or metric matching preserve the frozen backbone by retrieving similar historical instances from an external database during inference. While effective, this incurs high inference latency that scales linearly with the datastore size, making it unsuitable for real-time or edge deployment.

The core challenge is to capture the robust adaptability of retrieval-based methods while maintaining the constant-time inference and deployment simplicity of parametric adapters.

Methodology: Parametric Memory Distillation

The authors propose TS-Memory, a "Plug-and-Play" framework based on a new paradigm called Parametric Memory Distillation. The core insight is that the "knowledge" provided by online retrieval (specifically, the predictive distribution implied by similar inputs) can be compiled offline into a compact neural module. This shifts the retrieval cost from the inference path to the training phase.

TS-Memory operates in two stages:

1. Privileged Supervision Construction (Offline)

To prevent test-time leakage, a leakage-safe $k$NN teacher is constructed exclusively from the training set.

• Knowledge Base: A database of past contexts and their realized futures is built using the training split only.
• Shift Alignment: To handle distribution shifts (e.g., level offsets), candidate neighbors are re-aligned using a trailing mean shift computed channel-wise before retrieval.
• Teacher Aggregation: For a given input, the top-$K$ nearest neighbors are retrieved. Their future trajectories are aggregated into confidence-aware quantile targets ($\tilde{Q}_t$).
• Confidence Scoring: A confidence score ($Conf_t$) is derived from the concentration of retrieval weights. This score indicates the reliability of the retrieved distribution.

2. Confidence-Gated Memory Distillation (Training)

A lightweight, learnable module (PlugMem) is trained to predict the teacher's quantile targets directly from the input context, without accessing the external database.

• Architecture: PlugMem is a lightweight encoder-decoder Transformer that operates independently of the frozen TSFM backbone, ensuring plug-and-play compatibility.
• Loss Function: The training objective combines three components:
  • Task Supervision ($\mathcal{L}_{task}$): Standard quantile regression against ground truth.
  • Confidence-Gated Distillation ($\mathcal{L}_{align}$): The module aligns with the teacher's quantile targets only when the teacher provides a reliable signal. This is gated by a binary indicator ($\chi_t$) where the teacher outperforms the frozen backbone, weighted by the confidence score ($Conf_t$). It specifically distills the incremental correction ($\Delta$) relative to the backbone to avoid learning global biases.
  • Stability Regularization ($\mathcal{L}_{reg}$): Includes an anchor loss to prevent over-adjustment when retrieval is uncertain and a crossing loss to ensure monotonic quantile outputs.

3. Inference (Retrieval-Free)

During inference, TS-Memory performs a strictly retrieval-free process:

1. The frozen backbone ($f_\theta$) generates base quantile predictions.
2. The PlugMem module ($g_\phi$) generates complementary quantile predictions.
3. The final forecast is a quantile-wise linear interpolation of the two: $\hat{Q}_{final} = (1-\alpha)\hat{Q}_{base} + \alpha\hat{Q}_{mem}$.
   This results in $O(1)$ inference latency, independent of the historical data size.

Key Contributions

1. Paradigm Shift: The paper introduces Parametric Memory Distillation, transforming expensive online retrieval into a learnable, offline memory mechanism. This bridges the gap between parametric adaptation and non-parametric retrieval.
2. TS-Memory Design: A lightweight module that distills non-parametric $k$NN distributions into parametric representations. It features a novel confidence-gated supervision mechanism that selectively transfers knowledge only when retrieval signals are reliable, preventing negative transfer from noisy retrievals.
3. Efficiency & Performance: The method achieves consistent improvements in both point and probabilistic forecasting across diverse TSFMs and benchmarks while maintaining inference latency comparable to the frozen backbone (constant time overhead).

Experimental Results

The authors evaluated TS-Memory on eight multivariate long-term forecasting benchmarks (e.g., ETTh1/2, Electricity, Traffic, Weather) using four frozen TSFM backbones (ChronosBolt, Chronos2, Sundial, TimesFM).

• Performance: TS-Memory consistently outperformed the frozen backbone ("Origin") and other adaptation baselines.
  • Across all dataset-backbone pairs, it reduced MSE by 5.8% and MAE by 2.1% on average.
  • Peak improvements reached 16.0% in MSE and 6.0% in MAE.
  • It also improved probabilistic metrics (CRPS), indicating better-calibrated uncertainty, whereas LoRA sometimes degraded probabilistic performance.
• Efficiency:
  • Latency: Unlike online retrieval baselines (RAFT, TS-RAG) which incur significant latency (up to 100%+ increase due to search), TS-Memory adds only a small constant overhead (~4-5% increase), comparable to the frozen backbone.
  • Storage: It requires storing only a tiny module rather than multiple model copies (Parametric Adaptation) or a large vector database (Non-Parametric Retrieval).
• Generalization:
  • Cross-Model Transfer: A PlugMem trained on one backbone (e.g., ChronosBolt-Base) successfully improved smaller variants and even different TSFM families (e.g., Sundial, TimesFM) without re-optimization.
  • Domain Shift: The method showed robustness under domain shifts, with gains scaling positively with the alignment between the retrieval supervision domain and the target domain.

Significance and Claims

The paper claims that TS-Memory offers a superior trade-off for practical deployment by satisfying three critical requirements simultaneously:

1. Zero Inference Search: It eliminates dependencies on external vector databases at runtime.
2. Constant Time Complexity: Inference latency is $O(1)$, independent of the knowledge base size.
3. Non-Destructive Adaptation: It preserves the frozen backbone, avoiding catastrophic forgetting.

The authors position TS-Memory not as a replacement for retrieval, but as a mechanism to internalize the benefits of retrieval into a lightweight module. By distilling the "distributional corrections" induced by retrieval, TS-Memory enables frozen TSFMs to adapt to local domains with the speed of a standard forward pass, making retrieval-based adaptability feasible for real-time and edge applications.

Limitations Acknowledged: The authors note that the method focuses on long-horizon probabilistic forecasting and currently handles additive channel-wise offsets. It may face challenges with complex temporal warping or abrupt regime changes if the retrieval corpus poorly matches the deployment domain, though the gating mechanism mitigates this risk.