Technical Summary: Guard – Multi-Foundation Model Guidance for Lightweight, Robust Scientific Time Series Forecasting

1. Problem Statement

The deployment of Time-Series Foundation Models (TSFMs) in physical sciences faces a critical trade-off. While TSFMs encode rich, universal temporal dynamics, they suffer from severe distributional misalignment when applied zero-shot to specific scientific domains (e.g., soil moisture, carbon flux). Furthermore, their computational cost prohibits deployment in resource-constrained edge-computing sensor networks.

Existing knowledge distillation (KD) approaches typically treat teacher outputs as fixed target labels. This assumption fails in scientific contexts where data distributions shift, and sensor noise is stochastic. Consequently, student models often struggle to capture critical temporal patterns, including extreme events, because they blindly mimic teachers that may be misaligned with the target domain. The core challenge addressed is: How can we extract latent structural knowledge from misaligned foundation models to train lightweight, specialized forecasters without incurring the computational cost of running multiple giant models at inference time?

2. Methodology: Guard Framework

The authors propose Guard (Gated Uncertainty-Aware Routing for Distillation), a framework that reframes multi-teacher distillation as an instance-wise decision process. Instead of running all teachers at inference, Guard compiles them into a single lightweight student using two adaptive mechanisms:

A. Training Pipeline

The framework operates in two phases:

1. Phase 1 (Offline): Pretrained teachers (e.g., TimesFM, Chronos) are executed in zero-shot mode over training, validation, and test windows. Predictions and uncertainty estimates are cached. Pseudo-oracle routing weights are computed based on relative teacher errors to supervise the router during training.
2. Phase 2 (Joint Training): A lightweight student model, a Contextual Router, and an Adaptive Temperature Network are trained jointly using the cached teacher outputs.

B. Core Adaptive Mechanisms

1. Contextual Router:

   • Function: Dynamically selects and weights teachers based on local input statistics (regime features) rather than global averages.
   • Inputs: Extracts three domain-agnostic statistics from the trailing 12 timesteps: rolling standard deviation (variability), last observed value (signal magnitude), and local linear trend.
   • Mechanism: A small feedforward network predicts mixing weights ($w$) for each teacher. It learns to approximate "oracle" routing (which teacher is better for a specific window) using only observable features, enabling test-time routing without ground-truth labels.

2. Uncertainty-Gated Temperature Mechanism:

   • Function: Acts as a "circuit-breaker" to attenuate distillation strength when teacher confidence diverges from domain reality.
   • Mechanism: An adaptive temperature network ($\tau$) rescales the distillation loss based on teacher uncertainty. High uncertainty leads to high temperatures, which suppress gradients ($1/\tau^2$), preventing the student from learning spurious patterns from unreliable teachers.
   • Aggregation: The framework aggregates teacher predictions and uncertainties using the law of total variance, decomposing uncertainty into aleatoric (individual teacher spread) and epistemic (inter-teacher disagreement) components.

C. Student Architecture

The student is a compact multi-horizon forecaster (approx. 0.3M parameters, ~1.15 MB), roughly 400× smaller than TimesFM. It uses simple temporal processing (linear/convolutional projections) followed by a shallow decoder, ensuring performance gains stem from the distillation mechanism rather than raw model capacity.

D. Loss Function

The joint objective combines three terms:

1. Forecast Loss: MSE against ground truth.
2. Distillation Loss: Weighted by the effective temperature gate, encouraging the student to match the adaptively weighted teacher signal.
3. Entropy Regularization: Prevents the router from collapsing into a "winner-take-all" state, ensuring probabilistic distribution across teachers in ambiguous regimes.

3. Key Contributions

1. Dynamic Orchestration: Introduces an instance-wise adaptive weighting mechanism that allows the student to adapt to the heterogeneous dynamics of scientific time series by leveraging complementary strengths of multiple TSFMs.
2. Uncertainty-Aware Gating: Implements a mechanism that adjusts distillation strength per sample based on estimated teacher uncertainty, extracting useful structural priors while filtering noise caused by distributional misalignment.
3. Efficient Edge Deployment: Produces a lightweight student model that achieves state-of-the-art accuracy on scientific benchmarks, enabling high-precision forecasting suitable for resource-constrained edge deployment without repeated foundation model queries.

4. Experimental Results

The framework was evaluated on four scientific domains: Meteorology (Weather), Ecosystem Carbon Flux (Flux), Soil Moisture, and Energy Grids (ETTm1/ETTh1). Teachers used were TimesFM and Chronos.

• Performance Gains: Guard significantly reduced RMSE relative to fixed-weight multi-teacher distillation baselines.
  • Achieved a 28.3% reduction in Mean RMSE compared to the average of Deep SOTA baselines.
  • Established a new global Best RMSE of 0.095, outperforming the strongest baseline (DLinear) by 40.3% in peak performance.
  • Outperformed specialized baselines (Transformer, PatchTST) on challenging Flux and Soil Moisture datasets where zero-shot foundation models failed (RMSE > 400 for Flux).
• Handling Misalignment: Even when teachers exhibited suboptimal zero-shot accuracy due to domain shifts (e.g., TimesFM/Chronos on high-frequency carbon flux turbulence), Guard successfully distilled latent structural knowledge. The method demonstrated that domain-misaligned teachers can serve as critical correctives, outperforming globally superior FMs on 28.5% of the hardest instances.
• Ablation Studies:
  • Synergistic Improvement: In volatile domains (Flux, Weather), both routing and temperature gating provided monotonic gains (25–44% RMSE reduction).
  • Corrective Filtering: In structured energy data (ETTh1), the temperature network acted as a filter, recovering performance losses caused by the router selecting over-confident but inaccurate teachers.
• Scalability: When extended to a third teacher (Moirai), the router autonomously detected redundancy (high error correlation between Chronos and Moirai) and grouped them, confirming that teacher selection should prioritize error independence over sheer quantity.

5. Significance and Claims

The paper claims that Guard successfully bridges the gap between massive Time-Series Foundation Models and practical scientific forecasting. By reframing TSFMs as knowledge repositories rather than direct forecasters, the method distills their temporal priors into a model suitable for real-time edge inference.

• Scientific Impact: The framework enables high-precision forecasting in climate, agriculture, and energy monitoring where data acquisition is costly and label scarcity is common.
• Interpretability: The use of physical routing features (volatility, magnitude, trend) provides domain experts with an auditable decision path, explaining why a specific teacher was trusted for a given instance.
• Efficiency: The upfront GPU investment for Phase 1 (offline teacher inference) is amortized over an unlimited edge deployment lifetime, with the student running at sub-millisecond latency on standard CPU hardware.

The authors conclude that while the method relies on teachers providing useful temporal priors and reliable uncertainty estimates, it offers a robust solution for extracting value from foundation models even under significant domain misalignment. Future work aims to integrate physics-informed constraints and extend the approach to spatiotemporal forecasting.