Trio: Learning Time-Series Forecasting with Temporal-Spatial-Sample Attention and Structural Causal Priors

Technical Summary: Trio - Learning Time-Series Forecasting with Temporal-Spatial-Sample Attention and Structural Causal Priors

Problem Statement
Multivariate time-series forecasting requires models to reason over three complex dimensions: temporal dynamics, cross-variable dependencies, and historical input-output correspondences. While recent Prior-Data Fitted Networks (PFNs) have demonstrated success in tabular learning by training on synthetic tasks to learn transferable inference strategies, directly applying this paradigm to time series is challenging. Standard tabular priors fail to capture essential time-series properties such as temporal order, dynamic lags, and recurring historical patterns. Furthermore, most existing forecasting models are trained in a dataset-specific supervised manner, limiting their ability to learn generalizable forecasting mechanisms that transfer across different tasks. The core question addressed is how a forecasting model can explicitly reuse historical input-output examples within a long time-series context, rather than treating history merely as a flat sequence.

Methodology
The paper proposes Trio, a sample-aware time-series forecasting architecture, and a Time-Series Structural Causal Model (TS-SCM) generator to create structured synthetic priors.

1. Trio Architecture (Temporal-Spatial-Sample Attention):

   • Data Reorganization: Instead of processing a continuous historical sequence, Trio reorganizes the input into a set of historical input-output pairs (lookback-future segments) and a current query window. This explicitly models the correspondence between past inputs and their future outcomes.
   • Three-Dimensional Attention: The model employs an alternating encoder with three complementary attention mechanisms:
     • Temporal Attention: Captures within-window dynamic patterns using causal self-attention over patch tokens.
     • Spatial Attention: Models dependencies among variables at each window and patch position.
     • Sample Attention: Retrieves relevant historical lookback-future pairs to guide the current prediction. Unlike standard sample attention using a single CLS token, Trio performs patch-wise retrieval where current future-query tokens attend to aggregated historical future-prompt tokens. This allows the model to selectively absorb useful input-output mappings from previous windows.
   • Token Construction: Historical lookback and future segments are embedded separately. The current window uses learnable query tokens as future placeholders, which query the historical memory bank during sample attention.

2. TS-SCM Generator:

   • To support PFN-style learning, the authors introduce a generator that creates structured synthetic tasks reflecting real-world complexity.
   • Mechanisms: It simulates multivariate causal dynamical systems with heterogeneous node mechanisms, edge-level dynamic lags (which can be state-dependent or history-dependent), noise processes, feedback loops, and distributional drift.
   • Event Calendar: It uses a push-based event calendar to handle continuous lag scheduling and smooth delayed propagation, avoiding crude rounding of lags.
   • Stability: The generator includes stability-aware graph sampling to prevent unstable trajectories while ensuring diversity.

Key Contributions

1. Trio Architecture: A novel sample-aware architecture that decomposes multivariate forecasting into temporal, spatial, and sample-level attention. It treats long historical contexts as structured input-output examples rather than undifferentiated sequences.
2. TS-SCM Generator: A forecasting-oriented synthetic task generator capable of simulating complex dynamical systems with dynamic lags, cross-variable interactions, and non-stationary transitions, providing a richer prior for learning transferable behaviors.
3. Empirical Validation: Comprehensive experiments on synthetic delayed-dependency tasks, industrial datasets, and public benchmarks (ETT, Weather, Electricity) validating the architecture's effectiveness. The paper also includes exploratory zero-shot experiments to assess the transferability of TS-SCM-generated priors.

Experimental Results

• Synthetic Tasks: On datasets designed to test variable selection and delayed dependencies, adding Sample Attention significantly reduced Mean Squared Error (MSE) compared to a base model without it (e.g., MSE dropped from 0.706 to 0.298 on Synthetic Dataset #1).
• Public Benchmarks: Trio achieved the best average MSE on three out of four public datasets (ETTm1, ETTm2, Electricity). On the Weather dataset, it was competitive, achieving the best result at the shortest horizon (96 steps), though TimeMixer++ performed better on average.
• Long Contexts: The model benefits from longer context lengths, but performance gains plateau or drop if the context becomes excessively long, suggesting a need for moderate long-context modeling rather than blindly increasing history.
• Zero-Shot Transfer: In exploratory zero-shot experiments (training only on synthetic TS-SCM data and testing on real ETT benchmarks), the proposed generator outperformed other synthetic strategies (Causal Dynamics, LMC Synth) on three of four datasets in MSE and all four in MAE. However, the authors note that different priors capture different aspects of dynamics, and the results are not uniform across all metrics.

Significance and Claims
The paper modestly claims to study how historical input-output examples can be explicitly organized and reused within a forecasting model, rather than claiming to have solved the problem of fully general PFN-style time-series forecasting. The significance lies in:

• Demonstrating that explicitly modeling historical input-output mappings via sample attention improves performance on tasks with delayed dependencies.
• Showing that structured synthetic tasks (TS-SCM) can provide useful structural priors for transfer learning, as evidenced by the zero-shot results.
• Acknowledging that while TS-SCM-generated tasks offer promising transfer cues, "fully general PFN-style time-series forecasting remains an open problem."

The work positions itself as a step toward more generalizable forecasting mechanisms by leveraging structured historical evidence and causal synthetic priors, without overclaiming the maturity of zero-shot time-series forecasting.