Technical Summary: StateFlow – Dual-State Recurrent Modeling for Long-Horizon Time Series Forecasting

1. Problem Statement

Long-horizon multivariate time series forecasting (LTSF) is a critical challenge in domains such as energy demand, traffic flow, and financial modeling. The primary difficulties stem from non-stationarity, regime shifts, and the accumulation of errors over extended prediction horizons. While recent Transformer-based architectures have achieved strong performance by capturing long-range dependencies via self-attention, they often suffer from quadratic computational complexity relative to input length and rely on global interaction patterns that can be computationally expensive for long lookback windows.

Conversely, recurrent neural networks (RNNs) scale linearly with sequence length and offer a natural causal structure. However, conventional recurrent models (e.g., LSTMs, GRUs) typically compress temporal information into a single hidden state, failing to explicitly model systematic prediction deviations (such as drift or bias) that arise from non-stationary changes. This limitation often results in their performance lagging behind modern Transformer-based methods in long-horizon settings.

The paper addresses the gap in extending the Variability-Aware Recursive Neural Network (VARNN)—originally designed for one-step sequence regression—to the long-horizon forecasting setting. The core hypothesis is that explicitly modeling prediction residuals as a distinct recurrent state can enhance the robustness of linear recurrent models for long-term extrapolation.

2. Methodology: The StateFlow Framework

The proposed StateFlow framework utilizes VARNN as a dual-state recurrent backbone to capture two complementary signals from the lookback sequence, processed via a channel-independent strategy.

2.1 Dual-State Recurrent Encoder

The encoder maintains two distinct recurrent trajectories for each time step $t$:

1. Hidden-State Trajectory ($h_t$): Represents primary temporal dynamics, including trends, seasonality, level changes, and recurring patterns. It is updated using the current observation and previous states.
2. Residual-Memory Trajectory ($e_t$): Explicitly tracks structured local prediction deviations. It is driven by a nonlinear recurrent transformation of the residuals ($r_t$) between one-step base predictions ($\hat{x}_{base}$) and observed values ($x_{t+1}$).

Unlike conventional RNNs that implicitly encode deviations within a single latent state, VARNN separates these signals. The hidden state conditions the next step, while the residual-memory state provides a dedicated signal for systematic deviations, allowing the model to distinguish between primary dynamics and local drift.

2.2 Chunk-Based Horizon Decoder

To map the encoder trajectories to the future horizon without incurring the parameter explosion associated with flattening full sequences, StateFlow employs a chunk-based decoder:

• Chunk Summarization: The hidden-state and residual-memory trajectories are split into overlapping chunks. Each chunk is flattened and projected into a fixed-dimensional embedding via separate summarization branches. This preserves local patterns while reducing parameter growth.
• Forecasting Head: The summarized embeddings from both branches are concatenated and passed through a linear projection to generate the multi-step forecast.

2.3 Two-Stage Optimization Strategy

To bridge one-step residual learning with long-horizon forecasting, the model employs a two-stage training procedure:

1. Stage 1 (Encoder Pretraining): The VARNN encoder is trained using a one-step prediction objective over the lookback window. This provides an explicit learning signal for the residual-memory state to capture structured deviations.
2. Stage 2 (Horizon-Specific Decoding): The pretrained encoder is frozen, and a horizon-specific decoder is trained using multi-step forecasting loss. This separation stabilizes optimization and allows the same encoder to be reused across different prediction horizons.

2.4 Computational Complexity

StateFlow maintains linear computational complexity with respect to the lookback length ($L$) and the forecasting horizon ($H$). The recurrent encoder scales as $O(L)$, and the chunk-based decoder avoids the quadratic complexity ($O(L^2)$) typical of self-attention mechanisms.

3. Key Contributions

The paper outlines the following contributions:

• Investigation of Recurrence in LTSF: It demonstrates that recurrence can remain competitive for long-horizon forecasting when deviation dynamics are explicitly incorporated into state transitions.
• Extension of VARNN: It extends VARNN from one-step regression to long-horizon forecasting by utilizing hidden and residual-memory trajectories as dual-state forecasting representations.
• StateFlow Framework: It introduces a recurrent trajectory forecasting framework featuring a chunk-based horizon decoder that separately summarizes hidden and residual-memory dynamics before direct multi-step prediction.
• Two-Stage Optimization: It proposes a training strategy that first learns residual-aware representations via one-step prediction and then trains horizon-specific decoders, enabling encoder reuse.
• Empirical Validation: It provides systematic experiments and ablations showing that residual-memory recurrence is a competitive and parameter-efficient inductive bias for long-horizon forecasting.

4. Experimental Results

4.1 Benchmark Performance

StateFlow was evaluated on seven standard LTSF benchmarks (ETTh1, ETTh2, ETTm1, ETTm2, Weather, ECL, Traffic) against ten state-of-the-art baselines, including Transformer-based models (iTransformer, PatchTST, Autoformer), convolutional models (TimesNet), and linear models (DLinear, RLinear).

• Competitiveness: StateFlow achieved competitive performance, securing the highest number of first-place results (15 MSE wins and 14 MAE wins across 28 dataset-horizon settings).
• Comparison: It performed comparably to or better than strong linear, convolutional, and Transformer-based baselines while avoiding quadratic temporal self-attention.
• High-Dimensional Data: On high-dimensional datasets like ECL and Traffic, iTransformer remained stronger in some settings, suggesting that explicit cross-variate dependency modeling may still offer advantages in those specific contexts. StateFlow, however, focuses on channel-independent temporal modeling.

4.2 Ablation Studies

• Encoder Representations: The dual-state representation ($h, e$) consistently outperformed using only the hidden state ($h$), only the residual-memory state ($e$), or raw base predictions with residuals. This confirms that combining primary dynamics with structured deviation signals yields richer forecasting representations.
• Fusion Strategy: Separate chunk summarization for hidden and residual trajectories (before concatenation) proved superior to joint chunking, preserving the distinct roles of the two states.
• Chunk vs. Direct Decoding: The chunk-based decoder achieved comparable or better accuracy than direct flattening while reducing model parameters by nearly half.
• Recurrent Encoder Choice: VARNN outperformed standard RNN, GRU, and LSTM encoders in terms of accuracy-efficiency tradeoff. Notably, VARNN used significantly fewer parameters (e.g., ~80% fewer than LSTM) while achieving lower average MSE.
• Optimization Strategy: The two-stage optimization strategy provided a measurable gain (e.g., +1.88% improvement for VARNN) over one-stage training, highlighting the effectiveness of pretraining the residual-memory state via one-step prediction.

5. Significance and Claims

The paper claims that StateFlow offers an efficient alternative to attention-based models for long-horizon forecasting. By explicitly modeling residual-memory dynamics within a recurrent framework, the model captures non-stationary changes and systematic deviations that standard recurrent models miss.

The significance lies in demonstrating that linear recurrent encoding, when augmented with a dedicated residual-memory state and a chunk-based decoder, can compete with complex Transformer architectures without incurring quadratic computational costs. The authors suggest that incorporating deviation dynamics into recurrence provides a promising, parameter-efficient inductive bias for LTSF. The work does not claim to solve all challenges in high-dimensional multivariate forecasting (acknowledging the potential need for cross-variate modeling) but establishes a strong baseline for residual-aware recurrent approaches.

Future work is identified as investigating cross-variate dependency modeling and applying the framework to other time-series analysis tasks.