SPaRSe-TIME: Saliency-Projected Low-Rank Temporal Modeling for Efficient and Interpretable Time Series Prediction

1. Problem Statement

Time series forecasting has traditionally relied on sequence-based architectures like Recurrent Neural Networks (RNNs), LSTMs, and Transformers. While effective, these models suffer from three primary limitations:

• Computational Inefficiency: They often process all time steps uniformly, leading to high computational costs (e.g., $O(T^2)$ for Transformers due to self-attention).
• Lack of Interpretability: Most deep learning models function as "black boxes," making it difficult to understand which temporal features drive predictions.
• Inefficiency in Signal Processing: Real-world temporal signals are often heterogeneous, containing sparse informative patterns mixed with redundant observations. Uniform processing ignores this sparsity.

The paper posits that effective temporal modeling should mimic human perception by selectively emphasizing salient variations, maintaining compact historical representations, and identifying underlying trends, rather than processing the entire sequence uniformly.

2. Methodology: SPaRSe-TIME

The proposed framework, SPaRSe-TIME, reformulates time series forecasting as an adaptive decomposition-learn-recombine strategy. Instead of a monolithic sequence model, it decomposes the input time series $X$ into three complementary components: Saliency, Memory, and Trend.

A. Core Decomposition

The input signal is projected into three distinct subspaces:

1. Saliency ($S(X)$): A data-dependent sparsification operator.
   • It identifies informative time steps by computing temporal gradients ($\nabla X_t = X_t - X_{t-1}$).
   • A normalized weight vector $w$ is derived from the gradient magnitude.
   • The signal is projected via a diagonal matrix $D_w$, effectively filtering out redundant time steps and emphasizing abrupt changes or high-frequency variations.
2. Memory ($M(X)$): A low-rank representation of historical context.
   • It exploits the redundancy in temporal sequences by projecting the signal onto a low-dimensional subspace ($k \ll \min(T, d)$) using Singular Value Decomposition (SVD).
   • This captures dominant temporal patterns while suppressing noise, providing a compact summary of the history.
3. Trend ($G(X)$): A low-frequency dynamics encoder.
   • A linear smoothing operator (e.g., moving average) acts as a low-pass filter to extract long-term, smooth trends, separating them from high-frequency noise.

B. Integration and Prediction

The three components are processed through learnable linear transformations ($\phi_i$) and aggregated using adaptive weights ($\alpha_i$):
$$\hat{Y} = \psi \left( \sum_{i \in \{s,m,g\}} \alpha_i \phi_i(Z_i) \right)$$

• Adaptive Weighting: The weights $\alpha_i$ are learned via a softmax function, allowing the model to dynamically adjust the importance of saliency, memory, and trend based on the specific dataset characteristics.
• Output: A final nonlinear mapping $\psi$ produces the prediction.

C. Computational Complexity

The framework avoids sequential recurrence and quadratic self-attention.

• Complexity: $O(kTd)$, where $T$ is sequence length, $d$ is feature dimension, and $k$ is the low-rank dimension.
• Comparison: This is linear in $T$, significantly outperforming Transformers ($O(T^2d)$) and offering comparable efficiency to RNNs ($O(Th^2)$) but with better parallelization potential.

3. Key Contributions

1. Structured Decomposition Framework: Introduces SPaRSe-TIME, which explicitly models time series as a combination of saliency (high-frequency), memory (low-rank patterns), and trend (low-frequency), enabling disentangled temporal reasoning.
2. Saliency-Driven Low-Rank Formulation: Formulates temporal modeling as a projection onto informative subspaces, using saliency as a sparsification operator and low-rank approximation for memory efficiency.
3. Computational Efficiency: Achieves linear computational complexity $O(T)$, avoiding the quadratic cost of attention mechanisms and the sequential bottlenecks of RNNs.
4. Explicit Interpretability: Provides learnable component weights ($\alpha$) that reveal the relative contribution of different temporal structures (e.g., whether a prediction relies more on trends or sudden spikes) across different domains.

4. Experimental Results

The model was evaluated on five diverse real-world datasets: Household Power Consumption, Weather, Netflix Stock, NASDAQ-100, and UCI Air Quality.

• Structured Data (Household & Weather):

  • SPaRSe-TIME outperformed all baselines (RNN, GRU, LSTM, Transformer) in terms of MAE, RMSE, and $R^2$.
  • On the Household dataset, it achieved an $R^2$ of 0.503 vs. 0.385 for the Transformer.
  • On the Weather dataset, it achieved an $R^2$ of 0.568 vs. 0.524 for the Transformer.
  • Interpretability: The model correctly assigned high weight to "Trend" for weather data and "Saliency" for power consumption spikes.

• Highly Stochastic Data (Netflix & NASDAQ):

  • Performance was competitive but generally lower than the Transformer.
  • On NASDAQ, SPaRSe-TIME underperformed the Transformer ($R^2$ of -0.147 vs. -0.007).
  • Analysis: In highly noisy, stochastic environments where clear trends or low-rank structures are absent, the decomposition assumption becomes less valid, and global attention mechanisms (Transformers) remain superior.

• Complex Multivariate Data (Air Quality):

  • The Transformer outperformed SPaRSe-TIME. The paper notes that complex, nonlinear cross-variable interactions in air quality data are not fully captured by the current linear decomposition and aggregation.

• Efficiency:

  • SPaRSe-TIME required only 1.09K parameters and 0.01G FLOPs, significantly fewer than the Transformer (43.35K params, 0.25G FLOPs).
  • It achieved the fastest inference time (0.005ms) among all tested models.

• Ablation Study:

  • Removing the Memory component caused the largest performance drop, confirming its role as the fundamental baseline.
  • The Trend component was crucial for weather data but less critical for household data.
  • Single-component models performed poorly, validating the need for multi-scale modeling.

5. Significance and Conclusion

SPaRSe-TIME offers a principled alternative to monolithic deep learning sequence models. Its significance lies in:

• Bridging the Gap: It successfully balances efficiency, interpretability, and performance.
• Domain Suitability: It is highly effective for structured time series with clear temporal components (trends, seasonality, distinct events).
• Transparency: Unlike black-box models, it explicitly reveals how a prediction is made by quantifying the contribution of trends, memory, and salient events.

Limitations: The framework struggles in highly stochastic, low signal-to-noise ratio environments (like stock markets) and complex multivariate systems with intricate nonlinear dependencies where simple linear decomposition fails.

Future Work: The authors suggest incorporating adaptive saliency mechanisms, dynamic low-rank projections, and integrating lightweight attention or graph-based interactions to handle complex cross-variable dependencies while maintaining efficiency.