Horizon Activation Mapping for Neural Networks in Time Series Forecasting

Title: Horizon Activation Mapping for Neural Networks in Time Series Forecasting

Problem Statement
Neural network models for time series forecasting have achieved significant performance improvements, particularly in long-horizon tasks. However, the field lacks a unified framework for model selection and interpretability that is agnostic to specific layer types or model families. Existing interpretability methods, such as Grad-CAM, are designed for computer vision and rely on spatial attention maps, which do not translate trivially to the spatio-temporal nature of multivariate forecasting. Furthermore, while various model families (e.g., MLP-based, self-attention, SSM-based, diffusion) exist, there is no large-scale supervised dataset or analytical framework capable of comparing their internal learning dynamics and gradient activities across different architectures. Current model selection often relies on error metrics (MSE, MAE) which may be similar across different families, obscuring the underlying optimization landscapes and learning strategies.

Methodology: Horizon Activation Mapping (HAM)
The authors introduce Horizon Activation Mapping (HAM), a visual interpretability technique inspired by Grad-CAM but adapted for time series forecasting. HAM analyzes gradient update magnitudes across subseries within a forecasting horizon to identify model properties.

• Causal and Anti-Causal Modes: For a horizon of size $H$, HAM calculates gradient norm averages using two masking modes applied to the loss function before backpropagation:
  • Causal Mode ($c$): Includes subseries from timestep 0 to $h$.
  • Anti-Causal Mode ($a$): Includes subseries from timestep $h$ to $H$.
• Gradient Norm Averages: The framework computes the average gradient norm with respect to the model parameters $\theta$ over these masked subseries.
• Visualizations:
  • HAM Plots: Display gradient norm averages against timesteps, compared against a "line of proportionality" representing a uniform distribution of gradient norms.
  • Difference Plots: Show the difference between causal and anti-causal gradient averages, scaled to $[-1, 1]$, to highlight disparities in learning across the horizon.
  • Interpolated Area Plots: Normalize the maximum area per model to the $[0, 1]$ range to compare models of different horizon sizes.
• Key Metrics: The framework identifies the Gradient Equivariant Point, where the gradient norm average of the first $h$ timesteps equals that of the remaining $H-h$ timesteps. It also calculates the signed area between the curves and the line of proportionality to magnify variations.

Experimental Setup
The study evaluates HAM across diverse model families trained on the ETTm2 dataset (60-20-20 train-val-test split) using an NVIDIA A40 GPU. The models include:

• MLP-based: CycleNet, N-Linear, N-HITS.
• Self-Attention: FEDformer, Pyraformer.
• State Space Models (SSM): SpaceTime.
• Diffusion: Multi-Resolution DDPM.

Experiments cover various horizon sizes ($H = \{96, 192, 336, 720\}$) and investigate the effects of:

• Dropout rates.
• Batch sizes.
• Early stopping (epochs post-stopping).
• Dataset splits (train vs. validation vs. test).
• Univariate vs. Multivariate forecasting.
• Architectural hyperparameters (e.g., cycle queue length, normalization strategies).

Key Results and Observations

1. Dropout Effects: In multivariate models (specifically N-HITS), adding dropout (0.2) increased overall gradient magnitudes by more than 3 times, particularly in longer regions of the horizon.
2. Batch Size Dynamics:
   • Gradient magnitudes and error metrics showed complex relationships with batch size.
   • In converged models (N-HITS, N-Linear, CycleNet), gradient norm averages interpolated across batch sizes exhibited a polynomially saturating trend, suggesting a potential exponential approximation across batch sizes per epoch.
   • CycleNet showed subtle performance differences based on batch size, with anti-causal activity increasing at smaller batch sizes (50) and causal activity decreasing at larger ones (100).
3. Early Stopping: Models stopped early showed reduced gradient magnitudes, particularly in the causal mode. The gradient equivariant point of early-stopped models shifted closer to $H/2$, suggesting that activities in longer horizon regions did not improve performance once the model reached a local minimum.
4. Dataset Splits: Training data pushed curves closer to the line of proportionality. Validation sets in N-HITS showed "out-of-distribution" aspects more distinctly than in CycleNet.
5. Univariate vs. Multivariate: Multivariate forecasting presented greater spatial learning difficulties. The removal of initial timesteps in multivariate subseries pushed anti-causal curves below the line of proportionality, a behavior not seen in univariate cases.
6. Architecture Specifics:
   • N-Linear: Normalization (subtracting the last timestep's value) reduced errors by 16% and altered gradient magnitudes, decreasing them causally in the first 23% of the horizon and increasing them thereafter.
   • CycleNet: The cycle queue length influenced the ordering of values. The smallest cycle length (64) corresponded to the best performance and showed larger magnitudes interplaying between subseries lengths.
   • SpaceTime: Exhibited exponential autoregressive activities. Its causal mode transitioned from linear (at $H=96$) to exponential trends (at $H=\{192, 336, 720\}$), highlighting longer regions of the horizon.
   • N-HITS: Its behavior aligned with the neural approximation theorem, with anti-causal increases supporting infinite horizon assumptions.
   • Diffusion (Multi-Resolution DDPM): Showed characteristically larger gradient norms for smaller subseries compared to proportional expectations, suggesting a property of diffusion-based forecasting.

Significance and Claims
The paper claims that HAM provides a granular, model-agnostic framework for:

• Model Selection: Comparing learning strategies and optimization landscapes across different neural network families (MLP, Attention, SSM, Diffusion) beyond simple error metrics.
• Validation Set Choice: Using relative variabilities in HAM plots to select appropriate validation sets.
• Architectural Insight: Attributing specific trends in HAM plots to theoretical properties, such as N-HITS' neural approximation theorem and SpaceTime's exponential autoregressive nature.
• Generalizability: The framework is applicable regardless of the layer types used in the model, addressing the limitation of previous methods that were specific to certain architectures.

The authors note that while HAM uses overall norm averages, layer-wise plots can further elucidate learning dynamics for shorter and longer subseries. The study concludes that HAM serves as a tool to understand the "optimization landscape" of time series forecasting, enabling interpretable architecture changes and cross-family comparisons.