Multi-scale LSTM forecasting with attention-based feature fusion for anomaly detection in multivariate monitoring data

Technical Summary: Multi-scale LSTM Forecasting with Attention-based Feature Fusion for Anomaly Detection

Problem Statement
Engineering systems, including power equipment, industrial processes, and computing infrastructure, are monitored via multivariate time series data. Anomalies in these systems manifest at vastly different temporal scales, ranging from gradual drifts over long periods to transient impulses lasting only a few sampling intervals. Existing detection methods face two primary limitations:

1. Single-Resolution Dependency: Most approaches rely on fixed temporal windows or single-scale representations, making it difficult to distinguish between local spikes, seasonal fluctuations, and global contextual deviations simultaneously.
2. Static Correlation Modeling: While some methods use Graph Neural Networks (GNNs), they often capture only fixed correlations between variables, failing to reflect the dynamic changes in dependencies that occur over time.

Consequently, current systems struggle to detect anomalies that appear differently across temporal resolutions or to adapt to shifting variable correlations, leading to false alarms or missed detections.

Methodology: MAF-LSTM Framework
The paper proposes MAF-LSTM (Multi-scale Attention-based Feature-fusion Long Short-Term Memory), a forecasting-based anomaly detection framework designed to address these challenges. The architecture consists of three core components:

1. Multi-scale Feature Extraction:
   The framework employs a hierarchical LSTM ensemble where parallel branches process input windows of varying lengths ($w_1, w_2, \dots, w_k$). Shorter windows capture local fluctuations and rapid transitions, while longer windows preserve long-range contextual dependencies. Each branch produces a scale-specific hidden representation ($h^{(s)}$), which are concatenated to form a unified feature vector.

2. Attention-based Feature Fusion:
   To integrate these multi-scale representations effectively, a lightweight scale-level attention mechanism is introduced. This module adaptively weights the hidden representations from different scales based on their relevance to the current time step. A fully connected network with a $tanh$ activation generates scale scores, which are normalized via $softmax$ to produce attention weights ($\alpha$). The final fused representation ($H_t$) is a weighted sum of the scale-specific features, allowing the model to dynamically emphasize the most informative temporal scales.

3. Mahalanobis Distance-based Anomaly Scoring:
   Unlike methods that rely solely on point-wise error metrics (e.g., MSE), MAF-LSTM calculates anomaly scores using the Mahalanobis distance of the prediction residuals.

   • The model forecasts the next time step ($\hat{y}_t$) and computes the residual ($r_t = |y_t - \hat{y}_t|$).
   • A global mean vector ($\mu$) and covariance matrix ($\Sigma$) are estimated from the residuals of the validation set (representing normal operation).
   • The anomaly score ($S_t$) is calculated as the Mahalanobis distance: $S_t = \sqrt{(r_t - \mu)^T \Sigma^{-1} (r_t - \mu)}$.
   • This approach accounts for the correlation structure among variables, enabling the detection of "logical" anomalies where variables deviate from their expected joint behavior, even if individual values appear normal.

The model is trained in an unsupervised manner using Mean Absolute Error (MAE) as the objective function to minimize prediction errors without requiring labeled anomaly data.

Key Contributions
The paper outlines three primary contributions:

• Multi-scale Forecasting Perspective: It introduces a multi-scale perspective to forecasting-based anomaly detection, a dimension seldom explored in prior research, enabling the identification of anomalies that manifest differently across temporal resolutions.
• Attention-Enhanced Architecture: It designs an LSTM-based prediction framework integrated with an attention mechanism to capture multi-scale temporal dependencies, aiming to improve accuracy while reducing false alarms and missed detections.
• Empirical Validation: The framework is evaluated on five real-world multivariate datasets (MSL, SMAP, SMD, SWaT, and PSM), demonstrating superior performance compared to state-of-the-art baselines.

Experimental Results
The framework was tested on five benchmark datasets spanning spacecraft telemetry, server monitoring, and industrial control systems.

• Performance: MAF-LSTM achieved the highest F1-score on four of the five datasets (MSL, SMD, SWaT, PSM). On the PSM dataset, it outperformed the Isolation Forest baseline by 12.46 percentage points.
• Comparison: The method achieved an average F1-score improvement of 2.52% over the best-performing baseline (OmniAnomaly) across the tested datasets.
• Ablation Study: Experiments removing specific components confirmed their necessity:
  • Replacing the multi-scale module with a single-scale model reduced the F1-score by 4.44%.
  • Replacing the attention mechanism with equal weighting reduced the F1-score by 4.12%.
  • Replacing the Mahalanobis distance with Euclidean distance caused a significant performance drop (from 90.93% to 74.79% F1-score on MSL), highlighting the importance of modeling residual correlations.

Significance and Limitations
The paper claims that MAF-LSTM effectively bridges the gap in detecting multi-scale anomalies by combining temporal diversity with dynamic feature weighting and correlation-aware scoring. The authors note that the Mahalanobis scoring component is particularly critical, suggesting that modeling the correlation structure of residuals is as important as the temporal representation itself.

However, the authors maintain a modest stance regarding the current scope:

• Data Scope: While motivated by power equipment monitoring, the validation was conducted exclusively on public benchmark datasets from other domains (space, IT, water treatment). Validation on actual in-service power equipment data is noted as ongoing work.
• Fixed Parameters: The window lengths for the multi-scale branches were fixed across all datasets. The authors suggest that adaptive scale selection based on sampling rates and dominant periods could further improve performance.
• Static Covariance: The covariance matrix for Mahalanobis scoring is estimated from training residuals and is not updated online, which may limit robustness over very long operational periods where equipment behavior gradually drifts.
• SMAP Performance: The framework did not achieve the best result on the SMAP dataset, indicating that the benefits of multi-scale forecasting may depend on the specific temporal extent of anomalies within the data.

Future work is directed toward exploring adaptive scale selection mechanisms and extending the framework to real-time streaming environments.