Scalable Temporal Anomaly Causality Discovery in Large Systems: Achieving Computational Efficiency with Binary Anomaly Flag Data

Technical Summary: Scalable Temporal Anomaly Causality Discovery in Large Systems

Problem Statement
In large-scale industrial and scientific monitoring systems, such as the Compact Muon Solenoid (CMS) experiment at CERN, fault diagnosis requires identifying the root causes of anomalies. While anomaly detection (AD) systems are effective at flagging faults, determining the causal relationships between variables across multiple subsystems is challenging. Existing Graphical Causal Model (GCM) learning methods, particularly those designed for time-series data like PCMCI (Peter-Clark Momentary Conditional Independence), face significant computational burdens when applied to large datasets. Furthermore, modern monitoring systems often generate data in the form of binary anomaly flags (0 for normal, 1 for alert). These binary datasets present unique challenges:

1. Data Sparsity and Multicollinearity: Anomalies are rare events, leading to long periods of uniform status (zeros). This sparsity causes multicollinearity, where variables appear highly correlated simply due to long uniform regimes rather than true causal links, degrading the accuracy of conditional independence (CI) tests.
2. Computational Intractability: The computational cost of learning causal graphs grows rapidly with the number of variables and sample size, making real-time application infeasible for large systems.
3. Binary Data Characteristics: Standard CI tests often fail to distinguish the significance of the transition from 0 to 1 (anomaly occurrence) from the association of zeros, leading to spurious causal inferences.

Methodology: AnomalyCD
The authors propose AnomalyCD, a scalable framework designed to generate temporal GCMs from binary anomaly flag data. The approach integrates an online anomaly detection module with a specialized causal discovery pipeline that addresses the specific characteristics of binary flag data.

1. Online Anomaly Detection (AD):

   • The system employs an ensemble of unsupervised AD algorithms to generate binary anomaly flags for variables not covered by pre-trained models.
   • It decomposes time-series data into trend, seasonal, and residual components.
   • Three specific detectors are combined via bitwise union:
     • Temporal Transient ($\theta$): Uses a sliding z-score on residuals to detect sudden outliers.
     • Temporal Trend ($\iota$): Uses a cumulative sum algorithm on the trend component to detect gradual drifts.
     • Spectral ($\phi$): Uses Spectral Residual (SR) saliency detection to identify frequency or data rate changes.

2. Anomaly-Aware Causal Discovery:
   The core of AnomalyCD modifies the standard PCMCI algorithm through four key strategies:

   • Anomaly-Aware CI Test (ANAC): Instead of standard tests, the authors propose a test that focuses on positive associations corresponding to anomaly occurrences (0 $\to$ 1 transitions). It utilizes partial correlation with positive Pearson's correlation on residuals, explicitly excluding links with non-positive correlation weights to filter out spurious associations driven by uniform zero states.
   • Sparse Data Handling (SDH): To reduce the sample size $n$ and computational load, the algorithm compresses long regions of uniform anomaly status. It retains the first $l_m$ indices of uniform regions to preserve time-lag causality while discarding redundant samples, significantly reducing the dataset size without losing transition information.
   • Sparse Link Handling (SLH): Before running the full causal search, the algorithm compresses the initial search space (prior link assumption). It excludes self-lags and filters out variable pairs that do not exhibit temporal co-occurrence of anomaly flags above a threshold, reducing the number of conditional tests required.
   • Edge Pruning and Adjustment: A post-processing step removes spurious edges (multiple time-lags, bidirected, or cyclic links) generated by the PCMCI. It retains the link with the highest weight or earliest time-lag. For bidirected edges at time-lag $t=0$, a chi-square test is employed to determine directionality where correlation-based tests fail.

3. Bayesian Network (BN) Modeling:
   The resulting temporal DAG is reformulated into a Bayesian Network to enable probabilistic causality inference. The time-series data is "unrolled" to create nodes for specific time-lags, allowing the BN to query conditional probabilities of anomaly occurrences given observed evidence on other sensors.

Key Contributions

• AnomalyCD Framework: A novel approach specifically tailored for learning causal graphs from binary temporal anomaly flags, addressing the limitations of existing methods in handling data sparsity and binary transitions.
• Compression Algorithms: The introduction of Sparse Data Handling (SDH) and Sparse Link Handling (SLH) algorithms that drastically reduce the computational complexity of GCM learning by compressing the data size and the search space of potential edges.
• Anomaly-Aware Testing: The development of the ANAC test, which improves accuracy by focusing on the causal significance of anomaly transitions rather than general statistical associations.
• Empirical Validation: The framework is validated on two datasets:
  1. CMS HCAL Readout Box (RBX): Real-world sensor data from the Hadron Calorimeter at CERN, involving 12 diagnostic sensors across 4 readout modules.
  2. EasyVista: A public IT infrastructure monitoring dataset.

Results

• Computational Efficiency: On the CMS HCAL dataset (approx. 400,000 samples), the standard PCMCI failed due to processing time. AnomalyCD, utilizing SDH, reduced the data to ~900 samples (99.8% reduction), enabling execution in 9 seconds with 14 MB memory usage. On the EasyVista dataset, processing time was reduced by ~91% (from ~75s to ~6.6s) with a 55% data reduction.
• Accuracy Improvements:
  • HCAL: AnomalyCD achieved a 26.2% relative gain in F1-score and a 74.9% reduction in False Positive Rate (FPR) compared to baseline PCMCI with sparse handling. The Structural Hamming Distance (SHD) improved by 70.6%.
  • EasyVista: The approach showed a 20.5% relative gain in F1-score and a 47% reduction in FPR compared to standard PCMCI.
• Causal Inference: The BN model successfully quantified conditional probabilities. For instance, the probability of an anomaly in sensor Q1T increased from 0.05 to >0.90 when related sensors (Q2-4T) showed anomalies, aligning with the physical circuit connections of the HCAL system.

Significance and Claims
The paper claims that AnomalyCD provides a scalable solution for root cause analysis in large, complex systems where data is primarily available as binary anomaly flags. By integrating anomaly-aware testing and data compression, the method achieves a balance between computational efficiency and causal accuracy that existing state-of-the-art methods lack for this specific data type. The authors state that the tools are currently deployed at CERN's Large Hadron Collider to provide diagnostics for calorimeter systems. The study emphasizes that their approach facilitates real-time diagnostic tasks across various subsystems, offering a flexible, unsupervised, and lightweight alternative to deep learning models that require extensive annotated data. The work highlights that while existing methods struggle with the "curse of dimensionality" and binary data sparsity, AnomalyCD effectively mitigates these issues to produce causal graphs consistent with physical system interconnections.