A Hybrid CNN–SPN Framework for Reliable Monitoring and Instability Detection in Hadoop Clusters

Technical Summary: A Hybrid CNN–SPN Framework for Reliable Monitoring and Instability Detection in Hadoop Clusters

Problem Statement
Reliable monitoring of Hadoop clusters is critical due to the complex, concurrent, and dynamically loaded nature of distributed environments. Node instability—arising from overload, disk/network congestion, task anomalies, or recovery delays—can rapidly degrade system performance and availability. Existing monitoring approaches face complementary limitations:

• Rule-based/Threshold methods: Simple to deploy but lack adaptability to changing workloads and rely heavily on manual parameter selection.
• Statistical/Stochastic models (e.g., Markov chains, SPNs): Offer formal mechanisms for state transitions and uncertainty but often rely on restrictive assumptions and lack predictive flexibility.
• Machine Learning/Deep Learning: Provide strong predictive performance but often operate as "black boxes," sacrificing interpretability and failing to capture the structural causes of abnormal behavior.

The paper argues that a hybrid framework is necessary to combine the predictive learning capabilities of deep learning with the formal, interpretable stochastic modeling of event-driven dynamics.

Methodology
The authors propose a Hybrid Convolutional Neural Network–Stochastic Petri Net (CNN–SPN) framework. This approach integrates data-driven pattern learning with formal stochastic modeling to classify Hadoop nodes as stable or unstable.

1. Multi-Density Stochastic Petri Net (SPN):

   • The SPN models node-state evolution (Idle, Busy, Overloaded, Failed, Recovering) using a set of places ($P$) and transitions ($Tr$).
   • Multi-Density Firing: Unlike traditional SPNs using a single distribution, this model employs heterogeneous firing-time distributions for transitions: Exponential, Normal, Log-Normal, Poisson, and Weibull. This allows the modeling of diverse event timings (e.g., task execution, I/O, packet loss, failures).
   • Token Types: Seven distinct token types represent specific operational aspects: available capacity, active tasks, overload intensity, concurrent I/O, network contention, failures, and recovery status.
   • Synthetic Data Generation: The SPN is used to generate a reproducible synthetic dataset of 10,000 node observations, including rare and extreme scenarios difficult to capture in real logs alone.

2. Convolutional Neural Network (CNN):

   • The CNN extracts discriminative latent features ($H(t)$) from observed system metrics (CPU, memory, I/O, network) and the stochastic traces generated by the SPN.
   • The architecture utilizes 2–3 convolutional layers with pooling, followed by dense blocks, trained with Adam optimization.

3. Hybrid Coupling Mechanisms:

   • Hybrid Inference with Stochastic Token Timing (HI-tk): A fusion mechanism that combines SPN state information with CNN latent representations.
   • Dynamic Firing Strategy (HFiringT): A gating mechanism that dynamically adapts transition firing based on observed system states, strengthening the interaction between formal state evolution and learned decision signals.
   • Loss Function: The model is trained using Cross-Entropy Loss that explicitly incorporates the stochastic firing times of tokens ($HFiringT$) alongside latent CNN activations.

Key Contributions

1. Hybrid Framework: Integration of CNN and multi-density SPN to combine pattern learning with formal stochastic modeling for Hadoop monitoring.
2. Multi-Density SPN Design: Utilization of five different probability distributions to represent diverse event timings and generate controlled synthetic traces for rare scenarios.
3. Coupling Mechanisms: Introduction of HI-tk fusion and HFiringT dynamic gating to link formal state evolution with learned decision processes.
4. Reproducible Pipeline: A complete simulation and evaluation pipeline, including a public synthetic dataset (10,000 observations) and an ablation study quantifying the incremental benefits of SPN integration, fusion, and dynamic gating.
5. Empirical Analysis: Demonstration of improved detection performance and interpretability over a CNN-only baseline, while explicitly acknowledging the need for real-cluster validation.

Results
The framework was evaluated on a synthetic dataset of 10,000 observations. An ablation study compared four configurations: CNN-only, CNN+SPN, CNN+SPN+HI-tk, and the full Proposed CNN–SPN.

• Performance Metrics: The complete framework achieved the best overall scores:
  • Accuracy: 68.10%
  • Precision: 68.91%
  • Recall: 68.78%
  • F1-Score: 68.85%
  • ROC-AUC: 0.7434
• Ablation Findings:
  • Integrating the SPN state significantly improved performance over the CNN-only baseline (e.g., ROC-AUC increased from ~0.63 to ~0.72).
  • Adding the HI-tk fusion further enhanced classification accuracy.
  • The full model with the HFiringT dynamic gate achieved the highest mean performance across all metrics.
  • Statistical Significance: A paired t-test confirmed that adding SPN state and HI-tk fusion yielded statistically significant improvements ($p < 0.05$) over the CNN baseline. However, the incremental gain of the HFiringT gate over the CNN+SPN+HI-tk variant was not statistically significant across 5 random seeds, though it consistently showed higher mean performance.
  • Comparison with Baselines: On this specific synthetic dataset, the proposed model performed comparably to classical baselines (Logistic Regression, SVM, Random Forest), with an SVM achieving a marginally higher AUC (0.7500 vs. 0.7434). The authors note that the primary value of their approach lies in interpretability and structural grounding rather than raw accuracy dominance on this specific dataset.

Significance and Claims
The paper claims that the proposed framework provides an interpretable and structurally grounded basis for stochastic monitoring and node-instability detection. Key advantages highlighted include:

• Explainability: The HI-tk fusion weight and HFiringT gate offer insight into whether a prediction is driven by CNN-extracted patterns or SPN-derived event states, unlike opaque black-box classifiers.
• Robustness to Rare Events: The ability to generate synthetic traces with controlled probability distributions allows the model to generalize to under-represented operating regimes (e.g., fast-firing/high-load scenarios).
• Dynamic Adaptation: The framework supports online adaptation of SPN parameters and decision thresholds based on live metrics.

Limitations and Future Work
The authors explicitly state several limitations:

• Synthetic Data: The evaluation relies entirely on SPN-generated synthetic data; no real Hadoop cluster traces were used for training or testing. Consequently, the reported metrics reflect the model's ability to recover the specific stochastic structure of the simulation, not necessarily real-world performance.
• Statistical Modesty: The specific added value of the HFiringT dynamic gate (beyond the SPN state and HI-tk fusion) is not yet statistically confirmed and requires further evaluation with larger datasets or more seeds.
• Aggregate State: The current dataset uses an aggregate event-count signal rather than individually labeled firings for each transition, limiting the quantitative evaluation of per-transition explainability.
• Offline Evaluation: Dynamic adaptation was tested offline on static splits, not in a live streaming environment.

The paper concludes that while the hybrid design offers a promising trade-off between performance and interpretability, validation on real Hadoop traces and systematic comparison against a broader range of classical baselines on real-world data are necessary future steps.