Continual Learning via Ensemble-Based Depth-Wise Masked Autoencoders for Data Quality Monitoring in High-Energy Physics

This paper introduces DepthViT, a lightweight masked autoencoder combined with an ensemble-based continual learning framework, to achieve robust and precise anomaly detection for data quality monitoring in High-Energy Physics by effectively adapting to distributional shifts in dynamic data streams.

The Gist

The Big Picture: The "Forever-Listening" Ear

Imagine you are a security guard at a massive, high-tech factory (the CMS detector at CERN). Your job is to listen to the machines and spot any weird noises that mean something is broken. This is called Data Quality Monitoring (DQM).

In the past, you might have used a simple checklist. But now, we use Machine Learning (ML)—a super-smart computer program that learns what "normal" sounds like and screams "ALARM!" when it hears something strange.

The Problem:
The factory is huge and operates in extreme conditions (freezing cold, strong magnets, radiation). Over time, the machines naturally age and change. The "normal" sound from 2018 is slightly different from the "normal" sound in 2022.
If you train your security guard (the AI) only on the 2018 sounds, and then try to use them in 2022, the guard will get confused. They might think a perfectly healthy machine is broken (false alarm) or miss a real broken machine because it sounds "too different" from what they learned years ago. This is called Model Degradation.

The Solution:
The authors of this paper created a new system called DepthViT combined with a Continual Learning Ensemble. Think of it as upgrading your security team from a single, stubborn guard to a dynamic, rotating squad of experts who never forget their training.

---

1. The New Detective: DepthViT

Traditional AI models for looking at images (like photos) treat all parts of the image the same way. But the data from this particle detector is special. It has layers (depths), and the physics in one layer doesn't always look like the physics in another.

• The Analogy: Imagine looking at a layered cake. A standard AI looks at the whole cake as one big picture. DepthViT is like a detective who knows that the frosting, the sponge, and the filling are different materials. It looks at each layer separately but understands how they talk to each other.
• Why it matters: This makes the AI lightweight (it's tiny compared to other models, using 1/100th of the computer power) but incredibly smart at spotting the specific "layered" anomalies in the detector.

2. The Strategy: The "Rotating Squad" (Ensemble Learning)

The authors realized that one single AI model can't handle every change in the factory. So, they built a team.

• The Old Way: You train one model, and it stays the same forever. When the factory changes, the model fails.
• The New Way (The Ensemble): Imagine a team of detectives.
  • Detective A was trained on the factory conditions from last month.
  • Detective B was trained on conditions from last week.
  • Detective C is trained on today's conditions.
• How they work together: When a new sound comes in, they all listen. If anyone in the team says, "Hey, that sounds weird!", the whole team raises the alarm.
• The Magic: Because the team includes experts from the past and the present, they can handle both small changes (like a machine warming up) and huge changes (like a power outage in one section of the factory). If the "new" detective misses something because the data is too weird, the "old" detective might catch it, and vice versa.

3. The "Gap" Trick

How do they decide if a sound is actually an anomaly?

• They calculate a "weirdness score" (Z-score) for every piece of data.
• Usually, all the scores are clustered together (everything is normal).
• If there is a big gap between the highest score and the second-highest score, it means one specific thing is acting totally different from the rest.
• Analogy: Imagine a choir singing. If everyone sings at a volume of 50, and one person suddenly sings at 100, that's a huge gap. The system spots that gap and flags it as an anomaly.

4. The Results: Why This Matters

The team tested this on real data from the Large Hadron Collider (LHC) spanning several years.

• Without the new system: The AI started failing badly when the data changed slightly. It missed broken machines and cried wolf too often.
• With the new system (DepthViT + Rotating Squad): The system maintained 99% accuracy even when the data changed drastically between 2018 and 2022. It didn't forget the old rules, and it learned the new ones instantly.

The Takeaway for Everyone

This isn't just about particle physics. This is a blueprint for the future of Industrial AI.

Think about your car, a hospital, or a factory. Sensors age, weather changes, and new failure modes appear. If you rely on a single AI trained on old data, it will eventually fail. This paper shows us how to build AI teams that evolve with time, keeping a "memory" of the past while adapting to the present, ensuring our systems stay safe and efficient forever.

In short: They built a tiny, smart AI detective and put it in a team where the members rotate based on the current conditions. This team never gets confused by change, ensuring the "factory" (the particle collider) keeps running smoothly.

Technical Summary

1. Problem Statement

In High-Energy Physics (HEP), specifically within the Compact Muon Solenoid (CMS) experiment at CERN, Data Quality Monitoring (DQM) is critical to ensure detectors produce reliable data for physics analysis. Traditional Machine Learning (ML) approaches for Anomaly Detection (AD) in DQM face a significant challenge: distributional shifts.

• The Issue: HEP detectors operate in harsh environments (radiation, cryogenic temperatures, magnetic fields) leading to gradual or abrupt degradation of components. Additionally, machine parameters (like luminosity) change over time.
• The Consequence: ML models trained on static datasets suffer from model degradation and catastrophic forgetting when deployed on new data streams. They either fail to detect anomalies (high False Negative Rate) or flag normal data as anomalous (high False Positive Rate) due to shifts in the underlying data distribution between different "runs" or years of data collection.
• The Gap: Existing Continual Learning (CL) methods (e.g., regularization, experience replay, single-model adaptation) often struggle with large, unpredictable shifts or incur high computational/memory costs.

2. Methodology

The authors propose a two-pronged solution: a novel lightweight architecture (DepthViT) and a robust Ensemble-Based Continual Learning (CML) framework.

A. DepthViT Architecture

DepthViT is a lightweight Masked Autoencoder (MAE) designed specifically for the unique structure of CMS Hadron Calorimeter (HCAL) data.

• Depth-Wise Embeddings: Unlike standard Vision Transformers (ViT) that use shared convolutional kernels across channels (assuming spatial symmetry like RGB images), DepthViT uses depth-wise convolutions. This acknowledges that different depths in the HCAL detector represent distinct physical layers with different particle shower profiles, not just different color channels of the same point.
• Cross-Depth Attention: Instead of standard self-attention across the patch sequence, DepthViT employs a depth-wise attention mechanism. It computes attention weights along the channel (depth) dimension rather than the sequence dimension. This allows the model to learn relationships between detector depths while saving parameters.
• Efficiency: The architecture is extremely lightweight (~300k parameters) compared to standard ViT-B/16 (86M parameters), making it ideal for ensembling.

B. Anomaly Detection Mechanism

• Z-Score Calculation: The model reconstructs input data. Anomalies are detected by calculating the reconstruction error ($pred_{err}$). A Z-score is computed over a time window ($T$) of Lumisections (LS) using the mean ($\mu_{err}$) and standard deviation ($\sigma_{err}$) of errors from pristine data.
• Gap-Score Thresholding: Instead of a fixed threshold, the system uses a Gap-Score ($G$). It calculates the difference between the largest and second-largest Z-scores in a distribution. If $G > G_0$, the data is flagged as anomalous. This method is robust against degraded models that might shift the entire distribution, as it focuses on outliers.
• Dual Scaling: To handle different anomaly types (dead channels vs. hot channels), the system processes data through two parallel scaling pipelines: Max Scaling (sensitive to low values) and Quantile Scaling (sensitive to high values).

C. Continual Learning Strategy (Ensembling)

The core innovation is an ensemble strategy that combines static historical models with dynamic recent models:

1. Model Generation: New DepthViT models are trained on the most recent data runs.
2. Ensemble Construction: The system maintains an ensemble of models (e.g., 4 models) trained on different historical runs. As new data arrives, the oldest model is retired, and the newest is added.
3. Dual Adaptation:
   • Statistical Update: The $\mu_{err}$ and $\sigma_{err}$ baselines for all models in the ensemble are updated using the validation data of the current run (without retraining weights).
   • Logical OR Aggregation: The ensemble output is determined by a logical OR. If any model in the ensemble flags the data as anomalous, the final output is anomalous. This ensures high recall (low False Negatives) while the ensemble diversity maintains precision.

3. Key Contributions

1. DepthViT Architecture: A novel, parameter-efficient masked autoencoder that respects the physical depth structure of HCAL data, reducing parameters by ~99% compared to standard ViTs.
2. Ensemble-Based CML Framework: A strategy that decouples plasticity (adaptation to new data via new models) from stability (retention of past knowledge via older models) without requiring complex regularization or data replay.
3. Gap-Score Detection: A robust anomaly detection metric that avoids the pitfalls of fixed thresholds in non-stationary environments.
4. Dual-Scaling Pipeline: A preprocessing technique that simultaneously optimizes sensitivity to both "dead" (low signal) and "hot" (high signal) channel anomalies.

4. Results

The method was evaluated on CMS HCAL occupancy maps from 2018 (Run 2) and 2022 (Run 3), covering both small shifts (luminosity changes) and large shifts (detector failures/power loss).

• Baseline Performance: A single DepthViT model trained on 2018 data achieved >99% precision on 2018 data but suffered severe degradation on 2022 data (FNR rose to ~50-75% for various anomaly factors).
• Statistical Update Only: Updating $\mu_{err}$ and $\sigma_{err}$ improved performance significantly but was insufficient for large shifts.
• Ensemble Performance (Combined Approach): The proposed ensemble method (combining model ensembling with statistical updates) achieved:
  • Precision: >98% across all anomaly factors.
  • Recall: >99% for strong anomalies (factors 0.0, 1.5, 2.0) and >89% for subtle anomalies (factor 0.8).
  • FPR/FNR: Both False Positive and False Negative rates were significantly lower than single-model baselines or ensembles without statistical updates.
  • Comparison: The ensemble improved the False Negative Rate by 100% for strong anomalies and 11% for subtle anomalies compared to using only the latest model.

5. Significance

• Operational Resilience: The system provides a path toward adaptive DQM that can sustain operation in dynamic data environments without human intervention or retraining on historical data.
• Scalability: The lightweight nature of DepthViT allows for the deployment of large ensembles with minimal computational overhead, as the inference is trivially parallelizable.
• Broader Applicability: While demonstrated in HEP, the approach is directly applicable to industrial monitoring (e.g., manufacturing lines with aging sensors) where data distributions naturally evolve over time.
• Generalization: The DepthViT architecture offers a new paradigm for processing multi-channel data where channels represent distinct physical dimensions rather than redundant features, applicable to spectral analysis and multi-channel optical data.