Anomaly Detection for Automated Data Quality Monitoring in the CMS Detector

The paper introduces "AutoDQM," an automated data quality monitoring system for the CMS detector that utilizes unsupervised machine learning and statistical techniques to identify anomalous data at a rate 4 to 6 times higher than that of good data, thereby enhancing the rapid assessment of detector performance.

The Gist

Imagine the CMS detector at CERN as a massive, incredibly complex digital camera the size of a cathedral. It takes billions of "photos" of particle collisions every second to study the fundamental building blocks of the universe.

However, just like a real camera, this giant machine can get dirty, have a cracked lens, or suffer from a dead battery. If the camera is broken, the photos are useless for science.

In the past, checking if the camera was working meant having a team of human experts (called "shifters") stare at thousands of graphs and charts every day, looking for anything that looked "weird." It was like trying to find a single typo in a million-page book by reading every word with your eyes. It was exhausting, slow, and easy to miss mistakes.

This paper introduces AutoDQM, a new "smart assistant" that does the checking for them.

The Problem: The Needle in the Haystack

Every day, the CMS detector produces a mountain of data. Most of it is perfect ("Good Data"). But sometimes, a part of the detector glitches, creating "Bad Data."

• The Old Way: Humans had to look at every single graph to find the glitches. If they missed one, the bad data could ruin months of scientific research.
• The New Way: AutoDQM is an automated system that uses math and artificial intelligence to scan the data instantly and shout, "Hey, something looks wrong here!"

How AutoDQM Works: The Three Detectives

AutoDQM doesn't rely on just one method; it uses three different "detectives" to spot problems.

1. The Statistician (The Beta-Binomial Test)

The Analogy: Imagine you have a favorite playlist of songs you listen to every day. You know exactly how many times you listen to each song. One day, you notice you listened to "Song A" 10 times, but "Song B" zero times. That's weird!

• How it works: AutoDQM compares today's data graph to graphs from "good" days in the past. It uses a special math formula to calculate the odds of today's data happening by chance. If the odds are too low (like listening to a song 10,000 times in a row), it flags it as an anomaly.
• The Magic: It can look at a graph with millions of dots and instantly say, "This specific cluster of dots is missing," even if the human eye can't see the difference.

2. The Pattern Finder (Principal Component Analysis - PCA)

The Analogy: Think of a fingerprint. Every person's fingerprint has a unique pattern, but they all share the general shape of a fingerprint. If you see a handprint that looks like a fingerprint but has a giant hole in the middle, you know it's not a normal fingerprint.

• How it works: The system learns what a "normal" data graph looks like by studying thousands of good examples. It creates a mental "average" of what good data should be. When a new graph comes in, it tries to fit it into that average. If the graph doesn't fit the pattern (like a fingerprint with a hole), the system flags it.

3. The Artist (Neural Network Autoencoder)

The Analogy: Imagine an artist who is so good at copying a painting that they can recreate it from memory. If you give them a photo of a broken vase, they will try to "reconstruct" it as if it were whole. When they compare their perfect reconstruction to the broken photo, the cracks are obvious.

• How it works: This is a type of AI. It looks at a data graph, compresses it into a simple summary, and then tries to "draw" the graph again from that summary. If the AI draws a perfect version but the original data was messy or broken, the difference between the "drawing" and the "original" reveals the problem.

The Results: A Superpower for Scientists

The team tested this system on all the data collected in 2022. Here is what they found:

• Speed & Accuracy: AutoDQM found bad data 4 to 6 times more often than random chance would suggest.
• Fewer False Alarms: It rarely cried "wolf" when everything was fine. It only flagged bad data when it was actually broken.
• Visual Clarity: Instead of just saying "Error," the system highlights the exact spot on the graph where the problem is (like putting a red circle around a typo). This helps the human experts fix the machine immediately.

Why This Matters

In the world of particle physics, time is money, and bad data is a waste of both.

• Before: Humans might miss a broken detector part for hours, wasting valuable time.
• After: AutoDQM spots the issue in seconds, allowing experts to fix it before too much bad data is collected.

In short: AutoDQM is like giving the CMS detector a pair of super-vision glasses and a brain that never gets tired. It watches the data 24/7, spots the tiniest glitches, and lets the human scientists focus on the big discoveries rather than staring at boring charts.

Technical Summary

1. Problem Statement

The Compact Muon Solenoid (CMS) experiment at the CERN Large Hadron Collider (LHC) generates massive amounts of data from high-energy proton-proton collisions. Ensuring the quality of this data is critical for precise physics measurements and new physics searches.

• Current Limitations: Traditional Data Quality Monitoring (DQM) relies on human "shifters" visually inspecting thousands of histograms (representing detector performance) in real-time and offline. This process is labor-intensive, prone to human error, and fatiguing, especially when comparing hundreds of histograms against reference runs.
• The Challenge: Approximately 2–5% of collected data is often designated as "bad" due to detector malfunctions or reconstruction issues. Without automated tools, these issues might go unnoticed for extended periods, wasting valuable beamtime. The goal is to develop a robust, automated system that can rapidly identify anomalous data without requiring explicit "bad" data labels for training.

2. Methodology: The AutoDQM System

The authors introduce AutoDQM, a web-based service that employs a generalized approach to automated DQM using statistical techniques and unsupervised machine learning (ML). The system evaluates both online (Level-1 Trigger) and offline histograms.

A. Statistical Anomaly Detection

The system uses a beta-binomial probability function to compare a data run against one or more "reference" runs (historical "good" data).

• Likelihood Calculation: For each bin $i$ in a histogram, the number of entries $d_i$ is treated as a frequency. The likelihood $L_i$ of observing $d_i$ given a reference run with entries $r_i$ is calculated using the beta-binomial function.
• Pull Values: The relative likelihood ($L_{rel}$) is converted into a "pull value" ($Z_i$) in units of standard deviations.
• Metrics: Two primary metrics are derived:
  1. $\chi^2$ Score: The sum of squared pull values normalized by the number of bins.
  2. $Z'_{max}$: The modified maximum pull magnitude, adjusted for the "look-elsewhere effect" to account for multiple bin comparisons.
• Visualization: Results are displayed as heat maps (for 2D histograms) or overlays (for 1D histograms), highlighting regions of significant excess (red) or deficit (blue).

B. Unsupervised Machine Learning

To detect anomalies without specific "bad" data labels, AutoDQM uses two unsupervised algorithms trained exclusively on "good" data:

1. Principal Component Analysis (PCA):
   • Reduces the dimensionality of histograms (flattened 1D or 2D) into key components.
   • Reconstructs the input histogram from the latent space.
   • Anomaly Score: Calculated as the $\chi'^2$ (modified chi-squared) between the original and reconstructed histograms. Anomalous features not captured by the principal components result in high reconstruction errors.
2. Neural Network Autoencoders (AE):
   • Uses an encoder-decoder architecture with 1D convolutional layers to compress and reconstruct input histograms.
   • Anomaly Score: Similar to PCA, the score is based on the reconstruction error (Sum of Squared Errors, scaled to mitigate statistical fluctuations).
   • Note: While AEs were tested, they were excluded from the final global assessment in this study because they failed to properly reconstruct certain classes of Level-1 Trigger (L1T) histograms even in good runs.

3. Key Contributions

• Generalized Framework: AutoDQM provides a unified framework applicable to various subdetector systems (ECAL, HCAL, Muons) without needing system-specific tuning for every new anomaly type.
• Unsupervised Learning: By relying on unsupervised methods, the system avoids the scarcity of labeled "bad" data and remains agnostic to the specific nature of future anomalies.
• Visual Interpretability: The system does not just output a binary "pass/fail"; it generates heat maps and difference plots that allow experts to immediately locate the geometric or kinematic source of the anomaly (e.g., a specific detector chamber or momentum range).
• Robustness to Pileup: The methodology accounts for varying collision conditions (pileup) by using multiple reference runs or training on datasets spanning the full range of pileup conditions.

4. Results

The system was evaluated using the full 2022 CMS proton-proton collision dataset (36 fb$^{-1}$), comprising 265 "good" runs and 43 "bad" runs (labeled independently by the CMS Physics Performance and Datasets group).

• Detection Performance:
  • The combined approach (Beta-binomial statistical tests + PCA) identified over 50% of the "bad" runs (those with significant detector malfunctions).
  • The false positive rate was kept low, flagging less than 12–15% of "good" runs as anomalous.
  • Discrimination Power: In bad runs, the system flagged 4 to 6 times more histograms as anomalous compared to good runs.
• Algorithm Comparison:
  • Statistical tests performed significantly better when comparing against multiple reference runs (e.g., 8 runs) rather than a single one, as this naturally accounts for variations in pileup conditions.
  • PCA showed strong discrimination, while the Autoencoder was less effective for the specific L1T histograms tested in this study.
• Case Study (Muon Detectors):
  • In a specific case involving Cathode Strip Chambers (CSCs), AutoDQM successfully identified a run where a dozen chambers malfunctioned simultaneously.
  • The system highlighted the specific geometric regions with low muon track occupancy, allowing experts to pinpoint the issue immediately, whereas the anomaly was nearly invisible in standard DQM GUIs.

5. Significance

• Operational Efficiency: AutoDQM significantly reduces the cognitive load on human shifters by filtering out normal variations and focusing attention only on confirmed discrepancies.
• Scalability: As the LHC increases in luminosity and complexity (e.g., High-Luminosity LHC), the volume of data will make manual monitoring impossible. AutoDQM offers a scalable solution.
• Physics Impact: By rapidly identifying and isolating "bad" data, the system ensures that physics analyses are performed on high-quality datasets, preventing the dilution of results by faulty detector data.
• Future Outlook: The authors plan to extend AutoDQM to additional CMS subdetector systems, further automating the data quality assurance pipeline for future physics runs.