Data-quality monitoring of luminosity blocks with ATLAS Open Data: an open benchmark and a learned within-run detector

In the high-stakes world of particle physics, the Large Hadron Collider smashes protons together to reveal the fundamental building blocks of the universe. However, the machine is not perfect, and neither are the massive detectors that record the collisions. Every second, these detectors generate a torrent of data, but not all of it is trustworthy. Sometimes, a sensor glitches, a power supply fluctuates, or a specific region of the detector goes silent. If physicists were to use this flawed data to search for new particles, they might mistake a broken sensor for a discovery. To prevent this, the experiments employ a rigorous quality control system that checks the data in tiny, one-minute intervals. If a problem is found, that specific minute of data is marked as "bad" and discarded. The challenge is that the detectors are so complex and the conditions change so much from one run to the next that spotting these subtle errors is incredibly difficult. The methods used to find them have remained a secret, locked inside the laboratories, making it impossible for scientists outside the collaboration to compare different approaches or improve the system.

A researcher named Renjie Wang has changed this by turning a public release of data from the ATLAS detector into a transparent testing ground. By taking real collision data from 2015 and 2016 and secretly injecting known errors into it, Wang created a benchmark where the "correct" answer is known. This allowed for a fair comparison of different methods to see which ones could best spot the faked problems without crying wolf. The study revealed that the most successful strategy is not to compare every minute of data against a global average, but to look at how each minute differs from the minutes immediately surrounding it. Using this approach, a machine learning system trained on synthetic errors was able to identify 43 percent of the injected problems while keeping false alarms to just one percent. This is a significant improvement over older scanning methods, which only caught 25 percent of the errors under the same strict conditions.

The core of the problem lies in the nature of the data itself. The detector records collisions in "luminosity blocks," which are roughly one-minute windows where the conditions are assumed to be stable. To check if a block is good, scientists usually fill histograms, which are essentially bar charts showing how many particles of a certain type were found at different energies or angles. In the past, researchers tried to compare a new block of data against a reference built from many previous runs. However, Wang's analysis showed that this approach fails because the conditions between different runs vary too much. Factors like the number of overlapping collisions and the specific settings of the detector change from run to run, creating differences in the data that are far larger than the statistical noise of a single minute. When a new block is compared to a distant past run, these normal variations look like defects, causing the system to flag good data as bad.

The solution found in this study is to look much closer to home. Instead of comparing a block to the entire history of the experiment, the system compares it to the other blocks within the same run. Within a single run, the detector settings and the general conditions are fixed, and the only thing that changes smoothly is the number of collisions. By building a reference specifically for each run, the system can filter out the large, normal variations and focus on the sudden, jagged deviations that indicate a real problem. This method transforms the data into a set of "residuals," or differences, that behave much more predictably. It is within these cleaned-up differences that the defects become visible.

To test this, Wang took the public data and secretly injected eight different types of simulated detector failures into the events. These included scenarios like a dead region in the calorimeter where particles simply vanish, a "hot spot" where a noisy cell creates fake particles, or a sudden shift in the energy scale of the measurements. These errors were hidden inside the data, and the goal was to see which computer program could find them. The study compared several approaches, including simple statistical scans and more complex machine learning models. The simple scans, which looked for specific patterns of error, managed to find about a quarter of the problems. However, a machine learning classifier trained on the residuals of the data performed significantly better. This system learned to recognize the subtle, combined signatures of errors across many different histograms, rather than looking for a single obvious pattern.

The results were clear. The machine learning system, when trained on synthetic examples of defects and calibrated using a strict statistical method to ensure a low false-alarm rate, detected 43 percent of the injected errors. This was a substantial leap over the 25 percent detection rate of the best traditional scanning method. The system was particularly effective at finding errors that were spread out across different parts of the data, where no single histogram showed a dramatic spike. However, the study also highlighted the limits of what can be found. Some types of errors, such as a specific module turning off in a way that affects only a tiny fraction of the data, were too subtle to be seen by any method given the amount of data available in a single minute. Furthermore, the system struggled when faced with a type of error it had never seen before. If the training data only included specific examples of dead regions or hot spots, the system would miss a completely different kind of failure.

To address this, the researchers tested a "safety net" approach. They added generic, pattern-agnostic examples of detector failures to the training data—simulations that did not mimic any specific known error but represented the general idea of something going wrong. This adjustment allowed the system to catch 34 percent of the unseen error types, a major improvement over the 18 percent caught by the system trained only on known patterns. The trade-off was a small drop in performance on the known types, but the overall ability to detect unknown problems increased. This suggests that for a real-world detector, where the full list of possible failures is never complete, a system trained on a mix of known and generic failures is the most robust choice.

The study also examined how much historical data is needed to make these systems work. A common concern is that a new detector might not have enough past data to build a reliable reference at the start of a year. The research showed that the machine learning system needs a history of about eight previous runs to outperform the simple scanning methods. With fewer runs, the system is too conservative and misses too many errors, but with a modest amount of history, it quickly becomes the superior tool. This finding is crucial for the practical deployment of such systems, as it means they can be effective relatively early in a data-taking period.

Ultimately, this work demonstrates that the key to monitoring complex scientific instruments is not to look for global anomalies, but to understand the local context. By focusing on the differences within a single run and using machine learning to interpret the subtle patterns of deviation, scientists can catch a much higher percentage of errors without discarding good data. The benchmark created in this study is now public, allowing other researchers to test their own ideas against the same real-world data. This transparency is a vital step forward, moving the field from a collection of isolated, proprietary methods to a shared, competitive environment where the best techniques can be identified and refined for the benefit of all future physics discoveries.