⚛️ high-energy experiments

Uncertainty Quantification and Propagation for ACORN, a geometric deep learning tracking pipeline for HEP experiments

This paper presents an uncertainty quantification framework applied to the ACORN geometric deep learning tracking pipeline using the TrackML dataset, demonstrating that model performance is well-calibrated and that aleatoric uncertainty dominates as training data increases, indicating the model has reached its full potential.

Original authors: Lukas Péron, Paolo Calafiura, Xiangyang Ju, Jay Chan

Published 2026-10-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lukas Péron, Paolo Calafiura, Xiangyang Ju, Jay Chan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the heart of modern physics, giant machines smash particles together at speeds close to that of light, creating a chaotic spray of debris that scientists must sort through to understand the fundamental laws of the universe. These experiments, like the Large Hadron Collider, generate an overwhelming amount of data, where billions of collisions happen so frequently that their signals overlap, creating a dense fog of electronic signals. To make sense of this, physicists must reconstruct the paths of individual particles from the tiny blips of energy they leave behind in massive detectors. This process is known as tracking, and it is the first critical step in turning raw data into scientific discovery. However, as the machines become more powerful and the data more complex, the software used to trace these paths must be incredibly precise. If the software is unsure about a path, that uncertainty can ripple through the entire analysis, potentially hiding a new particle or creating a false one. Therefore, scientists need a way to measure not just the answer a computer gives, but how confident that computer is in its own answer.

A team of researchers has developed a new way to measure this confidence within a sophisticated software pipeline called ACORN, which is designed to track particles in high-energy physics experiments. This pipeline acts like a multi-stage filter, taking a cloud of raw detector signals and turning them into clear, reconstructed paths. The process begins by grouping nearby signals into a network, then using a type of artificial intelligence known as a graph neural network to decide which connections are real and which are just noise. Finally, a rule-based algorithm stitches these connections together into complete tracks. The challenge the researchers addressed was understanding how uncertainty travels through this chain. If the first stage makes a small mistake or is unsure about a connection, does that doubt grow larger as it moves to the next stage, or does the system correct itself? To answer this, the team applied a technique called Monte Carlo Dropout, which essentially asks the computer to make the same prediction many times with slight, random variations in its internal settings. By looking at how much the answers varied across these many attempts, they could map out exactly where the system was confident and where it was guessing.

The researchers found that the ACORN pipeline is remarkably robust. As they increased the amount of training data the system saw, the uncertainty in its predictions stopped being driven by the model's lack of experience and became dominated by the inherent noise of the physical data itself. This is a crucial distinction: it means the model has learned as much as it possibly can from the available data, and any remaining uncertainty is due to the messy reality of the detector, not a flaw in the software. The study revealed that the uncertainty from the earlier stages of the pipeline did not amplify as it moved downstream; instead, the system maintained a high level of confidence in its final track reconstructions. Even when the researchers tested the system with different settings and without perfect calibration of its internal scores, the final efficiency of finding real particle tracks remained stable at around 94.5 percent. This suggests that the pipeline is resilient, capable of delivering reliable results even when the inputs are imperfect.

One of the most interesting findings was that the uncertainty did not follow a simple, predictable mathematical pattern as it moved from one stage of the pipeline to the next. In many scientific models, errors might grow in a straight line or a simple curve, but here the behavior was complex and specific to the structure of the data. The researchers also discovered that the system's confidence was highest for the most important particles—those with high momentum that carry the most valuable physics information—and that the uncertainty was largely independent of the specific angle at which the particles traveled through the detector. Perhaps most importantly, the team showed that even if the computer's internal scores were not perfectly calibrated to represent true probabilities, the final physical result—the number of correctly reconstructed tracks—did not suffer. The system could be tuned to work effectively without needing perfect probability scores at every step.

This work provides a vital blueprint for the future of high-energy physics. As experiments move toward even higher levels of data production, the ability to quantify and trust the output of complex machine learning systems will be essential. The researchers demonstrated that by carefully measuring uncertainty at each step, they could ensure that the final scientific conclusions are solid. They concluded that their method can be applied to real data from the ATLAS experiment, and they suggested that future work could extend these techniques to the very first stage of the process, where the raw signals are first grouped together. By proving that these deep learning pipelines are not just black boxes but can be understood, measured, and trusted, this research helps ensure that the discoveries of tomorrow are built on a foundation of certainty.

Technical Summary: Uncertainty Quantification and Propagation for ACORN

Problem Statement
High-Energy Physics (HEP) experiments, particularly the upcoming High-Luminosity LHC (HL-LHC) phase, face significant challenges in data volume and complexity, necessitating advanced reconstruction algorithms. Geometric Deep Learning (GDL) pipelines, such as ACORN (A Charge Object Reconstruction Network), have emerged as state-of-the-art solutions for particle tracking, utilizing Graph Neural Networks (GNNs) to infer track candidates from detector hit data. However, the reliability of these multistep pipelines is critical for claiming discoveries. A primary gap exists in understanding how uncertainties propagate through chained models and how to distinguish between reducible (epistemic) and irreducible (aleatoric) uncertainties. Specifically, it is unclear how the uncertainty of an upstream filtering stage affects downstream GNN predictions and the final track reconstruction efficiency, nor is it established whether GNN outputs are well-calibrated probabilities.

Methodology
The authors developed an Uncertainty Quantification (UQ) and Propagation (UP) framework applied to the ACORN pipeline using the TrackML open dataset. The pipeline consists of three stages: graph construction (metric learning), edge scoring (GNN), and track construction (CC&Walk algorithm).

Uncertainty Quantification via Monte Carlo Dropout (MCD): The study employs MCD to sample from the posterior distribution of model outputs. Dropout layers were inserted into the GNN's message-passing MLPs and the Filter's MLP. For each input graph, $T=100$ stochastic predictions were generated.
Uncertainty Metrics:
- Total Uncertainty: Measured as the Shannon entropy of the average predicted score.
- Epistemic Uncertainty: Measured as the mutual information between the prediction and the model weights.
- Aleatoric Uncertainty: Derived as the conditional entropy (Total minus Epistemic).
- Standard Deviation: The empirical standard deviation of the 100 score predictions.
Propagation Analysis:
- Topological Uncertainty: The study quantifies how the Filter's uncertainty (which determines which edges are pruned) propagates to the GNN. This is termed "topological uncertainty" because different Filter instances produce different graph structures.
- Efficiency Propagation: The impact of upstream uncertainties on the final track building efficiency (CC&Walk) was measured by repeating the full stochastic pipeline $T$ times and fitting the resulting efficiency distribution.
Calibration: The authors evaluated the calibration of the GNN binary classifier using reliability diagrams and spline fitting to align predicted scores with true probabilities, testing the impact of calibration on tracking efficiency.

Key Results

Dominance of Aleatoric Uncertainty: The analysis reveals that the total uncertainty of the GNN is dominated by aleatoric (data) uncertainty rather than epistemic (model) uncertainty. As the training dataset size increases (from 100 to 1,400 events), the epistemic component diminishes, confirming the model has been trained to its full potential relative to the data noise.
Propagation Characteristics: The combined uncertainty of the GNN is roughly equally distributed between intrinsic uncertainty and uncertainty propagated from the Filter. However, no simple analytical power-law scaling was found between the Filter's standard deviation and the GNN's standard deviation, consistent with the theoretical difficulty of modeling UP in complex pipelines.
Non-Gaussian Posterior: The empirical posterior distribution of the GNN scores, sampled via MCD, exhibits non-Gaussian behavior (non-zero skewness and kurtosis) across the score range, contradicting assumptions of Gaussian error profiles.
Robustness to Calibration: Despite the GNN being significantly miscalibrated (underconfident) in its raw state, the study found that tuning the score cuts in the CC&Walk algorithm compensates for this. The final track reconstruction efficiency showed no statistically significant difference between calibrated and uncalibrated GNN scores (94.4% vs. 94.5%).
Pipeline Stability: The uncertainty in the final track building efficiency is approximately 0.05%, indicating the pipeline is robust against upstream uncertainties.
Dependence on Physics Variables: Uncertainty ( $\sigma$ ) shows a slight increase for high transverse momentum ( $p_T$ ) tracks and in the central pseudo-rapidity ( $\eta$ ) region, attributed to lower statistics of such tracks in the training dataset rather than inherent model failure.

Significance and Claims
The paper claims to provide a consistent methodology for measuring uncertainty in multistep geometric deep learning pipelines for HEP. Its primary contributions are:

Demonstrating that for the ACORN pipeline, the model is sufficiently trained such that irreducible data noise (aleatoric uncertainty) is the limiting factor, not model capacity.
Establishing that the pipeline yields high confidence in track reconstruction and does not suffer from performance degradation due to GNN miscalibration, provided the downstream reconstruction thresholds are appropriately tuned.
Highlighting the complexity of uncertainty propagation in topological pipelines, where upstream model uncertainty alters the graph structure itself, making analytical treatment intractable.

The authors conclude that while their UQ scheme is effective for the current pipeline, future work is needed to apply these methods to actual ATLAS data and to extend UQ to the metric learning stage of graph construction, which presents unique challenges due to the lack of a clear metric for adjacency matrix uncertainty.

More like this