🔬 materials science

Does System Size Matter in Uncertainty Quantification for Machine Learning Interatomic Potentials?

This study demonstrates that common uncertainty aggregation methods for machine learning interatomic potentials exhibit distinct dependencies on system size, revealing that size-unaware thresholds compromise out-of-distribution detection and motivating the need for size-aware calibration in large-scale atomistic simulations.

Original authors: Chayaphol Lortaraprasert, Nontawat Charoenphakdee, Wenwen Li, Yong-Bin Zhuang, Yuta Tsuboi

Published 2026-10-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Chayaphol Lortaraprasert, Nontawat Charoenphakdee, Wenwen Li, Yong-Bin Zhuang, Yuta Tsuboi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quest to discover new materials, scientists rely on a powerful tool called machine learning interatomic potentials. These are computer programs trained to predict how atoms will behave and interact, acting as a shortcut to the incredibly complex and slow calculations of quantum physics. Instead of solving the full equations for every single atom in a simulation, these programs learn from a smaller set of high-quality data and then apply that knowledge to vast systems, allowing researchers to simulate millions of atoms over long periods. To keep these simulations safe and accurate, the programs use a built-in alarm system called uncertainty quantification. This system flags when the computer encounters a situation it has never seen before, such as a strange arrangement of atoms or a defect in a crystal, signaling that the prediction might be unreliable and that a more rigorous, slower calculation is needed.

For years, a standard practice has been to treat these alarm signals the same way regardless of the size of the system being studied. If a simulation involves a small cluster of atoms or a massive block of material, researchers would simply average the uncertainty scores of all the individual atoms or look at the single highest score to decide if the whole structure was safe. The assumption was that a high score meant the same thing in a small box as it did in a giant one. However, a new study challenges this long-held belief, revealing that the size of the system itself changes how these alarm signals behave, potentially causing the safety net to fail exactly when it is needed most: in the largest, most complex simulations.

The researchers set out to test whether these uncertainty scores remain comparable as the number of atoms in a simulation grows. They used several different types of machine learning models to simulate various materials, including a metal alloy, liquid water, and ice. In each case, they created two versions of the same material: one that was perfect and familiar to the computer (in-distribution), and one that contained a specific flaw, such as a missing atom or a different phase of matter, which the computer had not been trained to recognize (out-of-distribution). They then ran simulations on these materials, gradually increasing the number of atoms from a few dozen to several hundred, and watched how the uncertainty scores changed.

What they found was a clear and surprising split in behavior depending on how the scores were calculated. When the researchers averaged the uncertainty of all atoms in the system, the score stayed steady for perfect materials but dropped as the system grew larger if it contained a defect. This happened because the few atoms near the defect were diluted by the thousands of perfect atoms surrounding them, making the average look deceptively safe. In contrast, when they looked at the single highest uncertainty score in the system, the behavior was the opposite. For perfect materials, this highest score kept climbing as the system got bigger, simply because a larger system has more chances to stumble upon a rare, slightly unusual local arrangement of atoms. However, for materials with a real defect, this highest score stayed flat and did not rise with size.

This divergence creates a dangerous blind spot. In a small system, a real defect produces a high uncertainty score that stands out clearly against the low scores of a perfect material. But in a large system, the perfect material's highest score rises so high that it begins to look just as uncertain as the defective material. The gap between the two disappears, and the computer can no longer tell the difference between a rare but harmless fluctuation in a perfect crystal and a genuine, dangerous flaw. The study suggests this happens because of a statistical principle: as you draw more samples from a distribution, the most extreme value you find naturally gets larger. A larger system simply reaches further into the tail of its own normal behavior, surfacing rarer local environments that the model finds slightly unfamiliar, even though the material is fundamentally the same.

To understand if these rising scores in large, perfect systems were actually warning of something important, the researchers tracked the specific atoms that triggered the highest uncertainty. They found that these atoms were indeed in unusual local environments and had higher errors than atoms in smaller systems. Logically, a standard learning loop would flag these structures for retraining, feeding the computer new data to fix the errors. However, when the researchers actually did this, retraining the model on these large, flagged structures yielded almost no improvement. The model remained stubbornly inaccurate on these specific atoms.

In a revealing comparison, the researchers then trained the model on small structures that were deliberately designed to be unfamiliar, rather than large structures that were just statistically rare. This approach was far more effective. Retraining on these small, clearly out-of-distribution structures significantly improved the model's performance and closed the gap between perfect and defective materials. The study concludes that the current method of using a single, fixed threshold to judge uncertainty across all system sizes is flawed. It suggests that the way these models are trained, which averages errors over all atoms, allows them to ignore a few bad apples in a large barrel, making the standard alarm system unreliable for massive simulations.

The findings point toward a necessary shift in how these simulations are managed. The researchers propose that future systems need to be "size-aware," meaning they must account for the fact that uncertainty naturally scales with the number of atoms. They tested a preliminary method to adjust the uncertainty scores after the fact to remove this size bias, but found it difficult to do without accidentally masking genuine warnings about real defects. The path forward likely involves either developing new ways to calibrate these scores so they are comparable across different sizes or changing the training process itself to ensure that rare, high-error atoms are not sacrificed for the sake of overall accuracy. Until these adjustments are made, the safety mechanisms in large-scale material simulations may be silently failing, unable to distinguish between the noise of a large system and the signal of a real problem.

Problem Statement

Machine Learning Interatomic Potentials (MLIPs) are increasingly used to accelerate atomistic simulations for materials discovery, enabling system sizes and time scales far beyond Density Functional Theory (DFT). A critical component of deploying MLIPs is Uncertainty Quantification (UQ), which flags out-of-distribution (OOD) configurations and drives active learning loops.

Current practices typically aggregate per-atom uncertainties (e.g., force uncertainties or descriptor-space novelty) into a single structure-level score using either the mean or the maximum of the atom-wise scores. These scores are then compared against fixed thresholds calibrated on small reference cells. The paper identifies a fundamental, untested assumption in this workflow: that structure-level uncertainty scores remain comparable across different system sizes. As simulations scale to millions of atoms and active learning datasets become mixed-size, the authors question whether the statistical properties of these aggregations change with system size, potentially causing OOD detection to fail in large-scale simulations.

Methodology

The authors investigate the scaling behavior of per-atom force uncertainty with respect to system size ( $N$ ) using three distinct Machine Learning Interatomic Potential (MLIP) architectures and uncertainty estimators:

Ensemble Variance: Using ensembles of MACE [7] and ORB [13] models.
Single-Model Metric: Using the ORB Force-Delta (the difference between non-conservative and conservative forces).

The study focuses on three In-Distribution (ID) vs. Out-of-Distribution (OOD) pairs:

Ni $_3$ Al: Pristine (ID) vs. Vacancy-containing (OOD).
Water/Ice: Liquid water (OOD) vs. Ice crystal (ID).
WBM Dataset: Mn $_2$ ScSe (ID) vs. CrPt $_2$ Sc (OOD).

For each system, supercells of increasing size were generated via replication. Structures were sampled from Molecular Dynamics (MD) trajectories driven by foundation potentials. The models were trained exclusively on ID structures of varying sizes. The authors analyzed two aggregation methods:

Mean Uncertainty ( $\bar{u}$ ): $\frac{1}{N} \sum u_i$
Maximum Uncertainty ( $u_{max}$ ): $\max_i u_i$

They further examined the local environment (via SOAP descriptors) and force errors of the atoms contributing to $u_{max}$ to determine if high uncertainty corresponds to genuine novelty. Finally, they tested the efficacy of active learning by retraining models on structures flagged as OOD by these metrics.

Key Results

1. Divergent Scaling Behaviors of Aggregations

Mean Aggregation: For homogeneous ID systems, the mean uncertainty remains size-independent. However, for defective systems (e.g., vacancies), the mean uncertainty decays as $O(1/N)$ in larger cells. In large systems, the mean uncertainty of a defective structure becomes indistinguishable from a pristine one, rendering it ineffective for detecting local defects.
Maximum Aggregation: For homogeneous ID systems, $u_{max}$ increases with system size $N$ . Conversely, for defective systems, $u_{max}$ remains approximately flat. This behavior is attributed to extreme-value statistics: as $N$ increases, a homogeneous system samples deeper into the tail of its own environment distribution, surfacing rarer, more distorted local environments that yield higher uncertainty.
Consequence: As system size grows, the gap between ID and OOD uncertainty scores narrows and eventually disappears for maximum-based aggregation. This leads to a failure in size-unaware OOD detection, where large ID structures are falsely flagged as OOD, or the distinction between ID and OOD is lost.

2. Validity of High-Uncertainty Atoms vs. Retraining Efficacy
The authors verified that the atoms with the maximum uncertainty in large ID structures indeed possess unfamiliar local environments (high distance in SOAP descriptor space) and high force errors. While these atoms are statistically valid OOD candidates, retraining on these large structures yields little improvement in model accuracy.

In contrast, retraining on small, "OOD-by-design" structures (e.g., small vacancy cells) significantly improves the model and closes the uncertainty gap.
Explanation: The standard mean-squared error (MSE) loss used to train MLIPs averages over all atoms. In a large system, a few distorted environments contribute $O(1/N)$ to the total loss. Consequently, the optimizer can tolerate poor fits for these outlier atoms while maintaining a low global error. Adding more large-ID structures does not force the model to fit these rare environments better.

3. Limitations of Post-Hoc Calibration
The authors attempted to implement a trainable, post-hoc size-aware uncertainty calibrator to remove the size bias. While the calibrator successfully flattened the size-dependent trend for pristine systems, it inadvertently suppressed the signal for defective systems (which naturally do not scale with size), thereby narrowing the ID/OOD gap. This suggests that removing size bias without distorting genuine OOD signals is non-trivial.

Key Contributions

Identification of System Size Bias: The paper demonstrates that current UQ aggregation methods (mean and max) are not size-invariant, challenging the convention of using fixed thresholds across mixed-size datasets.
Statistical Explanation: The authors attribute the scaling of maximum uncertainty to extreme-value statistics, providing a theoretical basis for why larger systems naturally exhibit higher uncertainty scores even when in-distribution.
Active Learning Inefficiency: The study reveals a practical limitation in current active learning loops: flagging large structures based on size-inflated uncertainty scores is computationally expensive and yields diminishing returns compared to training on smaller, targeted OOD structures.
Empirical Evidence: Through extensive experiments with MACE and ORB models across multiple material systems, the authors provide robust evidence that size-unaware UQ leads to incorrect OOD detection in large-scale simulations.

Significance and Claims

The paper claims that system size introduces a significant bias into uncertainty quantification for MLIPs, which can cause OOD detection to fail in large-scale atomistic simulations. The authors argue that the common practice of comparing structure-level uncertainty scores across different system sizes using fixed thresholds is flawed.

The significance of these findings lies in the need for size-aware calibration or new training objectives that account for outlier atoms without sacrificing bulk accuracy. The authors conclude that without addressing this size dependence, the reliability of MLIP-driven materials discovery at scale is compromised. They propose that future work should focus on developing size-aware post-hoc calibrators or training losses that can effectively fit rare, high-error environments, rather than relying on current size-unaware uncertainty criteria.

Problem Statement

Methodology

Key Results

Key Contributions

Significance and Claims

More like this