Cutting Through the Noise: On-the-fly Outlier Detection for Robust Training of Machine Learning Interatomic Potentials

This paper introduces an automated, unsupervised on-the-fly outlier detection method that uses an exponential moving average of the loss distribution to down-weight noisy training data, enabling the robust and efficient training of machine learning interatomic potentials without the need for manual filtering or additional expensive calculations.

The Gist

The Problem: The "Bad Apple" in the Digital Lab

Imagine you are a world-class chef trying to learn how to make the perfect soup. To learn, you study thousands of recipes written by other cooks. However, some of those recipes are terrible: one might say "add a cup of salt" instead of "a pinch," and another might have a typo that says "add 5 gallons of vinegar."

If you try to follow every recipe exactly as written, you will end up making a soup that is either undrinkable or wildly inconsistent. You’ll be "overfitting"—meaning you are working incredibly hard to master mistakes that aren't actually part of good cooking.

In the world of science, researchers use Machine Learning Interatomic Potentials (MLIPs). These are AI models that act like "digital chefs." They learn how atoms interact so they can simulate chemical reactions without needing a massive, expensive supercomputer to solve complex physics equations every single second.

The catch? The "recipes" (the data) used to train these AIs often come from digital simulations that didn't finish properly. They contain "numerical noise"—basically, digital typos that make the atoms behave in ways that are physically impossible.

---

The Solution: The "Smart Sieve" (On-the-fly Outlier Detection)

Usually, if a scientist finds bad data, they have to go back, fix the original math, and restart the entire training process from scratch. This is like a chef throwing away their entire cookbook and starting over because they found three bad recipes. It’s slow, expensive, and exhausting.

The authors of this paper created a way to fix this while the AI is learning. They call it "On-the-fly Outlier Detection."

Think of it like a Smart Sieve in a kitchen. As the chef (the AI) reads each recipe, they don't just blindly follow it. Instead, they keep a running mental note of what a "normal" recipe looks like.

1. The Comparison: As the AI reads a new recipe, it asks: "Does this instruction look wildly different from the last 1,000 instructions I've read?"
2. The Weighting: If the instruction says "add a pinch of salt," the AI gives it full attention. If the instruction says "add a bucket of salt," the AI recognizes this as an "outlier."
3. The "Muting" Effect: Instead of throwing the recipe away (which would be a waste of time), the AI simply turns the volume down on that specific instruction. It says, "I see you, but I'm not going to let you influence my cooking too much."

This is done using a mathematical trick called an Exponential Moving Average. It’s like a chef who has a "gut feeling" that evolves over time; as they gain more experience, their ability to spot a "bad recipe" becomes sharper and sharper.

---

Why It Matters: Real-World Results

The researchers tested this "Smart Sieve" on three different levels, and the results were impressive:

• Level 1: The Small Scale (The Soup Test). They took a standard dataset and intentionally "poisoned" 10% of it with bad data. The AI trained with the Smart Sieve ignored the poison and learned the correct physics, while the standard AI got confused and failed.
• Level 2: The Liquid Test (The Water Test). They trained an AI on "messy" data about liquid water. A normal AI predicted that water wouldn't flow correctly (it messed up the "diffusion"). The Smart Sieve AI ignored the messy data and correctly predicted how water molecules actually move.
• Level 3: The Giant Scale (The Foundation Model). They applied this to a massive "Foundation Model"—an AI trained on millions of different molecules (like a library of every recipe ever written). Even at this massive scale, the Smart Sieve helped the AI ignore "unphysical" structures (like atoms overlapping in impossible ways), making the AI three times more accurate in predicting energy.

The Bottom Line

This paper provides a way to train incredibly powerful scientific AIs using "imperfect" data. It allows scientists to build better digital tools for discovering new medicines and materials without having to spend years cleaning up their data by hand. It’s a way to cut through the noise and find the truth, even when the data is shouting lies.

Technical Summary

Technical Summary: Cutting Through the Noise

Title: Cutting Through the Noise: On-the-fly Outlier Detection for Robust Training of Machine Learning Interatomic Potentials
Authors: Terry C. W. Lam, Niamh O’Neill, Christoph Schran, and Lars L. Schaaf (University of Cambridge)

---

1. The Problem: Numerical Noise in MLIP Training

Machine Learning Interatomic Potentials (MLIPs) are essential for accelerating molecular simulations by replacing expensive ab initio quantum chemistry calculations. However, their accuracy is strictly limited by the quality of the reference data.

A significant challenge arises from numerical noise in these datasets, which stems from:

• Incomplete convergence: Electronic structure calculations (like DFT) that fail to reach self-consistent field (SCF) convergence.
• Stochasticity: Inherent noise in Monte Carlo methods (VMC/DMC).
• Inconsistency: Mismatched settings across large, diverse datasets.

Current limitations: Existing mitigation strategies are either manual (requiring expert chemical knowledge and being unscalable) or iterative (requiring multiple expensive retraining cycles to identify and remove outliers).

2. Methodology: On-the-fly Dynamic Bootstrapping

The authors propose an unsupervised, automated scheme to identify and down-weight noisy samples during a single training run, requiring no additional reference calculations.

Core Mechanism:
The method is based on the observation that outliers take longer to learn than clean data. The training process is modified through three main steps:

1. Loss Distribution Tracking: Instead of using noisy instantaneous batch statistics, the method maintains an Exponential Moving Average (EMA) of the mean ($\mu$) and variance ($\sigma^2$) of the training loss across all batches. This provides a smooth, robust estimate of the "expected" error distribution.
2. Z-Score Outlier Identification: For every configuration in a batch, a $z$-score is calculated: $z_{i,\beta} = (L_{i,\beta} - \mu_\beta) / \sigma_\beta$. This measures how many standard deviations a specific sample's loss is from the running average.
3. Dynamic Weighting (Bootstrapping): A per-configuration weight ($w_i$) is assigned using a smoothed thresholding function based on the Gaussian cumulative distribution function.
   • Clean data ($z \ll z_t$): Weight $\approx 1$ (full trust).
   • Noisy data ($z \gg z_t$): Weight $\approx 0$ (effectively ignored).

The modified loss function becomes: $L' = \frac{1}{N_B} \sum_{i=1}^{N_B} w_i^2 L_i$.

3. Key Contributions

• Single-Pass Training: Achieves noise resilience in one training cycle, eliminating the need for the "train-filter-retrain" loop.
• Unsupervised & Automated: Requires no manual curation or domain-specific expertise to identify "bad" data.
• Scalability: The method is computationally lightweight and can be implemented in multi-GPU environments (tested via the MACE framework).
• Robustness to Hyperparameters: The performance is shown to be relatively insensitive to the exact choice of the $z$-score threshold ($z_t$).

4. Results and Validation

The authors validated the method across three distinct scales:

• Controlled Benchmark (revMD17): By injecting 10% noise from the original MD17 dataset into the clean revMD17 dataset, they showed that standard training leads to massive overfitting. The proposed bootstrapping method prevented this, maintaining high accuracy on the "true" ground truth and achieving a three-fold improvement in validation force error.
• Physical Observables (Liquid Water): Using a dataset with intentionally loose DFT convergence, the authors demonstrated that the bootstrapped model recovers accurate macroscopic properties. Specifically, it significantly improved the self-diffusion coefficient and corrected the radial distribution functions (RDFs), which are typically distorted by noisy forces.
• Foundation Model Scale (SPICE 2.0): Applied to a massive dataset of 2 million configurations, the method successfully identified unphysical structures (e.g., steric clashes/overlapping atoms). It reduced energy prediction errors by a factor of three compared to vanilla training.

5. Significance

This work provides a practical solution for the "large data regime" of machine learning in chemistry. As the field moves toward massive foundation models, the ability to automatically "clean" data on-the-fly is critical. This framework allows researchers to utilize imperfect, high-throughput datasets to train highly accurate, robust models, significantly accelerating the discovery of new materials and drugs while reducing the computational overhead of data curation.