🔬 materials science

MLIP Detective: Active Failure Mode Discovery Beyond Benchmark Scores for Machine-Learning Interatomic Potentials

This paper introduces MLIP Detective, an agentic framework that uses physics-informed search to actively discover and characterize hidden failure modes in universal machine-learning interatomic potentials beyond standard benchmark evaluations, successfully identifying a systematic energy anomaly in the MACE-MPA-0 model.

Original authors: Ryuhei Okuno, Nontawat Charoenphakdee, Kaoru Hisama, Yuta Tsuboi

Published 2026-09-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ryuhei Okuno, Nontawat Charoenphakdee, Kaoru Hisama, Yuta Tsuboi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the microscopic world of atoms, scientists rely on powerful computer programs to predict how materials behave. These programs, known as machine-learning interatomic potentials, act as a shortcut. Instead of running incredibly slow and expensive calculations for every single atom in a new material, these programs use patterns learned from past data to guess the energy and forces at play. This allows researchers to simulate everything from battery chemistry to new alloys with remarkable speed. However, just like any shortcut, these programs can make mistakes. They might work perfectly for the specific materials they were trained on but fail completely when asked to predict something slightly different. The danger is that a program could look perfect on standard tests while secretly producing impossible results in the real world, potentially leading scientists down the wrong path.

To solve this, researchers at Preferred Networks have developed a new system called MLIP Detective. Instead of waiting for a human to manually check every possible scenario, this system uses an artificial intelligence agent to actively hunt for the specific conditions where these computer programs break down. The team tested this approach on a popular material simulation model and found that it successfully uncovered a hidden flaw: the model was predicting that certain atoms would stick to metal surfaces with impossible energy levels, suggesting the atoms were floating in mid-air rather than attached. By catching this error, the system proved that it could act as a safety inspector for the digital tools scientists use to design the future of technology.

The core challenge the researchers faced was that standard tests only check a tiny, pre-defined slice of the vast universe of possible atomic arrangements. A model might score perfectly on these tests but behave unphysically the moment it encounters a configuration it hasn't seen before. The team realized that the solution wasn't to test more random cases, which would be too slow and expensive, but to figure out exactly where to look. They framed this as an active search for failure, where the goal is to generate specific, testable guesses about where a model might be wrong, check those guesses with quick, cheap simulations, and only spend the most expensive computing power on the most suspicious cases.

To do this, they built an automated workflow driven by a large language model, which acts as the "Detective." This agent starts by looking at the results a model has already produced on standard benchmarks. It then uses its knowledge of physics to propose a hypothesis about a hidden flaw. For example, it might guess that a model is confused about how oxygen atoms interact with certain metals. The agent then sends out a team of "Probe" agents to test this idea. These probes run many quick simulations to see if the model behaves strangely in the predicted scenario. Crucially, the system compares the suspect model against a group of other, different models. If the suspect model acts strangely while the others act normally, or if it violates a basic law of physics, the system flags it as a likely error.

In their first major test, the MLIP Detective system investigated a widely used model called MACE-MPA-0. The system noticed a pattern in the data: the model was predicting that some combinations of metal surfaces and oxygen-containing atoms had higher energy than they should. In simple terms, the model thought these atoms were more stable when they were far apart than when they were stuck together, which is physically impossible for a stable bond. The system traced this error back to a specific cause: the model had been trained on a mix of two different types of calculation data that were not perfectly compatible. It was like trying to build a map using two different sets of rules for measuring distance. The system confirmed that this error only happened for specific metals and specific types of atoms, effectively mapping out the exact boundaries of the model's failure.

The researchers then took this discovery a step further to find a flaw that standard tests had completely missed. They looked at how a model predicted carbon monoxide gas would detach from a copper surface. Standard tests only check the moment the gas is stuck to the surface, but they don't check the journey of it pulling away. The MLIP Detective system hypothesized that the model might create a fake energy barrier along the path of detachment. When the probes ran the simulation, they found exactly that: the model predicted a small but significant energy hill that the gas would have to climb to escape, a feature that did not exist in the real physics of the situation.

To be absolutely sure, the researchers took the specific path the model had created and ran it through a traditional, high-precision physics calculation, which is the gold standard but takes much longer to run. This final check confirmed that the model was indeed wrong; the real physics showed a smooth path with no energy hill, while the model had invented one. The system had successfully identified a defect that would have been invisible to anyone only looking at the final result.

This work demonstrates that we can use intelligent agents to audit the tools of science itself. By combining the speed of machine learning with the rigorous logic of physics, the MLIP Detective framework can find hidden errors before they cause problems. It doesn't just tell us that a model is wrong; it explains why, shows exactly where the error happens, and provides a clear path for humans to verify the finding. This approach offers a new way to ensure that the digital models driving modern material discovery are trustworthy, turning the search for errors from a passive hope into an active, systematic process.

Technical Summary: MLIP Detective – Active Failure Mode Discovery

Problem Statement

Universal machine-learning interatomic potentials (u-MLIPs) are designed to generalize across diverse chemical configurations without system-specific fitting. While current evaluation relies heavily on leaderboard benchmarks (e.g., MLIP Arena, Matbench Discovery), these tools evaluate models only on predefined tasks covering limited regions of configurational space. Consequently, a model may achieve high benchmark scores while exhibiting unphysical behavior in untested configurations. Expanding benchmark coverage is prohibitively expensive because each new configuration requires a reference calculation, typically using Density Functional Theory (DFT). The core challenge is not to test more, but to decide where to test. Existing approaches for evaluating Large Language Models (LLMs) or "Proof-Carrying Materials" (PCM) do not directly transfer to MLIPs, as MLIP evaluation relies on specific physical protocols and the discovery of relational failures (e.g., unphysical dissociation limits) rather than isolated point errors.

Methodology: The MLIP Detective Framework

The authors propose MLIP Detective, an agentic framework for active failure mode discovery. The system treats failure discovery as a search problem where the goal is to generate falsifiable, physics-informed hypotheses, screen them with inexpensive signals, and reserve costly DFT and expert verification for the most promising candidates.

The framework operates through a closed-loop interaction between two agent roles and a human verifier:

Detective Agent:
- Hypothesis Generation: Starts from existing benchmark evidence to generate candidate failure hypotheses.
- Pre-registration: Before testing, the agent must declare expected failures (e.g., specific energy anomalies, force inconsistencies) and define a decision table mapping potential DFT outcomes to verdicts. This prevents post-hoc rationalization.
- Orchestration: Ranks hypotheses by application impact and severity, then spawns parallel Probe Agents.
- Evaluation: Reviews reports from Probe Agents using two label-free surrogate signals:
  - Cross-model disagreement: Discrepancies between the target model and a committee of auxiliary u-MLIPs.
  - Physical constraint violations: Breaches of documented physical laws (e.g., energy-force consistency, dissociation limits).
- Escalation: If a hypothesis carries strong surrogate evidence, the agent escalates it to a human expert with a self-contained verification package.
Probe Agents:
- Execute computational probes based on the Detective's hypothesis.
- Run identical protocols on the target model and the committee models.
- Analyze results from multiple perspectives (scope, mechanism, and application impact) and return structured reports.
Human Verifier:
- Reviews escalated candidates.
- Performs the costly DFT calculations only on the specific configurations provided in the verification package.
- Renders the final verdict (Confirmed/Refuted).

The framework relies on an inspection specification that encodes physical constraints (e.g., translational invariance, elastic stability, size consistency) and diagnostic expectations, steering agents toward scientifically important or under-tested regions without referencing specific known failure modes.

Key Contributions

Agentic Active Search: The first framework to cast MLIP failure discovery as an active search problem driven by LLM agents, moving beyond static benchmark evaluation.
Physics-Informed Hypothesis Generation: The system generates falsifiable hypotheses grounded in physical principles (e.g., training data inconsistencies) rather than random sampling.
Self-Contained Verification: The framework outputs a complete package for human review, including reproduction code, DFT-ready input structures, and a decision table, enabling independent confirmation.
Surrogate Signal Strategy: It utilizes cross-model disagreement and physical constraint violations as inexpensive proxies to prioritize candidates, significantly reducing the number of required DFT calculations.

Experimental Results

The authors present two case studies inspecting the MACE-MPA-0 model:

Case Study 1: Rediscovery of Adsorption-Energy Anomalies (U-Block)

Observation: MLIP Detective identified that MACE-MPA-0 predicted relaxed adsorbate–surface systems involving O- or F-containing adsorbates on specific transition metals (the "U-block") to be higher in energy than their separated fragments ( $E_{rel} > 0$ ), a physically impossible state.
Hypothesis: The anomaly stems from training on mixed PBE and PBE+U reference data, creating inconsistent energy references.
Verification: Cross-model comparisons and channel-switching experiments (comparing MACE-MPA-0 against different training channels) supported the hypothesis. The anomaly was localized to 27 of 40 combinations within the U-block and absent outside it.
Outcome: The human verifier confirmed the finding was consistent with recent literature, rendering additional DFT calculations unnecessary for this specific case.

Case Study 2: DFT-Verified Anomaly Beyond Benchmark Coverage

Observation: While MACE-MPA-0 performed well on the benchmark's single-point adsorption energy for CO on Cu(111), the framework detected a defect in the desorption pathway.
Hypothesis: The model exhibits a spurious energy barrier along the desorption coordinate.
Verification: A rigid scan and subsequent Climbing-Image Nudged Elastic Band (CI-NEB) calculations revealed a barrier of $\approx 0.21$ eV in the MACE-MPA-0 path, whereas the DFT reference and committee models showed a monotonic descent.
Outcome: DFT single-point calculations on the model's path confirmed the spurious barrier ( $+0.208$ eV vs. DFT's $+0.022$ eV). This defect was invisible to the standard benchmark, which only evaluates the relaxed minimum.

Significance and Claims

The paper claims that MLIP Detective successfully demonstrates that an agentic framework can complement benchmark-based evaluation by uncovering hidden failure modes that are outside the predefined scope of standard datasets.

Beyond Benchmarks: The system identified systematic anomalies (training data inconsistencies) and unphysical behaviors (spurious barriers) that standard benchmarks missed.
Efficiency: By using surrogate signals to filter candidates, the framework reserves expensive DFT calculations for high-probability failures, making the discovery process more efficient.
Trustworthiness: The "pre-registration" of hypotheses and the requirement for a self-contained verification package ensure that findings are auditable and that human experts retain final judgment.
Generalizability: The approach is not limited to specific models or datasets; it is designed to work with any u-MLIP and can be extended to broader classes of simulation protocols.

The authors acknowledge limitations, noting that the search is bound by the inspection specification and that escalated findings still require costly human verification. Future work aims to automate the quality control of escalated findings to further reduce the burden on experts.