🔬 materials science

Challenging Atomistic Foundation Models on Disordered Carbons

This study stress-tests six foundation machine-learned interatomic potentials on disordered carbon systems, revealing that low training errors do not guarantee out-of-domain reliability and identifying MACE-MH-1 as the most robust model for generating experimentally realistic nanoporous carbon structures.

Original authors: Joseph Hart, Samuel G. H. Brookes, Fabian L. Thiemann, Christoph Schran

Published 2026-10-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Joseph Hart, Samuel G. H. Brookes, Fabian L. Thiemann, Christoph Schran

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of materials science, researchers are constantly trying to build better batteries, supercapacitors, and water filters. A key ingredient in many of these technologies is a special kind of carbon that is full of tiny holes, known as nanoporous carbon. This material is incredibly useful because its vast internal surface area can trap energy or filter impurities. However, understanding exactly how this material forms and behaves is difficult. At the atomic level, the carbon atoms are arranged in a chaotic, disordered way, mixing different types of bonds and creating a complex network of pores. To study this, scientists use powerful computer simulations that act like a virtual microscope, tracking how every single atom moves and interacts over time. For years, these simulations relied on specific, hand-crafted rules for carbon, but a new generation of tools has emerged. These are called foundation models, which are like universal calculators trained on massive amounts of data to predict how atoms behave in almost any situation without needing to be retrained for every new material. The big question is whether these general-purpose tools are reliable enough to handle the messy, extreme conditions required to create realistic nanoporous carbon.

A team of researchers set out to test this reliability by putting six of the most popular foundation models through a rigorous stress test using disordered carbon. They did not just look at how well the models calculated energy on a static computer screen; they pushed the models to their limits by simulating the entire process of making this material. This involved heating carbon to extreme temperatures until it melted, then rapidly cooling it down to form a solid, and finally heating it again to let the atoms rearrange into a stable structure. They tested these simulations with pure carbon and also with carbon containing oxygen and hydrogen, which are common in real-world materials but often ignored in computer models. The researchers found that while some models performed well on simple, standard tests, they failed spectacularly when asked to simulate the complex, high-temperature journey of creating this material. Several models produced results that were physically impossible, such as atoms collapsing into each other or the simulation crashing entirely, even though their basic error scores looked good on paper.

The study revealed that low error numbers in standard tests do not guarantee that a model will work in the real world. When the researchers examined the binding curves, which show how atoms attract and repel each other, they found that some models developed strange, unphysical behaviors when atoms were squeezed together or pulled apart. These hidden flaws meant that when the models tried to simulate the melting and cooling process, they could not maintain a stable structure. One model, in particular, stood out as the most robust, successfully completing every single simulation across a wide range of densities and compositions. This model managed to generate structures that looked very similar to what scientists see in actual experiments, correctly reproducing the size of the pores and the way the atoms are bonded. In contrast, other models that seemed promising at first glance failed to create stable structures, especially when oxygen and hydrogen were added to the mix.

The researchers also discovered something surprising about how oxygen and hydrogen affect the material. While these extra atoms did not change the immediate arrangement of the carbon neighbors, they did significantly alter the larger structure. The presence of these elements pushed the pore sizes to be larger and created a more open network. This suggests that the chemical makeup of the material plays a crucial role in how the final structure assembles, a detail that pure carbon models might miss. The work highlights that choosing the right tool for a specific job is critical; just because a model is trained on a broad dataset does not mean it will work for every specific application. The team concluded that before scientists can trust these powerful new tools for designing real-world materials, they must be tested not just on their ability to calculate numbers, but on their ability to simulate the dynamic, messy reality of how materials actually form. This approach ensures that the virtual models are not just mathematically elegant, but physically trustworthy.

Technical Summary: Challenging Atomistic Foundation Models on Disordered Carbons

Problem Statement
Machine-learned interatomic potentials (MLIPs) have emerged as foundation models capable of approximating density functional theory (DFT) potential energy surfaces (PES) across broad chemical spaces. While these models promise broad transferability, their reliability in out-of-domain scenarios—particularly for complex, disordered materials like nanoporous carbon—remains difficult to establish. Conventional benchmarks relying solely on DFT energy and force errors often fail to detect spurious minima, unphysical extrapolations, or dynamical instabilities that can compromise simulations. Nanoporous carbon presents a stringent test case due to its formation spanning liquid, amorphous, and graphitic phases, its mixture of $sp$, $sp^2$ , and $sp^3$ bonding, and its relevance to energy storage and water treatment. Existing specialized potentials (e.g., GAP series) require extensive retraining for specific compositions, whereas foundation models aim to handle heteroatoms (O/H) without new reference data. However, it is unclear if current foundation models can reliably generate realistic disordered carbon structures or capture the effects of heteroatoms on framework assembly.

Methodology
The authors evaluated six popular foundation MLIPs (ORB-v3, MACE-MH-1, MatterSim-5M, MACE-MP-0b3, MACE-OMAT-0, and UMA-S-1.1) through a hierarchical validation protocol:

Static Benchmarks: Models were screened against two external DFT datasets (PBE functional): a diverse carbon dataset (>10,000 configurations covering $sp^2$ , $sp^3$ , amorphous, liquid, and clusters) and a graphene-oxide dataset (>3,000 O/H-functionalized structures). Metrics included mean absolute errors (MAE) for energy and forces, as well as relative force errors.
Physical Extrapolation Tests: Two- and many-body binding curves were analyzed for carbon dimers, isotropically scaled graphene, and diamond. These tests probed bond formation, short-range repulsion, and the presence of unphysical "holes" in the PES under extreme compression.
Dynamical Stability: A melt-quench-anneal molecular dynamics (MD) protocol was applied to simulate the formation of nanoporous carbon. This involved melting structures at 8,000 K, quenching to 300 K, annealing at 3,000 K to promote graphitization, and cooling. Simulations were run for both pure carbon and O/H-containing compositions (approximating commercial supercapacitor carbons) across seven densities (0.5–3.5 g cm $^{-3}$ ). Stability was defined by the completion of independent trajectories without simulation failure.
Structural Validation: Stable trajectories were analyzed for ring-size distributions, pair distribution functions (PDF), and pore-size distributions (PSD). Results were compared against specialized potentials (GAP-22) and experimental references (Ti-CDC-1000).

Key Results

Limitations of Static Benchmarks: Low DFT energy and force errors did not guarantee physical behavior. For instance, UMA-S-1.1 achieved low errors on periodic environments but exhibited massive force errors on isolated clusters and developed unphysical secondary minima in binding curves. Similarly, MACE-MP-0b3 showed stable performance in static benchmarks but failed dynamically at high densities.
Dynamical Failures: Only MACE-MH-1 successfully completed all 35 O/H-containing simulations across all tested densities. Other models exhibited failures during the annealing phase, particularly at densities relevant to supercapacitors (0.5–1.5 g cm $^{-3}$ ) or at high densities (>2.0 g cm $^{-3}$ ) where structures became $sp^3$ -dominated. These failures were not predicted by the static error metrics.
Structural Biases: Models trained on the OMAT dataset (ORB-v3, MACE-OMAT-0, MACE-MH-1) shared structural biases, such as an overpopulation of five-membered rings and under-coordinated carbon in amorphous precursors, despite differing architectures. This suggests that training data coverage, rather than architecture alone, dictates structural outcomes.
Heteroatom Effects: The incorporation of O and H left short-range carbon order (PDFs) largely unchanged but significantly shifted pore-size distributions toward larger pores. This was attributed to gas formation lowering framework density and the termination of under-coordinated sites, which suppressed ring closure and promoted a more open network.
Model Performance: MACE-MH-1 emerged as the most robust model, demonstrating low DFT errors, stable MD across all densities, and structural observables consistent with specialized potentials and experimental data.

Significance and Claims
The paper argues that selecting a foundation MLIP for specific out-of-domain workflows requires more than standard error metrics. The authors claim that:

Hierarchical Testing is Essential: A progressive testing strategy—moving from static accuracy to physical extrapolation (binding curves) and finally to dynamical stability (melt-quench)—is necessary to expose failures that low energy/force errors miss.
Training Coverage Dictates Transferability: Despite architectural differences, models trained on similar datasets (e.g., OMAT) exhibit shared structural biases, highlighting the critical role of training data diversity in achieving transferability.
Extension to Realistic Compositions: Validated foundation models can extend melt-quench generation beyond pure-carbon approximations. The study demonstrates that incorporating O/H is feasible without retraining and reveals that heteroatoms play a specific role in altering long-range morphology (pore size) even if short-range order remains similar.
Recommendation: The authors suggest that binding curves and melt-quench stability tests should be included in standard foundation-MLIP test suites to prevent downstream costs associated with unstable simulations and to identify targets for model improvement.

The work concludes that while current foundation models can reproduce principal experimental signatures of nanoporous carbon, their reliability is not uniform, and rigorous application-specific validation remains a prerequisite for generating experimentally realistic structures.

More like this