Benchmarking Universal Machine-Learned Interatomic Potentials for High-Temperature Metal-Organic Framework Chemistry

This study benchmarks five universal machine-learned interatomic potentials against a new high-temperature dataset of nine metal-organic frameworks, revealing that while ORB-v3 and fairchem OMAT perform best, all current models exhibit significant errors in high-temperature regimes, underscoring the limitations of existing potentials for simulating extreme MOF dynamics.

The Gist

The Big Picture: Predicting the Future of "Molecular Lego"

Imagine Metal-Organic Frameworks (MOFs) as incredibly complex, microscopic structures built out of "Lego bricks." Some bricks are metal, and others are organic molecules. Scientists love these structures because they are like sponges that can catch gases or help make chemicals.

However, when you heat these "Lego" structures up (like in a furnace), they start to melt, break apart, and turn into something completely different. This process is called pyrolysis, and it's how scientists make new catalysts (chemical helpers). The problem is, we can't easily see exactly how the bricks break apart at the atomic level because it happens too fast and too small for our eyes or standard microscopes.

The Problem: The "Crystal Ball" is Cracked

To see what happens inside, scientists use computer simulations.

• The Gold Standard (DFT): Think of this as a super-accurate, slow-motion camera. It tells you exactly what every atom is doing, but it's so slow and expensive that you can only film a few seconds of the movie before the computer runs out of battery.
• The Shortcut (Machine Learning Potentials): To film the whole movie, scientists use "Universal Machine-Learned Interatomic Potentials" (uMLIPs). Think of these as AI crystal balls. They are trained on millions of pictures of atoms to guess how they will move. They are fast and cheap, but we didn't know if they were accurate enough to handle the extreme heat of a furnace.

What the Researchers Did: The "Stress Test"

The authors of this paper decided to put five of the most popular AI crystal balls to the test. They created a new, massive dataset of "movies" (simulations) showing nine different types of MOF Lego structures being heated up to three different temperatures:

1. 300 K (Room Temp): Just sitting there, breathing normally.
2. 1000 K (Very Hot): Getting wobbly and distorted.
3. 2000 K (Extreme Heat): Starting to fall apart, with bricks breaking off and turning into gas.

They ran these simulations for a long time (40 picoseconds) to capture the moment the structures started to collapse. Then, they asked the five AI models to predict what was happening in these movies and compared the AI's guesses to the "Gold Standard" reality.

The Results: The AI is Good at Calm, Bad at Chaos

Here is what they found:

1. The Winners (and Losers)
Two models, ORB-v3 and fairchem OMAT, were the best at guessing the energy and forces when things were calm. They were like students who got an A on a math test when the numbers were simple. However, even the winners made mistakes.

2. The Heat Problem
As the temperature went up, the AI models started to fail.

• At Room Temperature, the AI was okay.
• At 1000 K, the AI started to get confused.
• At 2000 K, the AI was essentially hallucinating. It couldn't predict how the atoms were moving or how the structure was breaking. It was like asking a weather forecaster to predict a hurricane while they are only used to predicting sunny days.

3. The "Generative Error" Trap
This is the most important finding. The researchers ran a long simulation (1 nanosecond) using the best AI model (ORB-v3) to see how it performed over time.

• The Trap: When you check the AI's accuracy on a single frame (static check), it looks decent. But when you let the AI run the movie forward, the errors snowball.
• The Analogy: Imagine asking a GPS to navigate a car. If you check the map once, the GPS looks fine. But if you let the GPS drive the car for an hour, and it makes a tiny wrong turn every 10 seconds, the car will eventually end up in a completely different country. The AI models made tiny errors in how atoms moved, and over time, those errors added up, making the final structure look nothing like reality.

4. What Broke?
At 2000 K, the organic "bricks" (linkers) started to snap off, and the metal parts started to clump together. The AI models couldn't handle this "breaking" process. They predicted the atoms were moving in ways that didn't make physical sense.

The Bottom Line

This paper is a warning label for scientists. It says: "Don't trust these universal AI models to simulate what happens when you burn these materials down."

While these AI tools are great for looking at stable, calm structures, they are currently too inaccurate for studying high-temperature chemistry where things are falling apart. To fix this, the AI needs to be trained on more "chaotic" data—specifically, more movies of things breaking and melting—so it learns how to handle the heat. Until then, we can't rely on them to design new materials for extreme conditions.

Technical Summary

1. Problem Statement

Metal-Organic Frameworks (MOFs) are promising materials for catalysis, particularly when transformed into amorphous, MOF-derived catalysts via high-temperature pyrolysis or calcination. However, understanding the atomic-scale mechanisms of MOF decomposition (bond breaking, linker degradation, and metal node aggregation) is experimentally difficult due to the disordered nature of the products.

• Computational Challenge: While Density Functional Theory (DFT) offers high accuracy, it is computationally prohibitive for the large spatial and temporal scales required to simulate complete pyrolysis (typically nanoseconds to microseconds).
• Limitation of Current MLIPs: Universal Machine-Learned Interatomic Potentials (uMLIPs) offer a faster alternative but are typically trained on near-equilibrium, low-energy datasets. They often fail in high-temperature, non-equilibrium regimes (e.g., bond breaking, amorphous states) because these regions of the potential energy surface (PES) are sparsely represented in training data.
• The Gap: There is a lack of systematic, high-fidelity benchmark datasets for MOFs under extreme thermal conditions to rigorously test the robustness of current uMLIPs.

2. Methodology

Dataset Generation

The authors generated a new benchmark dataset consisting of 40 ps ab initio molecular dynamics (AIMD) trajectories for nine diverse MOFs (ZIF-8, CALF-20, MOF-10, MOF-5, MIP-206, UiO-66, UiO-67, UiO-66-NH2, and NU-1000).

• Conditions: Simulations were run at 300 K (equilibrium), 1000 K (thermal distortion), and 2000 K (early-stage decomposition).
• Software/Parameters: Calculations were performed using CP2K (v2023.2) with the PBE functional, DFT-D3 dispersion corrections, and triple-ζ valence polarized basis sets.
• Dynamics: NVT ensemble using a 0.5 fs timestep and CSVR thermostat.

Model Benchmarking

Five leading uMLIPs were evaluated against the generated AIMD data:

1. ORB-v3
2. MACE-MP-0a
3. MACE-MPA-0
4. fairchem ODAC23
5. fairchem OMAT

Validation Metrics:

• Static Validation: Mean Absolute Error (MAE) for Energy, Force, and Stress was calculated by sampling 1,000 random structures from each AIMD trajectory.
• Dynamic Validation: A 1 ns ramp simulation (300 K $\to$ 2000 K) was performed using ORB-v3. Structures were sampled and re-evaluated with DFT to measure "generative error" (error accumulation over time) compared to static validation metrics.

3. Key Contributions

1. High-Temperature MOF Dataset: Creation of one of the first large-scale, high-fidelity datasets capturing equilibrium dynamics, thermal distortions, and early-stage decomposition (linker degradation, metal node aggregation) for nine distinct MOFs.
2. Systematic Benchmarking: A rigorous evaluation of five state-of-the-art uMLIPs across a challenging out-of-domain regime (high temperature, non-equilibrium).
3. Generative Error Analysis: Demonstration that static validation metrics significantly underestimate the actual errors incurred during long-timescale molecular dynamics simulations, particularly as structures decompose.

4. Key Results

A. Trajectory Characterization (Physical Insights)

• 300 K: Systems remained crystalline with minor fluctuations.
• 1000 K: Enhanced thermal motion and RMSD increases were observed, but no significant bond breaking or decomposition products formed.
• 2000 K: Systematic decomposition occurred.
  • Linker Degradation: Organic linkers broke down into gases (CO, CO₂, H₂, H₂O). Notably, the imidazolate linker in ZIF-8 and the azolate linker in CALF-20 showed high thermal stability compared to others.
  • Metal Nodes: Metal coordination numbers decreased due to bond cleavage.
  • Structural Change: Loss of long-range order (amorphization) and significant RMSD increases.

B. Model Accuracy (Static Validation)

• Top Performers: ORB-v3 and fairchem OMAT achieved the lowest errors.
  • Energy MAE: ~3.2–3.6 meV/atom.
  • Force MAE: ~94–120 meV/Å (significantly better than others, which were >180 meV/Å).
• Poor Performers: MACE models and fairchem ODAC23 showed energy errors exceeding 10 meV/atom.
• Temperature Dependence:
  • At 300 K, all models performed reasonably well on energy but struggled with forces.
  • At 2000 K, errors for all models increased drastically (Energy MAE doubled, Force MAE reached 150–300 meV/Å).
• System Dependence: Accuracy correlated with structural complexity rather than metal identity (Zn vs. Zr showed similar error ranges).

C. Generative Error (Long-Timescale Dynamics)

• The "Generative Gap": When ORB-v3 was used to run a 1 ns simulation, the error accumulation was far worse than static validation suggested.
  • At 300 K, the dynamic loss was already higher than the static validation loss, indicating inaccurate force predictions even in stable regimes.
  • As temperature rose and bonds broke, the loss increased linearly and did not plateau.
  • At 2000 K, the weighted loss was 3–4 times higher than the static AIMD validation loss.
• Conclusion: Current universal models cannot reliably simulate the dynamics of high-temperature decomposition, even if they predict static energies with moderate accuracy.

5. Significance and Outlook

• Limitations of Current Tools: The study highlights that while uMLIPs like ORB-v3 and fairchem OMAT are excellent for equilibrium energetics, they are currently unsuitable for simulating high-temperature MOF pyrolysis without specific fine-tuning.
• Training Data Requirements: The failure of these models in extreme regimes is attributed to the lack of high-energy, non-equilibrium, and amorphous structures in their foundational training datasets (e.g., MPtrj, sAlex).
• Future Directions: To accurately model MOF-derived catalysts, future uMLIP training datasets must explicitly include:
  • High-temperature non-equilibrium states.
  • Bond-breaking events and intermediates.
  • Amorphous structures.
• Community Impact: This work provides a critical benchmark for the community to assess the robustness of new potentials and guides the development of next-generation models capable of handling extreme thermodynamic conditions.