How good are universal MLIPs for alloys? A benchmark is only as trustworthy as its controls

Imagine a world where scientists can design new super-strong alloys for jet engines or ultra-efficient batteries without ever melting a single piece of metal in a lab. Instead, they use powerful computer simulations to predict how atoms will behave. The gold standard for these predictions is a complex math method called Density Functional Theory (DFT). Think of DFT as a master chef who can cook up the perfect recipe for any material, but it takes hours to prepare a single dish. It's accurate, but painfully slow.

To speed things up, researchers have created "machine-learned interatomic potentials" (or MLIPs). You can think of these as sous-chefs. They have studied the master chef's recipes so thoroughly that they can cook up a meal in a split second, with almost the same taste. These AI models are getting better and better, and scientists are racing to see which one is the best. They usually judge them by looking at a scoreboard: the model that makes the fewest mistakes in predicting energy gets the top spot. But here's the catch: what if the scoreboard itself is rigged? What if the "mistakes" aren't because the sous-chef is bad at cooking, but because they are using a different measuring cup than the judge?

This is exactly the puzzle tackled in a new study by Gus Hart and Jacob Pursley. They decided to stop taking the leaderboards at face value and started acting like detectives. They gathered fourteen of the most popular AI models and put them to the test on a specific type of material: metallic alloys. But instead of just checking their energy scores against the usual reference, they introduced a series of "controls"—like a scientist checking their own equipment before trusting a result. They asked: Are these models actually learning the physics of atoms, or are they just memorizing the specific rules and conventions of the database they were trained on?

The results were a bit of a shock. The paper finds that the current way we rank these AI models is deeply flawed. The "winners" on the public leaderboards often aren't winning because they are smarter; they are winning because they happened to inherit the same measuring conventions as the database they were tested against. When the authors stripped away these artificial advantages, the rankings changed dramatically. Some models that looked like failures turned out to be quite good, while the "champions" saw their lead shrink significantly.

Here is what the study actually discovered, broken down into the story of the investigation:

The "Measuring Cup" Problem
Imagine you are judging a baking contest. The judge says, "The winner is the one whose cake weighs the least." But it turns out, the judge's scale is calibrated to show 50 grams heavier than it should. If a baker happens to use the same scale, their cake will look perfect. If another baker uses a different scale, their cake looks terrible, even if it's the same size.

In the world of atoms, the "weight" is the energy of the material. The study found that many AI models were trained on a database (like the Materials Project) that uses a specific set of rules for calculating these energies. When the researchers tested these models against a completely different, independent database (called AFLOW), the models looked bad. But when the researchers "re-calibrated" the models—essentially telling them, "Hey, use the same measuring cup as the new judge"—the errors vanished.

For example, one model called CHGNet looked like it was making huge mistakes (about 126 meV/atom). But once the researchers fixed the "measuring cup" mismatch, the error dropped to 73 meV/atom. More than half of its "failure" was just a difference in bookkeeping, not a lack of skill.

The "Relaxation" Trap
There was another trick in the test. When the models were asked to predict the shape of an atom arrangement, some models were allowed to "relax" (move the atoms around to find the most comfortable spot), while others were forced to stay in the exact position the judge gave them. One model, eqV2, was so good at reading the judge's mind that it barely moved at all. It got a high score because it didn't have to do any work.

The researchers realized this was unfair. They re-ran the test so every model had to start from the same spot and do the same amount of work. When they did this, the "winner's" lead shrank by half. It turned out that the top model wasn't necessarily better at physics; it just had a head start because of how the test was set up.

The "Noise Floor" Reality Check
The authors also asked a fundamental question: How good can a model possibly be? They compared three different, independent databases of "perfect" calculations (AFLOW, Alexandria, and OQMD). They found that even these perfect databases don't agree with each other perfectly. They disagree by about 6 to 8 meV/atom on typical structures, and up to 24–28 meV/atom when things get messy.

This means there is a "noise floor." No matter how smart an AI is, it cannot be more accurate than the disagreement between two human-made, perfect calculations. The study found that the best models are now reaching this floor. They are as good as the reference data allows. But the paper warns that we can't rank them much finer than that. If two models are separated by a tiny fraction of a meV, it might just be random noise, not a real difference in quality.

The "Bad Reference" Mystery
One of the most surprising findings involved "bad" data. The researchers noticed that for about 15,000 structures, the AI models seemed to be wildly wrong, missing the target by huge amounts. It looked like the models were failing miserably.

But when they looked closer, they realized the models were actually right, and the "judge" (the reference database) was wrong. The models were all agreeing with each other, but the reference database was an outlier. By using a "consensus detector" (asking the models what they think and seeing if the judge agrees), they found that most of these "failures" were actually errors in the reference data, not the models. Once they filtered these out, the number of "bad references" dropped from 15,000 to just 1,700.

The Real Winners and Losers
After applying all these controls—fixing the measuring cups, equalizing the starting positions, and filtering out bad data—the leaderboard looked very different.

• The models trained on the newer, larger datasets (the "OMat" generation) generally performed better and generalized well to new, unseen structures.
• The models trained on older data (the "MP" generation) struggled more, but some, like MACE-MPA, were actually quite good at predicting stability once the "measuring cup" bias was removed.
• The "winner" of the raw energy scores, eqV2, still did well, but its lead wasn't as massive as it seemed. In fact, when looking at other important properties like how stiff the material is (elasticity) or how well it predicts which shapes are stable, the rankings shuffled completely.

The Takeaway
The paper concludes that a benchmark is only as trustworthy as its controls. You cannot just look at a single number on a scoreboard and declare a winner. The "best" model depends entirely on what you are trying to measure. If you want to know which model is best at predicting energy, one might win. If you want to know which is best at predicting stability, a different one might win.

The authors argue that the field needs to stop chasing a single "best" model and start using a "vector" of metrics—a list of different scores that tell the whole story. They also suggest that the next generation of models shouldn't just try to get a lower error number; they need to be tested against independent data, calibrated against the "noise floor" of human calculations, and judged on their ability to handle the messy, real-world chemistry that isn't just a repeat of the training data.

In short, the AI models are getting incredibly good—so good that they are now bumping up against the limits of our own human calculations. But to know who is truly the best, we have to stop cheating with our measuring cups and start looking at the whole picture.