🔬 materials science

Probing out-of-distribution generalization in machine learning for materials

This study reveals that common heuristic evaluations in materials science machine learning often overestimate generalizability and the benefits of neural scaling because they primarily test interpolation within the training domain rather than true out-of-distribution extrapolation, where increasing data or model size yields diminishing or adverse returns.

Original authors: Kangming Li, Andre Niyongabo Rubungo, Xiangyun Lei, Daniel Persaud, Kamal Choudhary, Brian DeCost, Adji Bousso Dieng, Jason Hattrick-Simpers

Published 2026-08-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kangming Li, Andre Niyongabo Rubungo, Xiangyun Lei, Daniel Persaud, Kamal Choudhary, Brian DeCost, Adji Bousso Dieng, Jason Hattrick-Simpers

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quest to discover new materials, scientists have increasingly turned to artificial intelligence to speed up the process. Imagine a vast library containing millions of recipes for different substances, each with unique properties like strength, conductivity, or how much energy it takes to form. Traditionally, finding a new material meant testing one recipe after another in a lab, a slow and expensive endeavor. Machine learning offers a shortcut: by studying the existing library, a computer can learn the rules of chemistry and predict the properties of materials it has never seen before. The ultimate goal is to build models that are truly general, meaning they can make accurate predictions about completely new types of matter, not just ones that look similar to what they have already studied. This ability to handle the unknown is crucial for solving global challenges, from creating better batteries to designing more efficient solar panels. However, there is a lingering question about how well these digital tools actually work when faced with the truly unfamiliar.

A team of researchers set out to test the limits of these machine learning models in the field of materials science. They wanted to know if the models were genuinely learning the deep laws of nature or if they were simply memorizing patterns that happened to look like new discoveries. To do this, they designed a massive experiment involving over 700 different challenges. In each challenge, they trained a computer model on a large collection of known materials and then asked it to predict the properties of a specific group of materials that were completely absent from its training data. These groups were defined by specific rules, such as "all materials containing a certain element" or "all materials with a specific crystal shape." The researchers tested a wide variety of models, ranging from simple, established algorithms to complex, modern neural networks that mimic the human brain.

The results were surprising. In the vast majority of these 700 challenges, the models performed exceptionally well, even when the test materials contained elements or structures they had never encountered during training. Simple models, which use basic decision-making rules, often performed just as well as the most sophisticated deep learning systems. This high level of success suggested that the models were not struggling to generalize to new chemistry as much as previously thought. However, the researchers dug deeper to understand why this was happening. They discovered that the definition of "new" in these tests was often misleading. While the test materials were technically different in terms of their chemical ingredients, they often occupied the same general neighborhood in the mathematical space that the models use to understand materials. In other words, the models were not truly venturing into the unknown; they were simply filling in the gaps between what they already knew.

To prove this, the team mapped out the mathematical landscape where these materials live. They found that for the tasks where models succeeded, the new materials were actually surrounded by the training data, allowing the models to make safe guesses based on nearby examples. But when they looked at the few tasks where the models failed, they found that the test materials were truly isolated, far away from any data the models had seen. This distinction revealed a critical flaw in how the field has been measuring success. Many of the tests that were celebrated as proof of a model's ability to handle the unknown were actually just tests of its ability to interpolate, or guess the middle ground between known points. The models were not demonstrating a magical ability to extrapolate to the far reaches of chemical space; they were just connecting the dots in a region they already understood.

The study also challenged a popular idea in artificial intelligence known as scaling. The prevailing belief is that if you simply make the training dataset larger or let the model train for longer, its ability to generalize will always improve. The researchers tested this by feeding the models more and more data. For the easy tasks where the new materials were close to the old ones, performance did get better with more data. But for the truly difficult tasks, where the materials were far outside the training zone, adding more data or training time did not help. In some cases, it actually made the models worse, causing them to overfit to the known data and lose their ability to guess correctly about the unknown. This suggests that the promise of scaling up to solve all generalization problems is overstated when it comes to truly novel materials.

The researchers concluded that the field needs to be more careful about how it defines and tests for generalization. The current methods, which rely on simple rules to separate training data from test data, often create illusions of success. They show that models are good at what they are good at, but they do not necessarily prove that these models can handle the truly difficult, unseen challenges that will drive future scientific breakthroughs. The study does not say that machine learning is failing; rather, it suggests that we have been asking the wrong questions. By recognizing that many of our "hard" tests are actually quite easy for the models, scientists can now focus on designing truly difficult challenges. This shift will help identify the real gaps in our understanding and guide the development of models that can truly navigate the uncharted territories of materials science.

Technical Summary: Probing Out-of-Distribution Generalization in Machine Learning for Materials

Problem Statement

Scientific machine learning (ML) aims to develop generalizable models with broad applicability, particularly in materials science where foundational deep learning models are touted for achieving unprecedented out-of-distribution (OOD) generalization. However, the assessment of this generalizability often relies on heuristics (e.g., excluding specific elements or structural symmetries from training data) that may introduce significant biases. These heuristic-based evaluations frequently conflate interpolation (predicting within the training domain) with true extrapolation (predicting outside the training domain). Consequently, conclusions regarding the superiority of state-of-the-art architectures and the benefits of neural scaling (increasing model size, dataset size, or compute) may be overstated if the OOD test sets do not genuinely lie outside the training distribution.

Methodology

The authors conducted a systematic evaluation of over 700 OOD tasks across three large materials databases: Joint Automated Repository for Various Integrated Simulations (JARVIS), Materials Project (MP), and Open Quantum Materials Database (OQMD).

Evaluation Setup:

OOD Criteria: Six specific criteria were used to define OOD test sets: (1) materials containing a specific element X, (2) materials containing any element in period X, (3) materials containing any element in group X, (4) materials of space group X, (5) materials of point group X, and (6) materials of crystal system X. Tasks with fewer than 200 test samples were excluded.
Models: A representative set of ML models was evaluated, covering diverse architectures and input representations:
- Tree ensembles: Random Forest (RF) and XGBoost (XGB) using Matminer descriptors (compositional and structural features).
- Neural Networks: Gaussian Multipole (GMP) expansion on electron density.
- Graph Neural Networks: Atomistic Line Graph Neural Network (ALIGNN).
- Large Language Models: LLM-Prop using crystal text descriptions.
Metrics: Performance was measured using Mean Absolute Error (MAE) and the coefficient of determination ( $R^2$ ).

Analytical Techniques:

SHAP Analysis: To distinguish between compositional and structural sources of bias in poor-performing tasks, a secondary model was trained to correct the prediction errors of the primary model. SHapley Additive exPlanations (SHAP) values were used to quantify the contribution of compositional vs. structural features to these corrections.
Representation Space Analysis: Uniform Manifold Approximation and Projection (UMAP) was used to project high-dimensional material representations (embeddings for deep learning, descriptors for tree models) into 2D. Kernel density estimation was applied to the training data to classify test points as either representationally In-Distribution (ID) (high density) or representationally OOD (low density).
Scaling Laws: Learning curves were generated by varying training time (epochs) and training set size to observe their impact on OOD performance, specifically comparing representationally ID vs. representationally OOD tasks.

Key Results

1. Robust Generalization Across Most Heuristic OOD Tasks
Contrary to the assumption that OOD tasks are inherently difficult, most tasks (including leave-one-element-out, leave-one-period-out, and leave-one-group-out) yielded high performance ( $R^2 > 0.95$ ) across all models, including simple boosted trees (XGB).

Chemistry-based tasks: 85% of leave-one-element-out tasks for ALIGNN and 68% for XGB achieved $R^2 > 0.95$ . Poor performance was primarily observed for non-metals (H, F, O), often due to systematic biases (e.g., overestimation of formation energies) rather than a complete failure to predict trends.
Structure-based tasks: Performance was even higher, with 88% of leave-one-space-group-out tasks achieving $R^2 > 0.98$ . Even when using "prototype-like" grouping to ensure structural dissimilarity, deep learning models maintained high accuracy.

2. The Interpolation vs. Extrapolation Distinction
Analysis of the representation space revealed that most "challenging" heuristic OOD tasks actually contain test data that lie within the high-density region of the training distribution (representationally ID).

For example, in leave-one-element-out tasks for Mg, 76% of test structures were representationally ID.
Conversely, tasks with poor performance (e.g., leave-O-out) contained a high proportion of representationally OOD data (87% of test data had low density estimates and near-zero $R^2$ ).
The "emergent capability" of predicting materials with 5+ elements was found to be largely interpolation, as 92% of such test data fell within the training domain's representation.

3. Divergent Scaling Behaviors
The study found that neural scaling laws (improving performance by increasing data or compute) do not universally apply to OOD generalization:

Representationally ID tasks: Increasing training set size or training time consistently improved performance, adhering to standard scaling laws.
Representationally OOD tasks: Increasing training set size or training time often led to marginal improvements or even degradation in performance. For instance, in the leave-H-out task, increasing the training set size from $10^4$ to $10^6$ structures caused the OOD MAE to increase five-fold due to systematic overestimation biases reinforced by more training data.

4. Model Performance Nuances

Tree Ensembles vs. Deep Learning: Simple tree models (XGB, RF) often performed as well as or better than complex deep learning models (ALIGNN, GMP) in chemistry-based OOD tasks, particularly for difficult elements like H, F, and O.
LLM Limitations: LLM-Prop, despite good in-distribution performance, performed poorly in most leave-one-element-out tasks, potentially due to a lack of inductive bias regarding elemental similarity (treating elements as discrete text tokens rather than chemically related entities).

Significance and Claims

The paper argues that the current evaluation of ML generalizability in materials science is flawed because many heuristic-defined OOD tests are not genuinely difficult; they primarily evaluate interpolation capabilities.

Re-evaluating Generalizability: The authors claim that the "emergent" generalization capabilities attributed to large foundation models may be overstated because the test sets used often fall within the training domain's representation.
Scaling Misconceptions: The study challenges the assumption that scaling up models and datasets will automatically solve OOD generalization. For truly challenging tasks (representationally OOD), scaling can be ineffective or detrimental.
Call for Rigorous Benchmarking: The authors advocate for a shift in how OOD tasks are defined and evaluated. They suggest that future benchmarks must distinguish between statistical OOD (different attribute distribution) and representationally OOD (outside the training manifold) to truly test extrapolation capabilities and avoid overestimating the benefits of neural scaling.

In summary, the paper posits that while ML models exhibit remarkable transferability across chemical and structural groups, this is often because the "unseen" data is not truly unseen in the model's representation space. True extrapolation remains a significant challenge where scaling laws may not hold, necessitating more rigorous test scenarios to guide the development of truly generalizable scientific ML models.

Technical Summary: Probing Out-of-Distribution Generalization in Machine Learning for Materials

Problem Statement

Methodology

Key Results

Significance and Claims

More like this