🔬 materials science

BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models

This paper introduces BOOM, the first chemically-informed open-source benchmark for systematically evaluating out-of-distribution (OOD) generalization in molecular property prediction, revealing that current machine learning models struggle to extrapolate beyond their training data and highlighting the need for new approaches to achieve robust OOD performance.

Original authors: Evan R. Antoniuk, Shehtab Zaman, Tal Ben-Nun, Peggy Li, James Diffenderfer, Busra Sahin, Obadiah Smolenski, Everett Grethel, Tim Hsu, Anna M. Hiszpanski, Kenneth Chiu, Bhavya Kailkhura, Brian Van Esse

Published 2026-09-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Evan R. Antoniuk, Shehtab Zaman, Tal Ben-Nun, Peggy Li, James Diffenderfer, Busra Sahin, Obadiah Smolenski, Everett Grethel, Tim Hsu, Anna M. Hiszpanski, Kenneth Chiu, Bhavya Kailkhura, Brian Van Essen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The quest to discover new medicines and materials often begins with a simple, daunting question: which of the billions of possible chemical structures will actually work? For decades, scientists have relied on trial and error, but a new wave of artificial intelligence promises to speed this up by predicting how a molecule will behave before it is ever built. These computer models are trained on vast libraries of known chemicals, learning to recognize patterns that link a molecule's shape to its properties, such as how well it dissolves in water or how much energy it releases when burned. The hope is that these models can then look at a completely new, unseen molecule and accurately guess its behavior, effectively allowing researchers to design the future of chemistry on a computer screen.

However, there is a significant catch. Most of these AI models are excellent at recognizing patterns within the data they were taught, but they often stumble when asked to predict the properties of molecules that are truly different from anything in their training library. In the world of machine learning, this is known as the "out-of-distribution" problem. It is the difference between a student who memorizes the answers to a practice test and one who can actually solve a new, unexpected problem. If a model cannot generalize beyond its training data, it becomes a tool for confirming what we already know, rather than a tool for discovering the unknown. This limitation is a major bottleneck for the next generation of drug discovery and material science, where the goal is to find molecules that push the boundaries of what is currently possible.

A team of researchers from Lawrence Livermore National Laboratory and Binghamton University has taken a hard look at this problem with a new benchmark called BOOM. Rather than just testing how well models perform on familiar data, they designed a rigorous set of tests specifically to see if these artificial intelligence systems can handle the unfamiliar. They gathered ten different datasets containing thousands of molecules, ranging from small organic compounds to complex structures with specific electronic properties. For each dataset, they carefully separated the molecules into three groups: a training set, a standard test set of familiar molecules, and a special "out-of-distribution" test set. This special group consisted of molecules with extreme properties—values so high or so low that they rarely appear in the training data. The researchers then put fifteen different machine learning models through their paces, asking each one to predict the properties of these extreme molecules.

The results were sobering. Despite the impressive capabilities of modern artificial intelligence, the study found that no single model could consistently predict the properties of these extreme molecules across all tasks. Even the best-performing models made errors on these difficult, unseen cases that were three times larger than their errors on familiar data. The researchers observed a common failure mode: the models were good at grouping similar molecules together, but they failed to extend their predictions into the unknown territory. Instead of guessing a value that was truly outside the range of what they had seen, the models tended to compress their predictions, pulling extreme values back toward the average. This behavior suggests that the models are learning shortcuts that work well for standard data but break down when faced with the true novelty required for scientific discovery.

The team also investigated why these models struggled and tested several common strategies to fix the problem. They examined whether pre-training models on massive datasets of billions of molecules—a technique that has revolutionized language models—would help. Surprisingly, they found that while this pre-training made the models better at handling familiar data, it did not improve their ability to predict extreme values. In fact, for some models, the pre-training actually made the out-of-distribution performance worse. They also tested whether simply adding more data to the training set would help. While including a small number of extreme examples did improve performance for some properties, it was not a universal cure. The study suggests that the way these models are currently built, particularly their reliance on text-based representations of molecules, may be fundamentally limiting their ability to understand the deep physical laws that govern chemical behavior.

One of the most revealing findings was that the type of data representation mattered more than the size of the model. Models that used three-dimensional geometric information about the atoms and their positions performed significantly better than those that relied on linear text strings, which are like chemical names written out in a sequence. The three-dimensional models, which respect the physical symmetries of space, were able to generalize much better to the extreme cases. This points to a clear path forward: to build AI that can truly discover new chemistry, the models need to be grounded in the physical reality of how atoms move and interact, rather than just the patterns of how they are described in text. The researchers concluded that achieving strong performance on these difficult tasks will likely require a combination of larger, more diverse datasets and models that explicitly understand the electronic structure of molecules.

The BOOM benchmark serves as a reality check for the field, showing that the current generation of chemical AI, while powerful, is not yet ready to reliably navigate the unknown. The study does not claim that the problem is unsolvable, but it does rule out the idea that simply scaling up existing models or using standard pre-training techniques will be enough to solve it. Instead, it highlights that the next frontier in chemical machine learning requires a shift in how these systems are designed. By providing a standardized way to test for this specific weakness, the researchers hope to guide the community toward building models that are not just good at remembering the past, but are truly capable of imagining the future. The path to discovering the next generation of life-saving drugs and advanced materials may depend on solving this specific puzzle of generalization.

Technical Summary: BOOM – Benchmarking Out-Of-distribution Molecular Property Predictions

Problem Statement

Data-driven molecular discovery relies heavily on machine learning (ML) to screen and design novel molecules. However, the core challenge of molecular discovery is inherently an out-of-distribution (OOD) prediction problem. Success requires models to predict properties for molecules that either exhibit values extrapolating beyond the training dataset or possess previously unconsidered chemical substructures.

Despite the critical importance of OOD performance, current ML models for molecular property prediction lack systematic evaluation in this domain. Existing benchmarks often rely on insufficiently challenging datasets (e.g., random splits) that fail to measure real-world generalization. Furthermore, there is a lack of empirical knowledge regarding how pretraining tasks, model architectures, and dataset diversity impact the ability of chemical foundation models to generalize across all chemical systems.

Methodology: The BOOM Benchmark

The authors introduce BOOM (Benchmarking Out-Of-distribution Molecular Property Predictions), a standardized framework designed to assess the OOD generalization performance of molecule property prediction models.

1. Defining Out-of-Distribution

Unlike traditional OOD definitions based on input space shifts (covariate shift), BOOM defines OOD as a mismatch in the property space (concept or label shift). Formally, OOD is defined as low-density regions over the property space where the probability density of test labels $P(y_{test})$ is lower than that of the training labels $P(y_{train})$ .

Splitting Strategy: The authors use Kernel Density Estimation (KDE) to model the property distribution. Molecules with the lowest probability scores (the tails of the distribution) are held out to form the OOD test set. This ensures the model is tested on its ability to extrapolate to extreme property values not seen during training.

2. Datasets

BOOM comprises 10 diverse OOD tasks derived from:

QM9 Dataset: 8 quantum chemical properties (Isotropic polarizability $\alpha$ , Heat capacity $C_v$ , HOMO/LUMO energies, HOMO-LUMO gap, Dipole moment $\mu$ , Electronic spatial extent $\langle R^2 \rangle$ , Zero point vibrational energy ZPVE).
10k Dataset: Derived from the Cambridge Structural Database, containing 10,206 experimentally synthesized small organic molecules with Density Functional Theory (DFT) calculated values for Density and Solid Heat of Formation (HoF).
Lipophilicity Dataset: 4,200 experimental measurements of the octanol/water distribution coefficient, included as an exemplary dataset for performing OOD evaluations on experimentally measured properties, but not part of the core 10 tasks used for the main 150 model-task combinations.

3. Models Evaluated

The benchmark evaluates 150 model-task combinations (across 10 diverse OOD tasks and 15 models) categorized by architecture and representation:

Traditional ML: Random Forest and Multilayer Perceptron (MLP) using RDKit molecular descriptors.
Graph Neural Networks (GNNs): Chemprop (2D topological), EGNN, IGNN (E(3)-invariant/equivariant), MACE, and GotenNet.
Transformers/LLMs: MoLFormer, ChemBERTa, Regression Transformer (RT), and ModernBERT.
Hybrid Architectures: Graphormer-3D, Geoformer, and Equivariant Transformer (ET).

4. Metrics

RMSE: Root Mean Square Error for both In-Distribution (ID) and OOD splits.
Binned $R^2$ : To address the issue where OOD samples span disparate ranges (inflating standard $R^2$ ), the authors calculate the average $R^2$ specifically for samples in the lower and upper tails of the property distribution.

Key Results

1. Lack of Universal Generalization

No existing model achieves strong generalization across all OOD tasks.

Performance Gap: Even the top-performing models exhibit an average OOD error 3 $\times$ higher than their in-distribution error.
Task Dependency: While some models perform well on specific tasks (e.g., HoF, Density, ZPVE, $C_v$ ), they fail significantly on others. Notably, no model achieves good OOD performance on electronic structure properties (HOMO, LUMO, Gap, and Dipole moment $\mu$ ). The authors hypothesize this is due to a lack of explicit electronic structure in current molecular representations.

2. Architecture and Representation Impact

3D and Graph Models: Models with high inductive bias, particularly 3D equivariant GNNs (e.g., GotenNet, Geoformer) and hybrid architectures, generally outperform SMILES-based transformer models on OOD tasks.
SMILES Limitations: Transformer models relying on SMILES strings (e.g., ChemBERTa, MoLFormer) often struggle to extrapolate beyond the training regime, despite strong ID performance.
ModernBERT: Among NLP-style models, ModernBERT showed significant improvements over other transformers for OOD tasks (e.g., reducing HoF and $C_v$ RMSE by >58% and >78% respectively), suggesting architectural improvements in attention mechanisms can benefit chemistry domains.

3. Pretraining Strategies

Ineffectiveness of Current Pretraining: Surprisingly, large-scale language pretraining (Masked Language Modeling for ChemBERTa/MoLFormer; Permutation Language Modeling for Regression Transformer) did not improve OOD performance. In some cases, it significantly decreased OOD $R^2$ .
Correlation Matters: Ablation studies on supervised pretraining revealed that OOD performance only improves when the pretraining task is strongly correlated with the downstream fine-tuning task (Pearson correlation > 0.35). When tasks are uncorrelated, OOD performance degrades.

4. Hyperparameters and Data

Hyperparameter Tuning: Optimizing hyperparameters specifically for OOD performance yielded marginal improvements for some properties (e.g., -60% RMSE for Density) but did not solve the generalization gap.
Data Augmentation: Adding a small fraction (up to 4%) of OOD molecules to the training set improved generalization for most properties, highlighting the sensitivity of models to the presence of extreme-valued data.

Significance and Claims

The paper positions BOOM as a frontier challenge for the chemical machine learning community. Its primary contributions and claims are:

Systematic Benchmarking: It provides the first large-scale, standardized benchmark for OOD molecular property prediction, moving beyond random splits to test true extrapolation capabilities.
Current Limitations: The authors claim that current state-of-the-art (SOTA) models, including large foundation models, are not ready for reliable discovery of molecules with properties that extrapolate beyond known data. The gap between ID and OOD performance remains a critical bottleneck.
Strategic Insights:
- Pretraining: Current unsupervised pretraining strategies (MLM/PLM) may not convey the necessary chemical information for OOD extrapolation. Future foundation models may require pretraining tasks more closely aligned with downstream property prediction.
- Representations: High-inductive bias models (3D GNNs) currently outperform attention-based models for OOD, but scalability remains an issue.
- Evaluation: The paper argues that OOD performance must be a primary evaluation criterion, as high ID performance does not guarantee OOD generalization.

The authors conclude that achieving strong OOD generalization will likely require a combination of larger datasets, molecular representations that explicitly capture electronic structure, and new pretraining paradigms tailored to the specific challenges of chemical extrapolation. The benchmark is open-sourced to facilitate this progress.