🧬 biology

Monroe: A Molecular Foundation Model for In-Context Probabilistic Inference

This paper introduces Monroe, a scalable molecular foundation model trained on 81 million molecules with enhanced stereochemical representations and novel training objectives, which achieves state-of-the-art performance in bioassay activity prediction by leveraging a prior-data-fitted model (TabPFN) for in-context probabilistic inference and demonstrating that this downstream adaptation strategy also significantly improves existing models like MiniMol and CheMeleon.

Original authors: Blazej Banaszewski, Andrew W. Fitzgibbon

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Blazej Banaszewski, Andrew W. Fitzgibbon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

In the race to discover new medicines, scientists face a stubborn bottleneck: the most reliable way to know if a molecule will cure a disease is to build it and test it in a lab. This process is slow, expensive, and limited by the number of specialized facilities that can perform these experiments. Because of this, the data available to train computer programs is often scarce. To overcome this, researchers have turned to a strategy similar to how a child learns to recognize animals before ever seeing a specific dog or cat. They train massive computer models on vast libraries of chemical structures and their basic physical properties, teaching the software a general sense of how molecules behave. Once this foundation is built, the model can be adapted to predict specific biological outcomes, such as whether a compound will bind to a virus, even when only a handful of experimental results are available for that specific task. This approach, known as a molecular foundation model, aims to bridge the gap between the endless world of possible chemicals and the limited data of real-world drug testing.

A team of researchers has introduced a new model named Monroe, which represents a significant step forward in this field. The name honors Elizabeth Monroe Boggs, a pioneer in computational chemistry, but the model itself is a modern machine learning system designed to understand the complex language of molecules. The researchers trained Monroe on a massive dataset containing over 81 million molecules, a scale far larger than previous attempts. This dataset included detailed quantum chemistry calculations, which describe the energy and structure of atoms, as well as data from thousands of biological assays that measure how molecules interact with living systems. By exposing the model to this immense variety of information, the researchers enabled it to learn general rules of chemistry that apply across different types of problems, rather than just memorizing specific examples.

One of the most critical innovations in Monroe is how it handles the three-dimensional shape of molecules, specifically their "handedness." Many molecules exist in two forms that are mirror images of each other, much like a left and right hand. While these mirror images have the same atoms connected in the same order, they often behave completely differently in the human body; one might be a medicine, while its mirror image could be toxic. Standard computer models often struggle to tell these versions apart because they rely on mathematical descriptions that look identical for both shapes. Monroe solves this by explicitly adding extra connections in its internal map of the molecule that act as flags for these mirror-image differences. This allows the model to distinguish between the two forms with high accuracy, a capability that is essential for predicting how a drug will actually work in a biological system.

The researchers also refined how the model learns from its training data. Instead of treating every learning task as equally important, Monroe uses a smart weighting system that automatically adjusts its focus. It pays more attention to tasks where the data is clear and reliable, and less attention to tasks that are noisy or uncertain. Furthermore, the model is trained to clean up imperfect data. In the real world, the 3D shapes of molecules used in computer simulations are often rough approximations. Monroe is taught to take these rough shapes and refine them into more accurate, stable forms, a process that helps the model understand the underlying physical rules governing molecular stability. This ability to "denoise" the data strengthens its predictions for real-world applications.

When tested against other leading models, Monroe demonstrated superior performance, particularly in the most challenging scenarios known as "activity cliffs." These are cases where two molecules are nearly identical in structure but have drastically different biological effects. Predicting these sharp differences is notoriously difficult for artificial intelligence, yet Monroe outperformed its competitors in these tests. The researchers also found that their method of adapting the model to new tasks was highly effective. Instead of retraining the entire model for every new drug target, they used a technique that allows the model to learn from a small set of examples in a single step, similar to how a human can learn a new rule after seeing just a few examples. This approach, which they applied not only to Monroe but also to improve two other existing models, proved to be a powerful and general strategy.

The study concludes that combining large-scale pre-training with specific architectural improvements, such as better handling of molecular handedness and smart data weighting, creates a more robust tool for drug discovery. While the model does not solve every problem in molecular prediction, and the challenge of activity cliffs remains difficult, Monroe sets a new standard for what is possible. It shows that by teaching computers a deeper, more nuanced understanding of chemical structure and behavior, scientists can build tools that are better equipped to navigate the vast and complex landscape of potential medicines, potentially speeding up the journey from a chemical idea to a life-saving treatment.

Technical Summary: Monroe – A Molecular Foundation Model for In-Context Probabilistic Inference

Problem Statement

Bioassay activity prediction in drug discovery is inherently data-limited due to the high cost, time consumption, and specialized facility requirements of wet-lab experiments. Traditional machine learning approaches often fit low-capacity, task-specific models directly to these small datasets, limiting their generalizability. While recent Molecular Foundation Models (MFMs) aim to learn transferable molecular representations from large-scale data, challenges remain in handling stereochemistry, balancing multi-task objectives, and adapting to downstream tasks with scarce data without overfitting. Furthermore, existing benchmarks often lack rigorous statistical evaluation, making it difficult to distinguish genuine performance improvements from variance.

Methodology

The paper introduces Monroe, a new MFM that integrates several architectural and training innovations to address these challenges.

1. Architecture and Representation

Backbone: Monroe builds upon GRIT, a graph transformer augmented with inductive biases from graph learning. It utilizes random-walk positional encodings, graph attention mechanisms, and a virtual node for global information propagation.
Sparse Attention: Unlike standard dense attention, Monroe evaluates attention only on observed edges, reducing complexity to $O(|E|)$ and enabling scalability to large molecular graphs.
Stereochemistry-Aware Rewiring: To address the limitation of rotation-invariant encoders (which cannot distinguish enantiomers or E/Z isomers based on coordinates alone), Monroe augments the molecular graph with auxiliary edges.
- E/Z Isomers: "Parallel" edges connect same-side substituents, and "diagonal" edges connect opposite-side substituents, encoding the geometric configuration.
- R/S Chirality: Radial edges connect the lowest-priority substituent to others, while cyclic edges follow a specific direction (e.g., $a \to b \to c \to a$ ) to encode the chiral arrangement, allowing the network to distinguish enantiomers.
Featurization: Node and edge features combine categorical chemical codes (e.g., valence, hybridization), continuous RDKit descriptors, bond lengths (via Radial Basis Functions), and random-walk positional encodings.

2. Pre-training Strategy

Monroe is pre-trained on two complementary datasets:

PM6 Dataset: ~81 million molecules providing semi-empirical quantum chemistry properties (62 graph-level targets) and 3D conformer data.
PCBA Dataset: ~1.56 million molecules with binary bioassay readouts (1,089 filtered assays), providing sparse but biologically relevant supervision.

Training Objectives:

Multi-Task Learning (MTL): The model employs Uncertainty Weighting (UW) to balance losses across 1,152 tasks (62 PM6, 1,089 PCBA, 1 conformer denoising). This automatically downweights noisy or saturated tasks.
Conformer Denoising: Inspired by Uni-Mol+, the model is trained to recover high-quality equilibrium structures from noisy input coordinates (interpolated between RDKit approximations and PM6-optimized structures). The loss is computed after optimal rigid alignment to handle rotational/translational invariance.
Embedding Decorrelation: A covariance penalty is added to the loss to drive off-diagonal entries of the batch fingerprint covariance to zero, making the embeddings more "tabular-like" and suitable for downstream tabular predictors.
Task-Specific Losses: Loss functions are tailored to target types (e.g., Huber for signed values, Log-Huber for scale parameters, Beta NLL for bounded ratios, Ordinal Cross-Entropy for counts).

3. Downstream Adaptation: In-Context Prediction

Instead of standard fine-tuning, Monroe utilizes a Prior-Data-Fitted (PFN) approach for downstream tasks:

The pre-trained encoder is frozen to generate fixed embeddings for molecules.
These embeddings, paired with task labels, form a support set passed to TabPFN, a model trained to approximate Bayesian posterior-predictive inference on tabular data.
TabPFN performs in-context prediction in a single forward pass, adapting to the specific downstream task without gradient-based fine-tuning or hyperparameter optimization.

Key Contributions

Scale and Data: Pre-training on a massive dataset of ~81 million molecules from the PM6 quantum chemistry dataset, combined with PCBA bioassays.
Stereochemistry Handling: A novel graph rewiring strategy that explicitly encodes E/Z and R/S configurations as topological features, addressing a known failure mode of invariant encoders.
Training Innovations: The integration of conformer denoising, embedding decorrelation, and uncertainty-weighted multi-task balancing.
Downstream Strategy: Demonstrating that replacing standard fine-tuning with TabPFN-based in-context inference significantly improves performance, particularly in data-scarce regimes.
Rigorous Evaluation: Adoption of a principled pairwise comparison framework (Tukey HSD + Benjamini-Hochberg) to ensure statistically significant performance claims.
Generalization: The strategy of augmenting existing models (MiniMol and CheMeleon) with TabPFN (creating MiniMolPFN and CheMeleonPFN) yields new state-of-the-art variants, suggesting the downstream adaptation strategy is model-agnostic.

Results

The paper evaluates Monroe against leading models (MiniMol, CheMeleon, MolFormer) and classical baselines (GBM with ECFP4) on two primary benchmarks:

Polaris (Generalization): Monroe achieves a 100% win rate across 28 bioactivity tasks, being statistically indistinguishable from or better than the best method on all tasks. It achieves the lowest mean rank (3.71).
MoleculeACE (Activity Cliffs): Monroe leads on all metrics, achieving the lowest overall RMSE (0.629) and the lowest "Cliff" RMSE (0.737), with an 86.7% win rate. This suggests the model is better at capturing local activity changes in structurally similar molecules.
Ablation Studies:
- Stereochemistry: Removing stereochemical augmentation significantly degrades performance on Polaris.
- Conformer Denoising: Removing this objective collapses the Polaris win rate and degrades MoleculeACE performance.
- Datasets: Removing either PM6 or PCBA data substantially hurts performance, indicating both physical and biological signals are critical.
- Adaptation: For Monroe, using frozen embeddings with TabPFN outperforms both native fine-tuning and LoRA-based adapters.
Data Efficiency: Monroe maintains superior performance even when the support set size is reduced to 10% of the labeled data, though differences diminish at very low data regimes (<100 molecules).
Ensembling: Ensembling Monroe with other PFN-augmented models (MiniMolPFN, CheMeleonPFN) further improves performance, driven by error diversity rather than individual member quality.

Significance and Claims

The paper claims that Monroe represents a significant step forward in molecular foundation modeling by effectively integrating prior-data-fitted models into the MFM pipeline. The authors argue that the combination of large-scale pre-training, explicit stereochemical handling, and in-context probabilistic inference allows the model to generalize well to unseen biological tasks and activity cliffs, which are critical for drug discovery.

The work emphasizes that the downstream adaptation strategy (using TabPFN) is a generalizable improvement, as it successfully enhanced two other leading models (MiniMol and CheMeleon) to create new state-of-the-art variants. The authors conclude that while predicting sharp activity cliffs remains an open problem, Monroe provides a robust, open-source foundation for molecular property prediction that reduces the reliance on task-specific fine-tuning and improves performance in data-constrained scenarios. The code and models are made publicly available to facilitate further research.