Toward Controllable Catalyst Inverse Design via Large-Scale Autoregressive Pretraining

This paper introduces a large-scale autoregressive generative model pretrained on 133 million catalyst structures that enables controllable inverse design by generating valid catalysts with specific categorical and continuous properties, thereby significantly improving screening efficiency for reaction-targeted discovery.

The Gist

Imagine you are trying to invent a new type of engine part, but instead of building it out of metal, you are building it out of atoms. In the world of chemistry, finding the perfect arrangement of atoms to make a "catalyst" (a substance that speeds up chemical reactions) is like looking for a needle in a haystack the size of the universe.

Traditionally, scientists have used a "trial-and-error" approach. They guess a shape, test it, and if it fails, they try again. Later, they used computers to screen millions of guesses, but this is still slow and expensive because the computer has to check every single possibility one by one.

This paper introduces a new tool called CatGPT (Catalyst Generative Pretrained Transformer). Think of it not as a calculator that checks answers, but as a creative chef who has read every cookbook in the world and can now invent new recipes that are guaranteed to taste good.

Here is how the paper explains this breakthrough, broken down into simple concepts:

1. The "Chef" Needs to Read the Menu First (Pretraining)

Before the chef can cook a specific dish, they need to understand the basics of cooking. The researchers fed their AI model 133 million different catalyst structures. This is like the chef reading 133 million cookbooks to learn the "grammar" of atoms: which atoms like to hang out together, how they bond, and what shapes are physically possible.

• The Result: The model learned the rules of chemistry so well that it can now generate new structures that are physically valid (atoms aren't crashing into each other) 98% of the time.

2. Ordering a Specific Dish (Conditional Generation)

In the past, if you asked this chef to cook, they might just make any random dish. But scientists need specific things: "I need a catalyst that works with this specific gas" or "I need one that binds with this specific energy level."

The researchers taught the model to listen to two types of orders:

• The "Category" Order: Like saying, "I want a pizza with mushrooms and cheese." The model learned to generate structures with specific chemical ingredients (adsorbates and compositions) almost perfectly (93% accuracy).
• The "Number" Order: Like saying, "I want the pizza to be exactly 12 inches in diameter." This is harder because numbers are continuous. The researchers built a special "numerical ear" into the model's brain. Now, if you say, "I need a binding energy of -1.5," the model tries to cook a structure that matches that number.

3. The "Magic" of the Recipe Book (The Results)

The paper claims this new chef is a massive improvement over previous methods:

• Efficiency: If you were looking for a catalyst with a specific energy level, the old way was like searching a library for a book with a specific page number. You might find it 5% of the time. This new model finds it 20% of the time. That is a four-fold improvement. It means scientists can find the right catalyst 4 times faster without wasting time on bad guesses.
• Precision: When the researchers asked the model to make a catalyst for a specific reaction (like splitting water or reducing oxygen), the model successfully generated candidates that were much closer to the "perfect" target than random guessing.

4. Learning New Cuisines with Limited Ingredients (Foundation Model)

What if the chef needs to cook a dish they've never seen before, like a "Single Atom Catalyst" (a very rare type of structure)? Usually, you would need thousands of examples to teach a chef a new cuisine.

The researchers tested if their model could learn these rare dishes with very little data. They found that because the model had already read the "133 million cookbooks" during pretraining, it could adapt to these new, rare styles of cooking very quickly. It performed much better than a chef who tried to learn the new style from scratch with only a few recipes.

The Limitations (What the Chef Can't Do Yet)

The paper is honest about what the model cannot do:

• The Vocabulary Limit: The chef can only use ingredients they have seen in the 133 million cookbooks. If you ask for a brand-new element that doesn't exist in their training data, the model will get confused.
• The "Stability" Puzzle: While the model can build a great-looking "slab" (the surface of the catalyst), it's sometimes hard to know exactly what the "bulk" (the solid block underneath) looks like. It's like building a beautiful house facade but not knowing if the foundation is solid without doing extra work.

The Bottom Line

This paper presents a tool that moves catalyst discovery from "searching for a needle in a haystack" to "asking a master chef to cook exactly what you need." By training on a massive amount of data and teaching the AI to listen to specific numerical and categorical instructions, the researchers have created a system that can generate high-quality, target-specific catalysts much faster than ever before.

Technical Summary

Technical Summary: Toward Controllable Catalyst Inverse Design via Large-Scale Autoregressive Pretraining

Problem Statement
The inverse design of heterogeneous catalysts is hindered by the vast chemical space and the structural complexity of catalyst surfaces, which involve coupled surface-adsorbate interactions. While machine learning (ML) has accelerated discovery through high-throughput screening, its efficiency diminishes as the search space expands, necessitating the development of generative models capable of directly constructing catalysts with target properties. Existing autoregressive models, such as the previous CatGPT, were limited by their inability to condition generation on specific target properties (inverse design) and their lack of mechanisms to incorporate continuous numerical variables (e.g., binding energy) alongside categorical tokens. Furthermore, standard transformer architectures struggle to process scalar numerical values required for property-guided generation.

Methodology
The authors propose a conditional catalyst generative model based on the Generative Pretrained Transformer (GPT) architecture, specifically an extension of the GPT-2 framework. The methodology involves a two-stage training process and a novel architectural modification:

1. Architectural Innovation: To enable conditioning on continuous numerical properties (specifically binding energy), the authors integrated a numerical embedding layer directly into the self-attention mechanism of the transformer. A scalar condition value ($z_c$) is projected into an embedding and linearly combined with the token hidden states ($z_i$) to compute the queries, keys, and values in the attention blocks. This allows the model to jointly process tokenized structural information and continuous numerical features within a single autoregressive framework.
2. Tokenization: Catalyst structures are represented as string sequences comprising adsorbate type, chemical composition, space group, Miller indices, lattice parameters, and atomic coordinates. Continuous spatial data (coordinates, lattice lengths) are tokenized as fixed-precision strings.
3. Training Strategy:
   • Pretraining: The model was pretrained on 133 million catalyst structures from the OC20-S2EF dataset (single-point energy calculations) to learn the syntax of catalyst representations and capture global geometric patterns. A smaller baseline model was pretrained on 2 million structures.
   • Fine-tuning: The pretrained model was subsequently fine-tuned on approximately 460,000 optimized structures from the OC20-IS2RE dataset. This step biases the generative distribution toward energetically relaxed and physically stable configurations.
4. Evaluation: The model was evaluated on structural validity, optimization validity (convergence of geometry relaxation), uniqueness, novelty, and conditional match rates for both categorical properties (adsorbate type, composition) and continuous properties (binding energy).

Key Contributions

• Numerical Embedding Integration: The introduction of a numerical embedding layer that allows autoregressive transformers to condition generation on continuous variables (binding energy) without task-specific fine-tuning for each target.
• Large-Scale Pretraining: Demonstration that pretraining on 133 million structures significantly improves structural validity and the model's ability to capture relationships between condition tokens and physical structures compared to smaller-scale pretraining.
• Foundation Model Capability: Validation of the pretrained model as a "foundation model" capable of adapting to out-of-distribution (OOD) domains (oxide surfaces, single-atom catalysts) with limited data, outperforming models trained from scratch.

Results

• Generative Performance: The CatGPT-133M-FT model achieved 98% structural validity and 95% optimization validity, outperforming both the 2M-pretrained baseline and flow-matching models (CatFlow).
• Categorical Conditioning: The model achieved a 93% joint match rate for adsorbate type and composition, a significant improvement over the 2M-FT model (22%). The model adhered to categorical conditions with near-perfect fidelity.
• Continuous Conditioning (Binding Energy): For binding energy conditioning, the model achieved an approximate 20% match rate (structures within ±0.2 eV of the target). This represents a four-fold improvement over the baseline OC20 training distribution (~5%). The generated distributions shifted systematically toward target values.
• Screening Efficiency: The ability to condition on binding energy resulted in a 1.5 to 4-fold improvement in screening efficiency for reaction-targeted catalyst discovery (e.g., Hydrogen Evolution Reaction and Oxygen Reduction Reaction) without additional fine-tuning.
• Out-of-Distribution (OOD) Adaptation: When fine-tuned on OOD datasets (unseen metal alloys, oxides, and single-atom catalysts), the 133M-FT model consistently outperformed models trained from scratch in conditional generation, despite some challenges in structural validity for highly divergent domains like oxides.

Significance and Claims
The paper claims that large-scale autoregressive pretraining, combined with explicit property conditioning via numerical embeddings, provides a practical route toward controllable catalyst inverse design. The authors assert that this approach enables the direct generation of catalyst structures with target properties, overcoming the inefficiencies of traditional screening. The work establishes the model as a practical foundation model that can adapt to new catalyst domains with limited data, thereby accelerating the discovery of high-performing heterogeneous catalysts. The authors acknowledge remaining challenges, particularly in evaluating the novelty and stability of generated surface structures and the model's inability to generate truly unseen elements or structural attributes outside its pretraining vocabulary.