🧬 biology

Rethinking Benchmarks and Models for Enzyme Specificity Prediction

This paper demonstrates that current enzyme specificity prediction models often fail to outperform simple sequence-based baselines on discovery-relevant benchmarks, but shows that adapting structure-aware models like Boltz to learn from full biomolecular complexes can significantly improve enzyme prioritization performance.

Original authors: Elizabeth H. Mahood, Natália Komorníková, Tomáš Pluskal, Pranam Chatterjee

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Elizabeth H. Mahood, Natália Komorníková, Tomáš Pluskal, Pranam Chatterjee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are a detective trying to find the perfect key (an enzyme) that opens a specific lock (a chemical reaction). In the world of biology, there are millions of keys, and many of them look almost identical. Your goal is to pick the one correct key from a pile of thousands of lookalikes.

This paper is a report card on the new "AI detectives" scientists have built to help with this task. The authors found that while these AI models are very good at telling the difference between a key and a screwdriver, they are currently terrible at telling the difference between two very similar keys.

Here is a breakdown of their findings using simple analogies:

1. The Problem: The "Lookalike" Challenge

In the real world of drug discovery or bio-engineering, scientists often face a situation where they have a specific chemical reaction they want to happen, and they need to find which enzyme causes it.

The Old Way: Scientists used to look at the "family tree" of enzymes. If Enzyme A looks 90% like Enzyme B, they assumed they do the same job.
The New Way (AI): Recently, powerful AI models were trained to predict which enzyme does which job. These models were tested on easy puzzles where the keys were very different from each other (like finding a car key vs. a house key). They scored very high on these tests.

The Paper's Claim: The authors asked, "What happens when the puzzle gets hard?" What if the keys are all identical twins? They tested the AI models on these "hard" puzzles and found the models were barely doing better than guessing randomly.

2. The First Test: The "Random Guess" Reality

The researchers took two popular AI models (FusionESP and EZSpecificity) and tested them on enzymes they had never seen before.

The Analogy: Imagine you trained a dog to fetch a red ball. You then tested it with a blue ball it had never seen. The dog didn't know what to do.
The Result: When the AI models were asked to pick the right enzyme from a list of very similar candidates, they performed almost as poorly as if they were just flipping a coin. They couldn't distinguish the "true" enzyme from the "fake" ones.

3. The Big Database: The "CYP" Library

To get a fair test, the authors built their own massive library of data involving Cytochrome P450 (CYP) enzymes.

The Scale: They gathered nearly 3,000 chemical reactions involving 768 different enzymes.
The Test: They created a "ranking challenge." For a specific reaction, the AI had to pick the correct enzyme out of all the CYPs found in that organism.
The Result: Even after retraining the AI models on this new data, most of them still couldn't beat a simple, old-school method called BLAST.
- What is BLAST? Think of BLAST as a "photocopy matcher." It just looks for the enzyme that looks most like the one you already know. It's not "smart" in the AI sense; it just compares shapes. Surprisingly, this simple "photocopy" method was often just as good as the fancy AI.

4. The One Success: The "Structure" Approach

There was one method that actually worked better than the simple photocopy matcher.

The Method: They used a model called Boltz, which is designed to predict how an enzyme and a chemical substrate physically fit together, like a glove fitting a hand.
The Trick: Instead of just looking at the enzyme alone, they looked at the "handshake" between the enzyme and the chemical. They used the AI's internal understanding of this physical fit to make a prediction.
The Result: This approach consistently beat the simple photocopy matcher. It suggests that to find the right key, you need to understand how the key physically turns in the lock, not just what the key looks like on the outside.

5. The Warning: "Cheating" in the Tests

The authors also found a major flaw in how some previous AI models were tested.

The Analogy: Imagine you are taking a math test, but the teacher accidentally gave you the answer key in the study guide. You get a 100%, but you didn't actually learn math.
The Finding: One of the top-performing models (EnzymeCAGE) seemed great, but when the authors checked closely, they realized the model had already "seen" the test questions during its training. Once they removed those "cheated" examples, the model's performance dropped significantly.

The Bottom Line

The paper concludes that while AI has made huge strides in biology, the current "state-of-the-art" models are not yet ready to solve the hardest problems in enzyme discovery. They are great at spotting big differences but struggle when the candidates are very similar.

To move forward, the authors suggest:

Stop using easy tests: We need to test AI on "hard" puzzles where the candidates look alike, not just on easy ones.
Focus on the "Handshake": Models that understand the physical 3D fit between an enzyme and a chemical (like the Boltz approach) seem more promising than those that just look at the enzyme's shape in isolation.
Be careful with data: We must ensure AI models aren't just memorizing the answers from their training data.

In short: The AI detectives are smart, but they are currently getting lost in a crowd of twins. We need to teach them to look at how the twins interact with the world, not just how they look.

Technical Summary: Rethinking Benchmarks and Models for Enzyme Specificity Prediction

Problem Statement
The identification of true enzyme catalysts among many similar candidates is a central bottleneck in enzyme annotation, biosynthetic pathway elucidation, and biocatalyst retrieval. While recent advances in protein machine learning have produced numerous models for predicting enzyme-substrate or enzyme-reaction compatibility, their performance is often evaluated in "low-homology" regimes (dissimilar candidates) or using global discrimination metrics (e.g., AUROC, AUPR). These evaluation strategies fail to address the "high-homology" setting, where the primary challenge is prioritizing a few true positives from a pool of closely related enzymes (e.g., within a single gene family) that share high sequence similarity but possess distinct substrate specificities. Existing benchmarks often suffer from data leakage or lack the resolution to test generalization to unseen enzymes and substrates in practical discovery scenarios.

Methodology
The authors conducted a comprehensive benchmarking study of four state-of-the-art enzyme prediction models: two substrate-prediction models (FusionESP, EZSpecificity) and two reaction-prediction models (CLIPZyme, EnzymeCAGE).

Initial Benchmarking (Unseen Data): The authors first evaluated FusionESP and EZSpecificity on two enzyme screens (Nitrilases and $\alpha$ -ketoglutarate/Fe(II)-dependent enzymes) where all enzymes and substrates were strictly excluded from the models' training data. They utilized stringent ranking metrics, including micro/macro AUROC, AUPR, and $\Delta$ Normalized Enrichment Factor ( $\Delta$ NEF), to measure the ability to rank true substrates highly among negatives.
CYP Dataset Construction: To enable a controlled, leakage-aware comparison across all four models, the authors assembled the largest Cytochrome P450 (CYP) reaction dataset to date. This dataset comprises 2,922 reactions across 768 enzymes and 616 substrates, derived from UniProt and RHEA, filtered for manual annotation and mono-oxygenation reactions.
CYP Ranking Benchmark: A specific evaluation set was constructed containing 45 reactions with enzymes and substrates disjoint (sequence identity <40%, Tanimoto similarity <0.7) from the training data. The task required ranking the correct enzyme among all CYPs native to the organism of origin. The evaluation focused on top-K retrieval (top 10%, 20%, 50%), Enrichment Factor (EF), BEDROC, and Mean Reciprocal Rank (MRR).
Fine-Tuning and Baselines: All four models were retrained (fine-tuned) on the CYP training dataset to assess task-specific performance gains. Performance was compared against random baselines and sequence-based homology search methods (BLAST-sequence and BLAST-substrate).
Boltz Adaptation: The authors adapted Boltz, a bimolecular structure prediction model, for enzyme-substrate discrimination. Instead of relying solely on predicted complex structures, they extracted residue-ligand pair embeddings (interactions between substrate atoms and protein residues within 5Å) from Boltz's internal representations. Supervised classifiers (Random Forest and MLP) were trained on these embeddings to discriminate positive from negative CYP-substrate pairs.

Key Results

Limited Generalization of Existing Models: In the initial assessment on unseen enzyme families, FusionESP and EZSpecificity performed near random baselines when tasked with ranking true substrates. While they showed some ability to distinguish positives from negatives on a per-enzyme basis (macro AUROC > 0.5), they failed to achieve high early enrichment ( $\Delta$ NEF), suggesting they learned local within-enzyme signals rather than robust global interaction rules.
CYP Benchmark Performance: On the CYP ranking dataset, most models (including fine-tuned versions of FusionESP, EZSpecificity, and CLIPZyme) did not consistently outperform sequence-based BLAST baselines.
- EnzymeCAGE initially appeared to outperform BLAST baselines; however, the authors identified significant data overlap (39 of 45 test reactions contained substrates or enzymes present in EnzymeCAGE's original training data). When evaluated on the remaining non-overlapping reactions, its advantage diminished.
- Fine-tuning generally improved performance over random baselines but did not guarantee superiority over BLAST, indicating that global features conserved across enzyme classes (e.g., heme-binding domains) may dominate representations in broad training, masking specific substrate-determining features.
Boltz-Based Approach: The supervised classifiers trained on Boltz's residue-ligand interaction embeddings were the only method to consistently outperform both random and BLAST baselines on the leakage-aware CYP ranking dataset. This approach successfully leveraged the structural and physical constraints "learned" by the co-folding model to discriminate specific enzyme-substrate pairs.

Significance and Claims
The paper argues that current benchmarks for enzyme specificity prediction are often misaligned with practical discovery needs, which require prioritizing true hits from large pools of highly similar candidates. The authors claim that:

Benchmarking Reform: Evaluation should shift from global discrimination metrics (AUROC/AUPR) to ranking-based metrics (BEDROC, EF, MRR) and enforce strict train-test splits that ensure dissimilarity in both enzymes and substrates to prevent data leakage.
Representation Shift: Interaction-aware representations derived from full biomolecular complexes (specifically via co-folding structure prediction models like Boltz) offer a promising basis for enzyme prioritization. These embeddings appear to capture biophysical laws and specific interaction constraints that sequence-only or separate molecule-encoder models miss.
Current Limitations: Despite the success of the Boltz-based approach, the authors remain modest, noting that current models still fall short of the accuracy required for routine experimental screening in high-candidate settings (e.g., top 10% accuracy remains low). They emphasize that while the Boltz-head approach is computationally intensive, it provides a proof-of-concept that structural interaction embeddings can improve enzyme specificity prediction beyond homology-based methods.