Protein Language Models Outperform BLAST for Evolutionarily Distant Enzymes: A Systematic Benchmark of EC Number Prediction

This study systematically benchmarks protein language models against BLAST for enzyme commission number prediction, demonstrating that while simple MLP classifiers match BLAST's performance on in-distribution proteins, they significantly outperform it for evolutionarily distant organisms, establishing that smaller PLMs with lightweight classifiers are both efficient and superior for remote homology detection.

The Gist

Imagine you have a massive library of protein "recipes." Each recipe tells a cell how to build a specific machine (an enzyme) that does a specific job, like breaking down sugar or building DNA. Scientists use a special filing system called EC Numbers to label these jobs, kind of like how a library uses the Dewey Decimal System to organize books.

For decades, if a scientist found a new, unknown recipe, they would try to find a similar, already-labeled recipe in their library and copy the label. This method is called BLAST. It works great if the new recipe is very similar to an old one. But if the new recipe is from a weird, distant organism (like a microscopic parasite) and looks nothing like anything in the library, BLAST gets confused and often guesses wrong.

Recently, a new technology called Protein Language Models (PLMs) has arrived. Think of these not as simple search engines, but as super-intelligent chefs who have read millions of recipes. Instead of just looking for word-for-word matches, these chefs understand the grammar and flavor of proteins. They can look at a strange new recipe and guess the job based on the "vibe" of the ingredients, even if they've never seen that exact dish before.

This paper is a massive cooking competition to see if these super-intelligent chefs (PLMs) are actually better than the old search engine (BLAST) at labeling these recipes.

The Big Experiment

The researchers set up a rigorous test with 1,296 different "chefs" (combinations of models and classifiers) to see who wins. They tested them in two main scenarios:

1. The "Same Neighborhood" Test: They gave the chefs recipes that were very similar to ones they had already studied.

   • Result: It was a tie! The new chefs (PLMs) were just as good as the old search engine (BLAST), but they were faster and didn't need a giant library of reference books to work.

2. The "Foreign Country" Test: They gave the chefs recipes from organisms that are very different from anything in the training data (like distant parasites or weird bacteria).

   • Result: The new chefs crushed it. While the old search engine (BLAST) got lost and failed to find matches, the PLMs correctly identified the jobs with high accuracy. In some cases, the PLMs were 30% more accurate than the old method.

Key Takeaways (The "Secret Sauce")

• Keep it Simple: The researchers tried fancy, complex kitchen tools (like deep neural networks with many layers). Surprisingly, the winners were the simplest tools: a basic two-layer "Multilayer Perceptron" (MLP).

  • Analogy: It's like realizing you don't need a $10,000 robot chef to make a great sandwich; a simple, sharp knife and a good chef's intuition (the PLM embedding) are enough. The complex tools actually made things worse because they were overthinking it.

• Size Matters (But Not Too Much): They tested a "small" AI model (ESM2-650M) and a "huge" one (ESM2-3B). The huge model was slightly better, but only by a tiny fraction.

  • Analogy: The huge model is like a Ferrari; the smaller one is a reliable Toyota. The Ferrari is 5% faster, but the Toyota gets you to the destination just as well and costs way less to run. The authors recommend the "Toyota" (the smaller model) for most people.

• The "Leakage" Problem: The paper points out that many previous studies cheated by accidentally letting the test recipes be too similar to the training recipes.

  • Analogy: Imagine a student taking a test where the questions are identical to the homework they just did. They'd get 100%, but that doesn't mean they learned the subject. The researchers made sure the "test students" (the new proteins) had never seen the "homework" (training data) before, ensuring a fair test.

Why This Matters

This study proves that AI is ready to take over the job of labeling enzymes, especially for the weird, unknown, and distant organisms that traditional methods struggle with.

• For Science: It means we can now accurately map out the metabolic pathways of organisms we've never studied before, which is huge for drug discovery and synthetic biology.
• For You: It means we are getting closer to a future where computers can help us design new medicines or biofuels by understanding the "language" of life, even for species we've never met.

In short: The old way of finding enzyme jobs (looking for a twin) is being replaced by a smarter way (understanding the language). And the best tool for the job isn't the most expensive or complex one; it's the one that strikes the perfect balance between smarts and simplicity.

Technical Summary

1. Problem Statement

Accurate prediction of Enzyme Commission (EC) numbers is critical for genome annotation, metabolic reconstruction, and enzyme engineering. However, current computational methods face significant challenges:

• Data Imbalance: The number of sequenced genomes (hundreds of millions) vastly outpaces experimentally characterized proteins (~570k in SwissProt).
• Limitations of Similarity-Based Methods: Traditional tools like BLASTp rely on sequence homology. Their accuracy degrades sharply when query sequences diverge beyond ~30–40% identity, precisely the regime where novel enzyme discovery is most needed.
• Benchmarking Flaws: Existing studies often use random protein-level train/test splits, leading to "sequence leakage" where test proteins share high identity with training data. This inflates performance metrics and fails to measure true generalization capability.
• Lack of Systematic Evaluation: While Protein Language Models (PLMs) have shown promise, no study has systematically evaluated them across all four EC hierarchy levels, multiple PLM architectures, diverse downstream classifiers, and strict sequence identity thresholds simultaneously.

2. Methodology

The authors conducted a comprehensive benchmark involving 1,296 trained models across 3,888 experimental conditions.

• Dataset: 90,577 reviewed enzymes from UniProt/SwissProt (2023 release) with complete four-level EC annotations.
• Rigorous Train/Test Splitting:
  • Instead of random splits, the authors used cluster-based splitting via MMseqs2.
  • Splits were performed at four sequence identity thresholds: 30%, 50%, 70%, and 90%.
  • This ensures that no test protein shares more than the specified identity with any training protein, eliminating sequence leakage.
• Protein Language Models (PLMs): Three models were evaluated:
  • ESM2-650M: 650M parameters, 1,280-dim embeddings.
  • ESM2-3B: 3B parameters, 2,560-dim embeddings.
  • ProtT5-XL: 3B parameters, 1,024-dim (T5 encoder-decoder).
  • Note: Mean-pooled residue embeddings were used as fixed inputs.
• Downstream Architectures: Nine neural network classifiers were tested, ranging from simple to complex:
  • MLP variants: Standard, Deep, Wide, and Attention-based MLPs.
  • Complex variants: CNNs, ResNets, Multi-Head Attention, Hybrid CNN-Transformer, and Transformer Encoders.
• Baselines:
  • Fair BLAST: BLASTp against the specific training split (90K sequences) to ensure a like-for-like comparison.
  • Practitioner BLAST: BLASTp against the full UniProt/SwissProt database (~520K sequences).
• Generalization Validation: Performance was tested on 9 held-out prokaryotic proteomes and 13 phylogenetically distant eukaryotes (e.g., Giardia lamblia, Trichomonas vaginalis) to assess cross-organism generalization.

3. Key Contributions

1. Systematic Benchmarking: The first study to evaluate PLMs for EC prediction across all hierarchy levels, multiple PLMs, and multiple architectures under strict sequence identity constraints.
2. Methodological Correction: Demonstrated that cluster-based splitting is essential for honest benchmarking, revealing that random splits artificially inflate BLAST performance.
3. Architecture Discovery: Identified that simple Multi-Layer Perceptrons (MLPs) are sufficient and superior to complex architectures (CNNs, ResNets, Transformers) when using PLM embeddings.
4. PLM vs. BLAST Comparison: Quantified the exact regimes where PLMs outperform traditional homology search, specifically for evolutionarily distant organisms.
5. Open Resources: Released all code, models, and benchmark results to facilitate future research.

4. Key Results

A. Overall Performance (In-Distribution)

At the 50% sequence identity threshold, PLM-based classifiers achieved state-of-the-art accuracy:

• EC1: 98.0%
• EC2: 96.9%
• EC3: 96.6%
• EC4: 97.0%
• Architecture: Simple MLP classifiers (specifically 2-layer or Wide MLP) achieved these results, matching or marginally exceeding (±0.7 pp) the fair BLAST baseline. Complex architectures like Transformers showed instability and lower performance.

B. PLM vs. BLAST Comparison

• In-Distribution (50–90% identity): PLMs and BLAST perform similarly (within ±0.7 percentage points). However, PLMs offer 100% prediction coverage, whereas BLAST fails to find hits for ~9–10% of sequences.
• Evolutionarily Distant (Cross-Organism): PLMs dramatically outperform BLAST:
  • Prokaryotes: Mean gain of +16.9 pp at EC4 over BLAST across 9 held-out proteomes.
  • Eukaryotes: Gains reached +31.8 pp for Giardia lamblia (vs. 90K BLAST) and +26.4 pp for Trichomonas vaginalis (vs. 520K BLAST).
  • Failure Cases: BLAST retained an advantage only for specific apicomplexan parasites where eukaryote-specific enzyme families were underrepresented in the training data.

C. Model Efficiency and Selection

• PLM Size: While ESM2-3B performed slightly better statistically than ESM2-650M, the difference was negligible (0.35 pp). ESM2-650M is recommended as the default due to its superior accuracy-efficiency trade-off.
• Architecture: Transformer re-encoding of PLM embeddings failed due to convergence instability at standard learning rates. Simple MLPs were consistently superior.

5. Significance and Implications

• Paradigm Shift: The study confirms that PLM embeddings encode sufficient functional information that complex downstream processing (like CNNs or re-encoding) is unnecessary; a simple linear classifier on pooled embeddings is optimal.
• Novel Enzyme Discovery: PLMs provide a robust solution for annotating enzymes in understudied taxa and metagenomic datasets where homology-based methods fail.
• Benchmarking Standards: The paper establishes a new standard for evaluating bioinformatics tools, emphasizing that random splits are inadequate for assessing generalization in protein function prediction.
• Practical Recommendation: The authors recommend ESM2-650M + MLP as the default configuration for EC number prediction, offering the best balance of accuracy, computational efficiency, and generalization to novel enzyme families.

Limitations Noted: The study used frozen embeddings (no fine-tuning), treated EC prediction as single-label (ignoring multi-function enzymes), and relied on a training set dominated by prokaryotes, which slightly limited performance on specific eukaryotic families. Future work should explore fine-tuning and multi-label formulations.