Generalise or Memorise? Benchmarking Ligand-Conditioned Protein Generation from Sequence-Only Data

This paper benchmarks ligand-conditioned protein generation using sequence-only data, revealing a fundamental trade-off where models trained on sparse ligand-protein pairs produce foldable but less diverse sequences, while those trained on abundant pairs yield more diverse but less reliably foldable candidates, ultimately highlighting dataset redundancy and incompleteness as critical bottlenecks.

The Gist

The Big Idea: Teaching a Robot to Design Custom Locks

Imagine you have a massive library of keys (small molecules, or ligands) and a massive library of locks (proteins). In nature, specific keys fit into specific locks to open doors (start chemical reactions, send signals, etc.).

Scientists want to design new locks that fit custom-made keys that humans invent. This is incredibly hard. Usually, to design a lock, you need a 3D blueprint of the key and the lock, which is expensive and slow to get.

This paper asks a bold question: Can we teach a computer to look at the text description of a key (its chemical formula) and instantly write the text description of a new lock (the protein sequence) that fits it, without needing a 3D blueprint?

They tried to train an AI (a "Protein Language Model") to do this, treating it like a translation task: Translate "Key Text" $\rightarrow$ "Lock Text."

---

The Experiment: Two Different Classrooms

To test this, the researchers built two different "classrooms" (datasets) to teach the AI, representing two extremes of how much information is available.

1. The "One-to-Many" Classroom (The Substrate Dataset)

• The Setup: Imagine a teacher showing the AI one specific key and saying, "Here are 3,600 different locks that all fit this key."
• The Result: The AI gets confused. It realizes there are many ways to solve this problem. It starts generating very diverse locks.
• The Catch: Because it's trying to be so creative, many of the locks it designs are broken. They look like locks, but they won't actually fold up into a 3D shape. They are "unstable."
• The Metaphor: It's like asking a chef to cook a dish for 3,600 different people. They try to make 3,600 different versions, but half of them are burnt or taste terrible because they tried too hard to be unique.

2. The "One-to-Few" Classroom (The Binder Dataset)

• The Setup: Here, the teacher shows the AI a key and says, "Only 2 or 3 specific locks fit this key."
• The Result: The AI realizes the answer is very specific. It stops trying to be creative and starts memorizing. It looks at the training examples and says, "I know this one! I'll just copy it or make a tiny variation."
• The Catch: The locks it makes are very stable (they fold correctly), but they aren't very new. They are just copies of things the AI has already seen.
• The Metaphor: It's like a student taking a test who only memorizes the answers to the practice questions. If the test question is slightly different, they might fail, but if it's the same, they get an A.

The Big Trade-Off: Creativity vs. Reliability

The paper discovered a fundamental rule: You can't have both high creativity and high stability at the same time with current data.

• If you give the AI too many examples per key: It gets creative but makes broken locks.
• If you give the AI few examples: It makes perfect, stable locks, but they are just copies of old ones (Memorization).

Did the AI Actually "Understand" Chemistry?

The researchers wanted to know: Is the AI just copying, or is it actually learning the rules of chemistry?

• The "Caffeine" Test: They asked the AI to design a lock for caffeine. The AI had never seen a protein that binds to caffeine in its training data.
• The Result: The AI generated a sequence that looked nothing like the training data. When they tested it with a super-computer simulation (called Boltz2), it predicted that this new protein would actually bind to caffeine.
• The Takeaway: Even though the AI mostly "memorizes," it can sometimes "generalize." It figured out the underlying logic of how to make a lock for a key it had never seen before.

The Bottleneck: The Library is Incomplete

The biggest problem the paper highlights is data quality.

• For most custom keys we want to design, we don't have a list of existing locks that fit them.
• Because the "library" of known pairs is so empty, the AI is forced to guess or memorize. It's like trying to teach a language translator who only has a dictionary with 10 words. It can't learn the grammar; it can only guess based on the few words it knows.

Conclusion: What Does This Mean for the Future?

1. Current State: We can use these AI models to generate candidates for new drugs very quickly.
2. The Process: The AI spits out 25 different protein designs. Most are just copies of old ones, but a few might be new.
3. The Filter: We can't trust the AI 100%. We have to run these designs through a "simulator" (like a video game physics engine) to see if they actually fold and bind.
4. The Future: To get truly new, stable designs, we need better data. We need to fill in the gaps in our library so the AI doesn't have to guess.

In a nutshell: The AI is a talented but confused apprentice. If you give it too many choices, it makes a mess. If you give it too few, it just copies the master. But sometimes, when pushed, it surprises us by inventing something truly new.

Technical Summary

1. Problem Statement

Designing proteins that bind to specific, user-defined small molecules (ligands) is a critical challenge in synthetic biology and drug discovery. Traditional methods rely heavily on structural information and costly experimental iteration. While Protein Language Models (pLMs) have shown success in unconditional generation and conditioning on coarse functional labels (e.g., enzyme classes), instance-level conditioning on specific ligand molecules using purely textual inputs remains unexplored.

The core question addressed is: Can sequence-to-sequence translation models, trained on ligand-protein pairs, generalize to design novel binders for unseen ligands, or do they merely memorize training data?

2. Methodology

Data Curation

The authors curated large-scale datasets (>17 million ligand-protein pairs) representing different data regimes:

• Binder-Dataset: Aggregated from BindingDB, DTC, BioLiP, and AlphaFill. Contains ~1.8M unique ligands and ~774k unique proteins. It follows a long-tailed distribution where a few "promiscuous" ligands bind thousands of proteins, while most bind few.
• Substrate-Dataset: Derived from enzyme databases (Rhea, BRENDA, etc.), mapping reactants to enzymes. This dataset has far fewer unique ligands (4k) but maps thousands of proteins to each ligand (3600 proteins/ligand), creating a high-entropy conditional distribution.
• SAIR (Test Set): A structurally augmented repository used for rigorous testing, containing ~735k unique ligands and ~1M pairs, curated to ensure chemical validity and structural context.

Model Architecture & Training

• Task Formulation: Framed as a sequence-to-sequence translation problem where the input is a ligand string (SMILES or SELFIES) and the output is a protein amino acid sequence.
• Architecture: Primarily used an Encoder-Decoder (T5) architecture. Experiments also included Decoder-only (GPT-2) and pre-trained Llama3-based decoders.
• Sampling Strategy: To address data imbalance (where common ligands dominate training), a Unique-Ligand Sampling strategy was employed. In this approach, each unique ligand is sampled once per epoch, and a random target protein is selected from its binding list. This prevents the model from overfitting to highly annotated ligands.
• Training Stages: A two-stage training process was used for pre-trained decoders: Stage 1 trained the encoder and cross-attention layers (freezing the decoder), followed by Stage 2 training all parameters.

Evaluation Metrics

• Structural Reliability: Predicted folded structures using ESMFold and evaluated via pLDDT scores (higher scores indicate better foldability).
• Novelty & Retrieval: Measured sequence similarity to the training set using MMSeqs2 (Train ID). High similarity suggests memorization/retrieval; low similarity suggests generalization.
• Generalization (Ground Truth): Calculated GT Accuracy by checking if generated sequences match known binders (or close homologs) for the test ligand.
• Ligand Novelty: Assessed if the model could generate a binder for a test ligand that is chemically dissimilar to any ligand seen in the training set for that protein family.

3. Key Results

The Generalization vs. Memorization Trade-off

The study reveals a fundamental trade-off driven by supervision ambiguity (the number of proteins per ligand):

• Low Annotation Density (Few proteins per ligand): Models exhibit retrieval-like behavior. They generate sequences highly similar to training data (high Train ID) but with high foldability (high pLDDT). They effectively "memorize" or retrieve near-neighbors.
• High Annotation Density (Many proteins per ligand): Models are forced to learn a richer, higher-entropy distribution. This leads to more diverse generations but significantly lower foldability (lower pLDDT), as the model struggles to converge on a single stable fold.

Performance Metrics

• Best Model: The T5-Base model (~200M parameters) trained on the Binder-Dataset with Unique-Ligand sampling achieved the best balance.
  • pLDDT: ~79.07 (SAIR test set), indicating high structural confidence.
  • GT Accuracy: ~38.8% on the SAIR test set. Performance was highest (90.89%) in the moderately annotated regime ([10, 49] proteins per ligand) and dropped for highly promiscuous ligands.
• Sampling Impact: Unique-Ligand sampling significantly outperformed standard Pair Sampling, particularly in sparsely supervised regimes, improving both structural confidence and GT retrieval.
• Architecture: Encoder-Decoder architectures outperformed Decoder-only models for this conditioning task. Pre-training on large protein corpora (Llama3) did not improve performance over training from scratch, suggesting the specific conditioning task required learning a new distribution that the pre-trained weights hindered.

Evidence of Generalization

Despite the strong retrieval behavior, the study found evidence of true generalization:

• Ligand Novelty: The model successfully generated binders for test ligands that were chemically dissimilar to any ligand associated with the retrieved protein family in the training set.
• Protein Novelty: In rare cases, the model generated sequences that matched the Ground Truth binder more closely than any training sequence.
• Case Study: The model generated a sequence for caffeine (a molecule not seen in training) that, when co-folded with Boltz2, showed high binding confidence (pLDDT 0.97), despite being 10% divergent from the closest known annotation.

4. Key Contributions

1. First Benchmark: Established the first comprehensive benchmark for ligand-conditioned protein generation using purely sequence-based inputs (SMILES/SELFIES to Protein).
2. Dataset Curation: Released curated datasets (>17M pairs) and evaluation tools to support future research in this domain.
3. Insight into Data Regimes: Demonstrated that the success of sequence-only binder design is heavily bottlenecked by dataset redundancy and incompleteness. The "Generalize vs. Memorize" behavior is dictated by the ratio of proteins to ligands in the training data.
4. Practical Pipeline: Proposed a workflow where pLMs generate multiple candidate binders rapidly, which are then filtered via downstream tools (co-folding, docking) to select viable candidates for experimental validation.

5. Significance and Future Outlook

This work highlights that while current sequence-only pLMs are powerful tools for retrieving known binders and discovering novel uses for known proteins, they currently struggle to design de-novo proteins for unseen ligands without structural guidance.

• Limitations: The field is limited by the sparsity of high-quality ligand-protein annotations. Highly promiscuous ligands create multi-modal distributions that are difficult for current models to converge on.
• Future Directions: To achieve true de-novo design, the authors suggest:
  • Scaling text-based datasets (though expensive).
  • Incorporating richer annotations (e.g., binding affinity values).
  • Moving toward multi-modal approaches that integrate structural data.
  • Using physics-based inductive biases to guide generation.

In conclusion, the paper provides a rigorous foundation for understanding the capabilities and limitations of "brute-force" conditional language modeling in protein design, positioning it as a valuable first-pass filter for drug discovery pipelines rather than a standalone solution for de-novo design.