Predicting peptide aggregation with protein language model embeddings

The paper introduces PALM, a deep-learning model that leverages transfer learning with protein language model embeddings to effectively predict peptide aggregation from limited data, while demonstrating that accurately assessing the impact of single mutations requires larger training datasets.

The Gist

The Big Problem: The "Bad Clumps"

Imagine your body is a bustling city made of tiny building blocks called proteins. Usually, these blocks are well-behaved and do their jobs. But sometimes, they get confused and start sticking together in messy, sticky piles called amyloid fibrils.

Think of these fibrils like traffic jams or clogged drains. When they form, they can cause serious problems, leading to diseases like Alzheimer's or Type-2 diabetes. They also mess up the manufacturing of life-saving medicines (biologics), causing them to clump up and become useless before they even reach the patient.

Scientists need to predict where and when these proteins will start clumping so they can fix the design before it's too late. But the old way of testing this is like trying to find a needle in a haystack by looking at every single piece of hay one by one. It's slow, expensive, and we don't have enough data.

The Solution: PALM (The "Smart Translator")

The researchers at Novo Nordisk built a new AI tool called PALM (Predicting Aggregation with Language Model embeddings).

To understand how PALM works, imagine that proteins are written in a language. Just like humans use words to communicate, proteins use a sequence of 20 different amino acids (the "letters" of the alphabet) to build themselves.

1. The "Pre-trained" Brain (ESM2):
   Before PALM was even built, a massive AI model called ESM2 was trained on millions of protein sequences from nature. Think of ESM2 as a super-linguist who has read every book in the library. It understands the "grammar" and "context" of proteins. It knows that certain letters usually go together, just like a human knows that "th" usually goes together in English.

2. The "Translator" (PALM):
   PALM takes the "understanding" from the super-linguist (ESM2) and uses it to translate protein sequences into a prediction: "Will this clump?"

   • Instead of just looking at the letters, PALM looks at the meaning behind them.
   • It acts like a detective that can spot the specific "clumping zones" (called Aggregation-Prone Regions or APRs) within a long protein chain.

The Hurdle: The "Short Story" Problem

There was a catch. The data PALM was trained on (called WaltzDB) only contained tiny snippets of proteins—just 6 letters long. It was like trying to teach a student to write a novel by only showing them 6-letter words.

When the researchers tried to use this model on real, long proteins (like a whole novel), the model got confused. The "clumping zones" in the short snippets looked different in the long books because the context was missing.

The Fix: The "Padding" Trick
To fix this, the researchers used a clever trick called padding.

• Imagine you have a short sentence: "CAT."
• To make it look like a longer sentence without changing the meaning of "CAT," you add harmless words around it: "The CAT sat on the mat."
• In the computer model, they added "non-sticky" amino acids to the ends of the short snippets. This tricked the AI into thinking it was looking at a longer protein, helping it learn how to spot clumps in real-world scenarios.

The Results: How Good is it?

The researchers tested PALM against other famous tools (like TANGO and AggreScan).

• The Verdict: PALM is a top-tier player. It performed just as well as, or better than, the best existing tools at predicting if a whole protein will clump up.
• The Superpower: Unlike older tools that just give a "Yes/No" answer, PALM can point to the exact letters in the sequence that are causing the trouble. It's like a doctor who doesn't just say "You're sick," but points to the exact spot on your body that needs attention.

The Weakness: The "Single Letter" Challenge

However, PALM hit a wall with a specific task: Predicting the effect of a single mutation.

• Imagine a protein is a sentence: "THE CAT."
• If you change one letter to "THE BAT," does it start clumping?
• PALM (trained on the small dataset) couldn't tell the difference. It was like a student who knows the story of "The Cat" so well that they can't imagine how changing one word changes the whole plot.

The Fix: More Data!
When the researchers retrained PALM on a massive new dataset (NNK1-3) containing over 100,000 sequences, the model woke up. Suddenly, it could spot that changing a single letter (like the mutations that cause Alzheimer's) would make the protein clump faster.

The Takeaway

This paper shows that AI is getting better at understanding the "language of life."

• By using a pre-trained "linguist" (ESM2) and a smart "translator" (PALM), we can predict dangerous protein clumps much faster and cheaper than before.
• While the model needs more data to spot tiny, single-letter changes, it is already a powerful tool for designing safer drugs and understanding diseases.

In short: They taught a computer to read the "grammar" of proteins so it can predict which ones will turn into sticky, disease-causing messes, saving us time and money in the lab.

Technical Summary

1. Problem Statement

Amyloid fibrils are protein aggregates associated with diseases like Alzheimer's and Type-2 diabetes, and they pose significant challenges in the development of biologic drugs by altering physical properties. While experimental characterization of aggregating peptides is resource-intensive and data is scarce, existing computational methods have limitations:

• Traditional methods (e.g., TANGO, AggreScan) rely on simple physicochemical descriptors or statistical mechanics but lack the ability to learn from new data via machine learning.
• Existing ML models are often trained on small datasets (e.g., WaltzDB-2.0, containing only 1,416 hexapeptides) and struggle to generalize to longer peptide sequences or predict the effects of single amino acid mutations.
• Data scarcity limits the ability to train robust deep learning models capable of identifying aggregation-prone regions (APRs) at single-residue resolution.

2. Methodology

The authors developed PALM (Predicting Aggregation with Language Model embeddings), a deep learning framework designed to overcome data limitations through transfer learning and architectural innovation.

A. Data Strategy & Augmentation

• Training Data: The primary training set is WaltzDB-2.0 (1,416 hexapeptides labeled amyloid/non-amyloid via ThT assays and FTIR).
• Sequence Padding: To bridge the gap between short training hexapeptides and longer evaluation sequences, the authors employed a data augmentation strategy. They appended non-hydrophobic residue padding to the N- and C-termini of the hexapeptides.
  • Rationale: This assumes the 6-residue core is sufficient to drive aggregation and that non-hydrophobic flanking residues do not introduce new APRs.
  • Oversampling: Each sequence was oversampled 10-fold with unique padding to prevent the model from learning spurious patterns between specific padding sequences and the core APRs.
• Evaluation Datasets:
  • Serrano157: 157 peptide sequences (sequence-level labels).
  • AmyPro22: 22 proteins with residue-level APR annotations.
  • NNK4 / NNK1-3: Large-scale datasets (>100k sequences) from massively parallel selection assays.
  • Aβ42 Mutants: 753 single amino acid substitutions in Amyloid-beta, including 13 familial Alzheimer's disease (fAD) mutations.

B. Model Architecture

• Embeddings: PALM utilizes embeddings from the pretrained ESM2 (Evolutionary Scale Modeling) protein language model. The authors tested four model sizes (8M, 35M, 150M, 650M parameters).
• Aggregation Predictor Module (APM): Adapted from the Light Attention architecture, the APM processes the sequence embeddings:
  1. Convolution: Two independent 1D convolutions (kernel size 5) extract local patterns, generating a value tensor and an attention tensor.
  2. Attention Mechanism: Softmax is applied to the attention tensor to create weights.
  3. Feature Fusion: Element-wise multiplication of values and attention weights.
  4. Prediction Head: A Multi-Layer Perceptron (MLP) processes the fused features to output a residue importance score ($r \in [0,1]$).
  5. Aggregation: The final sequence score is computed via a softmax-weighted mean of the residue scores, enhancing interpretability.

C. Training

• Loss Function: Binary cross-entropy on the sequence-level score.
• Optimization: Stochastic Gradient Descent (SGD) with early stopping and checkpointing.
• Hyperparameters: Optimized via grid search (Kernel size=5, LR=0.05, MLP layers=2).

3. Key Contributions

1. Transfer Learning for Aggregation: Demonstrated that transfer learning using pLM embeddings (ESM2) significantly outperforms models trained from scratch or using simple physicochemical descriptors (z-scales/one-hot) on small datasets.
2. Data Augmentation via Padding: Proved that padding short training sequences with non-hydrophobic residues aligns the embedding space of training data with longer evaluation sequences, drastically improving generalization.
3. Residue-Level Prediction without Labels: Showed that the model can identify aggregation-prone regions (APRs) at single-residue resolution despite being trained only on sequence-level binary labels.
4. Scaling Insights: Revealed a counter-intuitive finding where smaller language models (ESM2 8M) outperformed larger models (ESM2 650M) for this specific task, likely because larger models encode evolutionary constraints unrelated to aggregation that act as noise for small datasets.

4. Results

• Benchmark Performance:
  • Serrano157 (Sequence Level): PALM achieved an ROC AUC of 0.918, outperforming TANGO (0.894), AggreProt (0.888), and AggreScan (0.817).
  • AmyPro22 (Residue Level): PALM achieved an ROC AUC of 0.678, comparable to or better than existing methods (e.g., TANGO 0.645, ANuPP 0.649).
• Ablation Studies:
  • Padding: Non-hydrophobic padding ($L_{max}=10$) was critical for performance; removing it caused significant drops.
  • Embeddings: Replacing ESM2 embeddings with one-hot or z-scales caused a large performance decline, confirming the value of pLM representations.
  • Model Size: The 8M parameter ESM2 model yielded the best results; larger models led to overfitting and lower validation scores.
• Mutation Prediction (The "Failure" and Fix):
  • Initial Failure: The base PALM (trained on WaltzDB) failed to identify single mutations in Aβ42 that increase aggregation rates (ROC AUC ~0.51), as the model's scores for the wild-type sequence were already saturated near 1.0.
  • Success with More Data: Retraining the PALM architecture on the larger NNK1-3 dataset (100k+ sequences) significantly improved mutation prediction (ROC AUC ~0.62–0.70).
  • Feature Sensitivity: Interestingly, for the mutation task, using one-hot encodings with the NNK1-3 dataset outperformed ESM2 embeddings, suggesting that for fine-grained mutation effects, the specific features of the large dataset matter more than the general pLM context.

5. Significance

• Therapeutic Development: PALM provides a robust tool for screening peptide libraries for potential amyloid formation, aiding in the design of stable therapeutic peptides.
• Disease Mechanism Insight: The model's ability to identify APRs without explicit residue-level training offers a new approach to understanding the structural basis of aggregation in diseases like Alzheimer's.
• Data Efficiency: The study highlights that while pLMs are powerful, their utility in small-data regimes depends heavily on the model size (smaller is often better) and data augmentation strategies.
• Open Science: The authors released code and model weights (PALM, PALM-NNK, PALM-NNK-OH) to facilitate community adoption and further research into aggregation prediction.

In conclusion, PALM represents a state-of-the-art approach to peptide aggregation prediction, successfully leveraging transfer learning to overcome data scarcity while highlighting the specific data requirements needed to predict the nuanced effects of single-point mutations.