🤖 machine learning

LC-SEPLM: long-range contact-supervised adaptation for sequence-only protein representation learning

LC-SEPLM enhances the ESM2 protein language model by integrating long-range residue-pair contact supervision via LoRA and cross-attention, significantly improving performance on downstream tasks like remote-homology recognition while maintaining sequence-only inference capabilities.

Original authors: Chen Wang, Boming Kang, Qinghua Cui

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chen Wang, Boming Kang, Qinghua Cui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to understand the secret language of life. For years, scientists have been training "protein language models" on massive libraries of amino acid sequences—the chemical letters that spell out every protein in your body. Think of these models like super-smart readers who have memorized billions of books. They are incredibly good at guessing the next word in a sentence, which helps them understand how proteins evolve and function. However, there's a catch: these models are reading the story line-by-line, from start to finish. They don't naturally "see" that the first word of a sentence might be physically touching the last word because the whole paragraph has folded up into a tight, 3D ball. In the real world, proteins fold into complex shapes, and distant parts of the chain often bump into each other to hold the structure together. The big question in this corner of science is: Can we teach these text-reading computers to understand the 3D shape of the protein without actually showing them a picture of the shape? If we could, we might unlock better ways to design new medicines or understand diseases, all while keeping the system simple enough to run on just the text.

Enter LC-SEPLM, a new method that acts like a clever tutor for these protein-reading computers. The researchers took a powerful existing model (called ESM2) and gave it a special training exercise. Instead of just asking the model to guess the next letter in the sequence, they added a "contact quiz." They showed the model a protein's sequence and asked, "If this protein folds up, will the amino acid at position 10 touch the one at position 500?" To answer this, the model had to look at the entire sequence at once, learning to spot the hidden long-distance friendships between amino acids that only exist when the protein is folded. The cool part is that the model learned this using predictions from a super-accurate structure-predicting AI (AlphaFold) as a cheat sheet during training, but once the training was done, the "cheat sheet" was thrown away. The final model is still a "sequence-only" reader; it doesn't need 3D coordinates to work, it just carries the knowledge of how shapes form in its brain.

The results of this experiment were quite promising, though not a magic bullet for every problem. When the researchers tested their new model on eight different protein-related tasks, it beat the original model in every single one. The biggest victory came in remote-homology recognition, a task where the model has to find proteins that are distant cousins but share a similar shape. Here, the model's score jumped from 0.6122 to 0.6769, a significant leap of 0.0647 (or 6.47 percentage points). This suggests that teaching the model to look for long-range contacts really helped it understand the "big picture" of how proteins are built.

However, the paper is careful to point out where this new trick didn't work. When the model was tested on tasks that require looking at tiny, local details—like predicting how a single mutation changes a protein's function (ProteinGym) or finding specific spots where chemical tags attach (PTM sites)—it actually performed slightly worse or stayed the same. This tells us that while the model got much better at understanding the global 3D structure, it didn't necessarily get better at the fine-grained, local chemistry. It's like a student who became a master at understanding the plot of a whole novel but didn't improve their ability to analyze a single sentence's grammar.

The researchers also compared their model to another famous structure-aware model called ESM-S. On some specific enzyme classification tasks, LC-SEPLM did much better, with gains as high as 0.1771. But on other tasks, like classifying broad protein families, it didn't quite win. This suggests that while adding structural supervision is a powerful tool, it's not a universal fix that makes a model perfect at everything. The study concludes that this "contact-supervised adaptation" is a practical and bounded strategy: it's a great way to enrich a protein model's understanding of global shape without needing complex 3D data during real-world use, but it has clear limits when it comes to local, residue-level predictions.

Technical Summary: LC-SEPLM

Problem Statement

Protein language models (pLMs), such as ESM2, have demonstrated success in learning transferable representations from amino-acid sequences by modeling contextual dependencies via masked language modeling (MLM). However, these models primarily learn residue identity based on sequence context rather than explicit three-dimensional structural relationships. A critical limitation is that standard MLM objectives do not explicitly constrain the model to identify long-range residue contacts—pairs of residues that are distant in the primary sequence but spatially proximal after folding. While evolutionary constraints in sequence data indirectly encode some structural information, predicting masked residues does not force the encoder to identify specific spatially adjacent residue pairs.

Existing approaches to incorporate structural information often require structural coordinates, predicted structures, or structure-derived tokens (e.g., 3Di states) as inputs during downstream inference. This dependency limits their applicability in scenarios where only sequence data is available. The central challenge addressed by this work is how to introduce direct structural supervision during the adaptation of a sequence-only model without requiring structural inputs during downstream deployment.

Methodology

The authors introduce LC-SEPLM (Long-range Contact-supervised ESM Protein Language Model), a framework that adapts the pre-trained ESM2 model using Low-Rank Adaptation (LoRA) and long-range residue-pair contact supervision.

Core Architecture and Adaptation:

Base Model: The approach adapts ESM2-t30-150M-UR50D.
LoRA Integration: LoRA modules are inserted into the attention projection matrices ( $q\_proj$ and $v\_proj$ ) of the last eight Transformer layers (Tail-8). The original ESM2 weights remain frozen, while low-rank matrices ( $A$ and $B$ ) are trained.
Contact Supervision Branch: A training-only branch is added to predict long-range contacts. This branch utilizes AlphaFold-predicted structures from Swiss-Prot proteins to generate ground-truth labels.
- Label Definition: A positive contact is defined as a residue pair $(i, j)$ where the sequence separation $|i-j| \ge 12$ and the C $\alpha$ -to-C $\alpha$ distance $d_{ij} \le 8$ Å.
- Pair-Query Mechanism: For each sampled residue pair, the model constructs a pair-specific query. This query concatenates the residue states, their absolute difference, element-wise product, and normalized sequence distance.
- Cross-Attention: The pair-specific query attends to the complete residue sequence via cross-attention. This allows the contact prediction to incorporate global sequence context associated with the specific pair, rather than relying solely on local features.
- Loss Function: The total loss combines the binary cross-entropy loss for contact prediction ( $L_{contact}$ ) and the auxiliary masked language modeling loss ( $L_{MLM}$ ), weighted by $\lambda_{MLM} = 0.6$ .

Inference:
Crucially, the contact supervision branch (including the pair-query, cross-attention, and contact head) is removed after adaptation. Downstream inference relies exclusively on the adapted ESM2 encoder and the amino-acid sequence, producing a 640-dimensional protein embedding via mean pooling. No structural coordinates, contact maps, or structure-derived tokens are required at inference time.

Training Data:
The model was trained on 500,000 AlphaFold-predicted Swiss-Prot proteins. Sequences were truncated to 512 residues, and candidate pairs were sampled with an approximate 2:1 negative-to-positive ratio to manage computational complexity.

Key Contributions

Novel Adaptation Strategy: LC-SEPLM proposes a method to inject structural knowledge into a sequence-only pLM via long-range contact supervision while maintaining a sequence-only inference pipeline.
Pair-Specific Global Context: The introduction of a pair-query mechanism that uses cross-attention over the full sequence allows the model to learn global structural context relevant to specific residue pairs, distinguishing it from methods that compress pairwise relationships into independent positional states.
Bounded Structural Supervision: The work demonstrates that structural supervision can be effectively bounded to the training phase, avoiding the need for structural inputs during downstream application.

Results

The authors evaluated the frozen adapted encoder on eight protein-level tasks and two residue-level diagnostics, comparing performance against the base ESM2 and the structure-informed ESM-S model.

Performance on Protein-Level Tasks:

Universal Improvement: LC-SEPLM improved performance on all eight protein-level tasks relative to the base ESM2 model.
Remote Homology: The most significant gain was observed in remote-homology recognition (Pfam-clan), where the macro-F1 score increased from 0.6122 to 0.6769 (+0.0647).
Other Tasks: Modest but consistent gains were observed for Gene Ontology Molecular Function (GO-MF), Pfam family classification, and subcellular localization.
Independent Benchmark: On the official ESM-S benchmark, LC-SEPLM outperformed ESM-S (150M) on Enzyme Commission (EC) prediction tasks, with F1-max gains up to 0.1771. However, it underperformed ESM-S on certain fold classification and GO-MF splits, indicating that the benefits are not universal across all structural tasks.

Performance on Local Diagnostics:

No Improvement: The method did not improve performance on ProteinGym mutation-effect prediction (Spearman correlation decreased) or UniProt post-translational modification (PTM) site prediction.
Interpretation: This suggests that long-range contact supervision strengthens global protein representations (fold-level) more effectively than local residue-level signals.

Checkpoint Selection:
The authors noted that the checkpoint maximizing contact prediction accuracy (Epoch 12) differed from the checkpoint yielding the best downstream transfer performance (Epoch 10), highlighting the necessity of downstream validation for model selection.

Significance and Claims

The paper positions LC-SEPLM as a practical but bounded strategy for enriching sequence representations with structural information.

Scope of Benefit: The authors claim that the method is most effective for downstream tasks dependent on global structural context, such as remote homology recognition and enzyme function prediction.
Limitations: The work explicitly acknowledges that the approach does not universally improve all prediction tasks. It is less compelling for tasks dominated by local biochemical effects (e.g., mutation effects or PTM sites).
Trade-offs: The use of AlphaFold-predicted structures for supervision is a strength for scalability but a potential limitation, as it may transfer prediction biases from the structure source.
Conclusion: LC-SEPLM demonstrates that residue-pair contact supervision provides a viable route to introduce structural constraints into sequence-only models, provided the downstream application aligns with the global nature of the learned representations. The method avoids the complexity of structure-aware inference while capturing key aspects of protein folding relevant to evolutionary and functional classification.