💻 bioinformatics

TEDlm: domain-centric protein language models with optional structural pre-training

The paper introduces TEDlm, a domain-centric protein language model pretrained on structurally defined domain segments that outperforms larger full-sequence models in remote-homology detection and molecular function prediction, demonstrating that focusing on domain-intrinsic signals yields compact, structurally informed representations.

Original authors: Wei, T., Kandathil, S. M., Buchan, D. W. A., Jones, D. T.

Published 2026-07-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wei, T., Kandathil, S. M., Buchan, D. W. A., Jones, D. T.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine proteins as giant, intricate Lego castles. For a long time, scientists trying to understand these castles by reading their instruction manuals (the amino acid sequences) were looking at the entire book at once. The problem? These books are messy. They contain the instructions for the castle, but they also include the instructions for the bridge, the garden, and the random scribbles in the margins (linkers and disordered regions) that hold everything together. When you try to learn the shape of a specific tower by reading the whole messy book, the signal gets drowned out by the noise.

Enter TEDlm, a new team of AI detectives who decided to stop reading the whole book and start reading just the individual, perfectly formed Lego towers (called "domains").

The Big Idea: Smaller Books, Bigger Brains

The researchers, led by David T. Jones and Tiejun Wei, built a new kind of AI called TEDlm. Instead of training on full-length protein sequences, they fed it millions of these clean, isolated "domain" segments from a massive library called The Encyclopedia of Domains (TED).

Think of it like this: If you want to learn how to bake a perfect chocolate cake, you wouldn't read a cookbook that includes recipes for soup, car manuals, and the history of flour. You'd just read the cake chapters. That's what TEDlm did. It learned the "cake recipes" (the specific shapes and folds of protein domains) without the distraction of the "soup recipes" (the messy parts of the full protein).

The Results: Small is Mighty

Here is the surprising twist: Size isn't everything.

Usually, in the world of AI, bigger is better. A giant brain with 3 billion neurons (like the famous ESM2 3B model) is expected to crush a smaller brain with only 650 million neurons. But in this race, the smaller, domain-focused brain won.

The TEDlm 650M model (the small, focused one) scored an AUROC1 of 0.28 on a tough test called CATH S40, which measures how well a model can spot distant relatives among proteins.
The giant ESM2 3B model (the big, unfocused one) only scored 0.22.

The paper suggests that by training on cleaner data, the smaller model learned the "fold" of the protein much better than the giant model that was distracted by the full-length noise. It's as if the small detective, focusing only on the clues that matter, solved the mystery faster than the giant detective who was trying to read every single page of the case file.

The Superpower: Seeing the Invisible Structure

The team also created a super-charged version called TEDlm3D. This model didn't just read the Lego instructions; it was also given a secret cheat sheet showing how the bricks should touch each other in 3D space (specifically, the distance between the "alpha carbons," or the core bricks of the structure).

This extra training was a game-changer.

TEDlm3D reached an AUROC1 of 0.50.
This is incredibly close to Foldseek, a tool that uses actual 3D structures to find matches, which scored 0.53.

The amazing part? TEDlm3D only needs the text (the sequence) to work. It doesn't need the 3D structure at the moment of testing. It learned the 3D rules during its training and now "sees" the shape just by reading the letters. The paper shows that this structural signal isn't just stored in a separate "output head" (like a calculator attached to the brain); it's woven right into the brain's own thinking process.

What About Other Tasks?

The researchers tested these models on other jobs, like guessing what a protein does (its molecular function) or how well it binds to metals.

Molecular Function: The domain-focused models (TEDlm) were great at this, beating the big ESM2 models. This makes sense because a protein's main job is usually done by a single Lego tower (domain).
Biological Process & Location: Here, the big ESM2 models held their ground. Why? Because knowing where a protein lives in a cell or how it helps a whole organism often requires seeing the entire castle, not just one tower. The domain-only models missed the big picture context.

The "Middle Layer" Surprise

One of the coolest discoveries was about where the AI stores its knowledge. In many AI models, the final layer is the "smartest." But here, the paper found that the middle layers of the TEDlm models were actually the best at spotting structural relationships. The final layers seemed to get a bit too focused on just predicting the next letter in the sequence, forgetting the big structural picture. It's like a student who, after studying for a test, gets so focused on memorizing the last sentence of the chapter that they forget the main plot of the story.

What the Paper Rules Out

The authors are careful to say what they didn't do. They didn't use multiple sequence alignments (looking at evolutionary family trees) to train these models; they relied purely on the sequence and the structural data they fed in. They also note that while their models are great at finding structural relatives, they aren't perfect at predicting things that depend on the whole protein's context, like thermal stability (how well it survives heat), because heat resistance often comes from how different towers interact with each other, not just the towers themselves.

The Bottom Line

This paper suggests that we don't necessarily need to build bigger, more expensive AI brains to understand proteins. Instead, we might just need to feed them cleaner, more focused data. By teaching AI to look at proteins one "domain" at a time, the researchers created compact, smart models that can predict protein shapes and functions almost as well as tools that require actual 3D blueprints. It's a reminder that sometimes, to see the whole picture, you have to zoom in on the details.

Technical Summary: TEDlm and TEDlm3D

Problem Statement
Conventional protein language models (pLMs), such as the ESM family, are typically pretrained on full-length protein sequences. These sequences often interleave multiple functional domains with disordered regions and linkers. The authors argue that training on such full-length sequences dilutes fold-specific signals, as the model must learn patterns across longer, structurally heterogeneous contexts. Furthermore, while recent bimodal models (e.g., ProstT5, SaProt) incorporate explicit structural tokens, they rely on ground-truth structures during training or require structural inputs at inference, limiting their applicability to sequence-only scenarios. The challenge remains to develop compact, structurally informed pLMs that can detect remote homology and predict function without requiring Multiple Sequence Alignments (MSAs) or explicit structural inputs at inference.

Methodology
The paper introduces TEDlm and TEDlm3D, transformer-based protein language models pretrained on structurally defined domain segments rather than full-length proteins.

Data Source: The models utilize The Encyclopedia of Domains (TED), a dataset derived from the AlphaFold Protein Structure Database (AFDB) containing over 365 million structurally defined domains. The authors curated specific subsets (TED50 and TED90) by clustering these domains to manage redundancy.
TEDlm Architecture: This model adopts a standard BERT-like Masked Language Modeling (MLM) objective. It is pretrained exclusively on the amino acid sequences of individual domains, aiming to capture motifs and long-range dependencies inherent to specific folds and superfamilies without the noise of inter-domain linkers.
TEDlm3D Architecture: This variant extends the TEDlm framework by adding a structural auxiliary loss. During pretraining, in addition to the MLM objective, the model is optimized to predict C $\alpha$ distances between residues. This contact-guided loss supervises the attention maps, encouraging the model to explicitly encode residue–residue proximity and geometric relationships derived from the domain structures.
Evaluation Strategy: The models were evaluated on several zero-shot and transfer learning tasks:
- Homology Detection: Using the CATH S40 dataset (remote homology, <40% identity) with AUROC1 as the metric.
- Contact Prediction: Using categorical Jacobian analysis and attention map probing to assess the encoding of residue–residue contacts.
- Functional Prediction: Zero-shot Gene Ontology (GO) term prediction (Molecular Function, Biological Process, Cellular Component) via KNN label transfer on CAFA3 and Swiss-Prot benchmarks.
- Biophysical Properties: Prediction of enzyme catalytic efficiency ( $K_{cat}$ ), metal-ion binding, and thermostability.

Key Results

Homology Detection: Domain-centric pretraining yielded significant gains over model scale. A 650M-parameter TEDlm outperformed ESM2 3B (a 3-billion parameter model) and the alignment-based tool MMseqs2 on the CATH S40 benchmark. TEDlm3D, incorporating the structural loss, achieved an AUROC1 of 0.50, approaching the performance of the structure-based search tool Foldseek (0.53) using sequence input alone. Notably, even the 160M parameter TEDlm3D variant surpassed all sequence-only baselines.
Layer-wise Analysis: The study found that intermediate layers in all tested models contained richer structural information than the final layers. However, TEDlm3D models retained strong structural signals even in their final layers, unlike ESM2 models where the final layer performance dropped significantly below intermediate peaks.
Contact Prediction: Probes on attention maps and categorical Jacobians confirmed that TEDlm3D encodes stronger coevolutionary and contact signals than TEDlm or ESM2. The supervised contact head in TEDlm3D achieved high precision (P@L/5 $\approx$ 0.789 for 1.3B) in a single forward pass, significantly outperforming unsupervised Jacobian baselines.
Functional and Biophysical Tasks:
- Molecular Function (MF): TEDlm variants substantially outperformed ESM2 in zero-shot MF prediction, consistent with the hypothesis that many molecular functions are intrinsic to individual domains.
- Biological Process (BP) & Cellular Component (CC): ESM2 models maintained an advantage in these tasks, likely because these annotations depend on multi-domain context and protein interactions, which domain-level pretraining does not capture.
- Biophysical Properties: TEDlm and TEDlm3D matched or slightly exceeded ESM2 on enzyme catalytic efficiency and metal-ion binding (tasks defined by domain-level active sites). However, ESM2 performed better on cross-species thermostability, suggesting organism-level adaptations extend beyond individual domain properties.

Significance and Claims
The paper posits that domain-centric pretraining is a highly effective strategy for creating compact, structurally informed protein language models. The primary contribution is demonstrating that the quality and granularity of the pretraining data (isolated, structurally coherent domains) can outweigh the benefits of increased model scale or full-length sequence training.

The authors claim that:

Data Efficiency: Restricting training to domain segments allows smaller models (e.g., 650M) to outperform much larger models (e.g., 3B) on remote homology detection.
Structural Inductive Bias: Adding a C $\alpha$ distance-guided contact loss (TEDlm3D) effectively injects structural knowledge into the representation space, enabling sequence-only models to rival structure-based search tools like Foldseek.
Granularity Matters: The performance gains are task-dependent. Domain-level representations excel at tasks intrinsic to single domains (fold recognition, molecular function, catalytic efficiency) but may struggle with tasks requiring multi-domain context (subcellular localization, organism-level stability).
Representation Quality: The structural signal is encoded within the model's internal representations (attention maps and embeddings) rather than solely in a trained output head, as evidenced by the strong performance of unsupervised probes.

The work concludes that domain-centric pretraining offers a viable route to "compact, structurally informed protein language models" that generalize well across structural and functional prediction tasks without requiring structural inputs at inference.