IMMREP25: Unseen Peptides

The IMMREP25 benchmark demonstrates that incorporating structural modeling significantly advances the prediction of TCR:pMHC binding for unseen peptides, achieving a macro-AUC_0.1 of 0.60 and marking a notable improvement over previous random-guessing performance on such data.

The Gist

Imagine your immune system as a massive, high-tech security force patrolling the body. Its job is to spot intruders (like viruses or cancer cells) and sound the alarm.

Here is how the security works:

1. The Bouncers (MHC): Specialized cells act as bouncers. They hold up little "wanted posters" (peptides) on their surface.
2. The Intruders (Peptides): These posters show pieces of viruses or bad cells.
3. The Detectives (TCRs): T-cells are the detectives. They have unique "magnifying glasses" (called T-cell receptors, or TCRs) that scan the bouncers. If a detective's magnifying glass fits perfectly with a specific wanted poster, the alarm goes off, and the body attacks.

The Big Problem:
Scientists want to predict which detective fits which poster before they even see them in real life. This would help us design better vaccines and cancer treatments.

For years, scientists could predict matches for "famous" posters (peptides they had seen before). But what about brand new, never-before-seen posters? This is like trying to guess which key fits a brand-new lock you've never seen. Previous attempts failed miserably; computers were just guessing randomly.

The Great TCR Prediction Contest (IMMREP25)

To solve this, a group of scientists organized a massive competition called IMMREP25. They gave 126 teams a challenge:

• The Task: Predict which 1,000 specific detectives (TCRs) would fit with 20 brand-new, "unseen" wanted posters (peptides).
• The Catch: The teams had never seen these specific posters before. They couldn't just look up the answer in a database. They had to use pure logic and physics.

The Results: A Shift in Strategy

In the past, teams tried to solve this by looking at the text of the sequences (like comparing the letters in the words). It didn't work well for new posters.

This time, the winners changed the game. Instead of just reading the text, they started building 3D models.

Think of it like this:

• Old Way: Trying to guess if a key fits a lock by reading the description of the key's teeth.
• New Way (The Winners): Using a super-powerful 3D printer to build a virtual model of the key and the lock, then physically trying to turn the key in the lock to see if it fits.

The Winners:
The top teams used advanced AI tools (like AlphaFold 3 and Chai-1) that are famous for predicting how proteins fold into 3D shapes. They built a virtual 3D model of the Detective (TCR) and the Poster (Peptide) sitting together.

• The best team (Bradley) used a specialized version of this 3D builder. They didn't just guess; they measured how "confident" the computer was that the two pieces fit together.
• Their success rate was significantly better than random guessing, proving that understanding the 3D shape is the key to solving the puzzle.

The Catch (Why it's not perfect yet)

While the winners did much better than before, they still didn't get it 100% right.

1. It's Expensive: Building these 3D models is like running a supercomputer marathon. It takes a lot of time and energy. You can't do this for every detective in the body (millions of them) right now.
2. The "Bouncer" Bias: The computers were much better at predicting matches for one type of bouncer (HLA-A02:01) than another (HLA-B40:01). This is because the computers had seen more examples of the first type in their training data.

The Bottom Line

This paper tells us that we have finally cracked the code on predicting how immune cells recognize new threats, but only if we stop looking at the "text" and start looking at the "shape."

The future of this field isn't just about better math; it's about better 3D modeling. The next step for scientists is to figure out how to make these 3D models faster and cheaper, so we can use them to design life-saving treatments for diseases we haven't even encountered yet.

Technical Summary

1. Problem Statement

The central challenge addressed is the prediction of T Cell Receptor (TCR) binding to peptide-MHC (pMHC) complexes for "unseen" peptides.

• Context: While machine learning models have achieved reasonable accuracy for "seen" peptides (those with existing experimental data in public databases like VDJdb or IEDB), they fail to generalize to peptides with no prior TCR binding data.
• Gap: Previous iterations of the IMMREP competition (2022, 2023) demonstrated that standard sequence-based deep learning models perform no better than random guessing when applied to unseen peptides.
• Goal: IMMREP25 aimed to evaluate the state-of-the-art in predicting TCR:pMHC specificity for entirely novel viral epitopes, testing whether the field has advanced beyond sequence similarity and look-up methods to a point where structural understanding can drive generalization.

2. Methodology

Dataset Construction

• Source: A novel dataset was generated by Adaptive Biotechnologies specifically for this competition.
• Composition:
  • 1,000 Unique TCRs: Experimentally determined to be specific to one of 20 unseen peptides.
  • 20 Unseen Peptides: 9-mer viral epitopes restricted by two MHC alleles: HLA-A*02:01 (10 peptides) and HLA-B*40:01 (10 peptides).
  • Novelty Constraint: Peptides were selected to have a minimum Levenshtein distance of 4 and no shared substring >5 amino acids with any previously published epitope in major databases (IEDB, VDJdb, McPAS).
• Labels:
  • Positive: 1,000 TCR-pMHC pairs (50 TCRs per peptide).
  • Negative: 9,000 TCR-pMHC pairs formed by matching each TCR to the 19 non-target peptides within its specific MHC context.
  • Total Records: 10,000 (downsampled to 8,938 to prevent distribution bias assumptions).
• Split: 2 peptides (175 records) were used for the public leaderboard; the remaining 18 peptides (7,344 records) formed the private evaluation set.

Competition Framework

• Participants: 126 named submissions from various academic and industrial groups.
• Constraints: No restrictions on training data usage; participants could use any external datasets or structural models.
• Evaluation Metric: Macro-AUC_0.1 (Area Under the Curve for the first 10% of False Positive Rate), averaged across the 18 private peptides. This metric prioritizes early retrieval of true positives.

Approaches Submitted

Methods fell into two primary categories:

1. Structural Modeling: Utilizing AlphaFold 3 (AF3), Chai-1, Boltz-1, or TCRdock to predict the 3D structure of the TCR-pMHC complex and deriving binding scores from confidence metrics (e.g., pLDDT, ipTM, PAE).
2. Sequence-Based: Using protein language models (ESM) or traditional machine learning (XGBoost) without explicit 3D structure generation.

3. Key Contributions

• Benchmarking Unseen Peptides: Established the first rigorous benchmark showing that TCR:pMHC prediction is possible for unseen peptides, moving beyond the "random guessing" baseline observed in previous years.
• Validation of Structural Methods: Demonstrated that structural modeling is the dominant paradigm for this specific task. The top-performing methods all relied on predicting the 3D complex structure.
• Performance of AF3: Showed that AlphaFold 3 (AF3) and its open-source derivatives (Chai-1, Boltz-1) are capable of capturing the physical constraints of TCR-pMHC interactions even without prior sequence-specific training data for the target peptide.
• Distillation Success: Highlighted TAPIR3, a sequence-based model that distills knowledge from structural models (Chai-1), proving that structural information can be effectively compressed into sequence-only models for faster inference.

4. Results

• Overall Performance:
  • 73% of submissions exceeded random performance (AUC_0.1 > 0.50).
  • Top Performer (Bradley Method): Achieved a Macro-AUC_0.1 of 0.601. This method used a modified TCRdock pipeline with AF3, utilizing pLDDT (predicted Local Distance Difference Test) averaged across peptide and CDR loops as the scoring metric.
  • Runner-ups: Altin (0.582) and Pierce (0.576) also used AF3-based structural modeling, scoring based on Predicted Alignment Error (PAE) at the interface.
• Method Comparison:
  • Structural vs. Sequence: The top 15 methods (AUC_0.1 ≥ 0.52) were almost exclusively structural.
  • Sequence-Only Baseline: Purely sequence-based methods (e.g., STAPLER, TAPIR3's raw input, and legacy tools like NetTCR2.2) generally scored between 0.500 and 0.518, confirming their inability to generalize to unseen peptides without structural priors.
  • TAPIR3 Exception: TAPIR3 (AUC_0.1 = 0.538) outperformed native Chai-1 methods by using a knowledge distillation approach, bridging the gap between structural accuracy and sequence speed.
• HLA Bias: Performance was significantly higher for HLA-A*02:01 peptides compared to HLA-B*40:01. This correlates with the scarcity of HLA-B*40:01 structural data in the PDB and training sets (only 1 complex vs. 265 for A*02:01).
• Clustering: While clustering TCRs improved performance on the public leaderboard (2 peptides), it did not provide a systematic advantage on the private leaderboard (18 peptides), suggesting that dataset-specific biases were being exploited rather than generalizable biological rules.

5. Significance and Future Directions

• Paradigm Shift: The results mark a definitive shift in the field from sequence-based similarity matching to structure-based prediction. For unseen peptides, understanding the physical geometry of the interaction is crucial.
• Clinical Potential: Reliable prediction of TCR binding to novel viral or neoantigen peptides is a critical step for developing TCR-based therapies and diagnostics. IMMREP25 proves this is now computationally feasible, albeit with room for improvement.
• Computational Cost: A major limitation identified is the high computational cost of running AF3/Chai-1 for large-scale repertoire screening.
• Future Outlook: The paper suggests that knowledge distillation (like TAPIR3) is the path forward to make structural-level accuracy accessible at the deep repertoire scale. The authors anticipate IMMREP26 will focus on refining these distillation techniques and addressing the data scarcity for underrepresented HLA alleles.

In summary, IMMREP25 demonstrates that while predicting TCR specificity for unseen peptides remains a difficult challenge, the integration of next-generation protein structure prediction tools (AF3, Chai-1) has broken the "random guessing" barrier, offering a promising new trajectory for immunological AI.