🧬 biology

A Comprehensive Benchmark of Computational Models for Antibody--Antigen Binding Affinity

This paper introduces AbBiBench, a comprehensive benchmark of over 216,000 experimental measurements that evaluates 20 computational models for antibody-antigen binding affinity, revealing that global structure-conditioned inverse folding outperforms fine-tuned approaches and demonstrating the framework's utility in prioritizing functional variants with high experimental success rates.

Original authors: Xinyan Zhao, Yi-Ching Tang, Victor J Cantu, Junseok Lee, Adam E Stogsdill, Ibraheem M Hamdi, Ashwin Kumar Ramesh, Zhiqiang An, Xiaoqian Jiang, Yejin Kim

Published 2026-11-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinyan Zhao, Yi-Ching Tang, Victor J Cantu, Junseok Lee, Adam E Stogsdill, Ibraheem M Hamdi, Ashwin Kumar Ramesh, Zhiqiang An, Xiaoqian Jiang, Yejin Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The immune system relies on a sophisticated defense force of Y-shaped proteins called antibodies. These molecules act as precise sentinels, scanning the body for invaders like viruses or bacteria. Their success depends on a lock-and-key mechanism: the antibody must physically grasp a specific target, known as an antigen, with enough strength to neutralize the threat. Scientists have long sought to engineer better versions of these antibodies, tweaking their shapes to bind more tightly to dangerous pathogens. This process is vital for creating new medicines, but it faces a massive hurdle. The number of possible antibody variations is so vast that testing them one by one in a lab is impossible. Researchers need a way to predict which designs will work before they ever touch a test tube.

For years, the standard approach to evaluating computer models designed to create these antibodies has been flawed. Many programs were judged simply on how closely their generated sequences resembled natural antibodies found in nature. However, the immune system creates antibodies through a chaotic, random process of mutation and recombination, producing shapes that often look nothing like existing natural ones. Judging a new design by how much it looks like something that already exists misses the point. The true test of an antibody is not its resemblance to the past, but its ability to grab onto a specific target in the present. A new study introduces a comprehensive benchmark called AbBiBench, which shifts the focus entirely to this functional goal. Instead of asking if a design looks natural, the researchers asked if the computer models could correctly predict which antibody variants would actually bind tightly to their targets.

To build this new standard, the team gathered a massive collection of experimental data. They compiled over 216,000 measurements of how strongly different antibody variants bound to nine different antigens, including parts of the flu virus and the SARS-CoV-2 virus. These measurements came from 16 different laboratory tests, covering a wide range of antibody targets and mutation types. The researchers then used this dataset to test twenty different computational approaches. Crucially, they tested these models in their "zero-shot" state, meaning they evaluated the software exactly as it was released, without teaching it specifically about these antibodies or fine-tuning it with the new data. This ensured the results reflected the models' genuine ability to generalize, rather than their ability to memorize a specific dataset.

The results revealed a clear winner in the race to predict binding strength. Models that used the full three-dimensional structure of the antibody-antigen complex as a guide performed significantly better than those that looked only at the sequence of letters in the protein or focused on small, local parts of the structure. Specifically, a method known as inverse folding, which predicts the best amino acid sequence for a given 3D shape, emerged as the most consistent and accurate tool. The top-performing models could rank the best candidates with high reliability, identifying variants that were likely to bind much more strongly than the original. In contrast, models that relied solely on the order of amino acids, or those that were specifically fine-tuned on antibody data, did not consistently improve performance and sometimes performed worse than their general-purpose counterparts.

The study also uncovered a surprising asymmetry in how these computer models should be used. The models proved excellent at finding the "winners"—they were very good at spotting the few variants that would likely bind strongly and should be tested in the lab. However, they were much less reliable at identifying the "losers." A low score from a computer model was not a strong enough reason to discard a variant, as many weakly ranked candidates still showed some binding ability. This suggests that while computers are powerful tools for narrowing down the search and prioritizing the most promising designs, they cannot yet be used to safely rule out other possibilities. The best strategy remains using these models to enrich a shortlist of candidates for experimental testing, rather than using them to filter out the rest.

To prove that these retrospective findings could translate into real-world design, the researchers conducted a prospective case study. They asked the models to design new versions of an antibody that naturally binds to one strain of the flu but fails to bind to another. The goal was to create variants that could grab the second strain. The models generated thousands of new sequences, and the researchers used a two-step filtering process to select the most promising ones based on predicted binding strength and sequence stability. They chose twenty-one top candidates and tested them in the lab. Every single one of the designed variants successfully bound to the target flu virus, a result that confirmed the models' ability to generate functional, high-affinity designs. This success demonstrated that by focusing on the physical interaction between the antibody and its target, rather than just the sequence, scientists can now use artificial intelligence to navigate the vast landscape of antibody design with greater confidence and efficiency.

Technical Summary: A Comprehensive Benchmark of Computational Models for Antibody–Antigen Binding Affinity

Problem Statement

Computational antibody design currently lacks a comprehensive benchmark that evaluates models based on functional binding affinity rather than sequence similarity to natural variants. Traditional validation strategies often rely on comparing designed sequences to naturally occurring proteins, an approach that fails to capture the stochastic recombination and hypermutation inherent to antibody generation. Furthermore, existing benchmarks (e.g., ProteinGym, BindingGYM) focus on general protein mutational landscapes or pairwise interactions without treating the full antibody–antigen (Ab-Ag) complex as the fundamental evaluation unit. While machine learning models (protein language models, generative models, and structure predictors) show promise, their ability to predict binding affinity without task-specific training remains unclear, and there is no standardized framework to compare diverse computational approaches across a broad range of antibody-antigen pairs.

Methodology

The authors introduce AbBiBench (Antibody Binding Benchmarking), a unified framework designed to evaluate computational models in a "zero-shot" setting (without fine-tuning on the benchmark data).

1. Dataset Curation

AbBiBench harmonizes 216,560 experimental measurements from 16 distinct assays, covering 13 antibodies and 9 antigen targets (including Influenza A strains, SARS-CoV-2 Spike, VEGF, HER2, and others).

Data Types: The dataset includes 11 direct $K_d$ assays (81.8% of observations) and 5 enrichment assays (18.2%).
Variants: The collection spans nearly three orders of magnitude in library size, from 67 to 65,535 variants per assay. Mutations cover heavy chains, light chains, and both, with mutation depths ranging from single substitutions to libraries dominated by variants with five or more mutations.
Structural Templates: Where experimentally resolved complexes were unavailable (specifically for five SARS-CoV-2 assays), AlphaFold 3 was used to generate structural templates for structure-conditioned models.

2. Model Evaluation

The study compares 20 computational approaches across seven methodological families:

Masked Protein Language Models (PLMs): ESM-2, ESM-3, SaProt, ProSST, AntiBERTy, CurrAb.
Autoregressive PLMs: ProGen2, ProtGPT-2.
Inverse-Folding Models: ProteinMPNN, ESM-IF1, AntiFold.
Diffusion-Based Generative Models: DiffAb.
Geometric CDR-Imputation Models: MEAN, dyMEAN.
Structure-Prediction Confidence Models: AlphaFold 3, Boltz-2, ESMFold2 (evaluated primarily using ipSAE).
Biophysics-Based Methods: FoldX (binding free energy), FreeSASA (epitope SASA), ESMDance.

3. Evaluation Metrics

Ranking Correlation: Spearman rank correlation ( $\rho$ ) between model scores and experimental binding affinities, calculated within each assay to prevent large libraries from dominating.
Prioritization: Fivefold Precision@k (for $k \in \{10, 20, 50, 100\}$ ) to measure the ability to identify variants with at least fivefold affinity improvement.
Negative Filtering: The ability to exclude variants worse than the wild type.
Prospective Case Study: A gain-of-binding study redesigning the F045-092 antibody (specific to H3N2) to bind H1N1. Variants were generated, filtered via a two-phase computational pipeline, and validated via endpoint ELISA.

Key Results

1. Global Structure Conditioning Outperforms Sequence-Only Models

Inverse-folding models demonstrated the strongest and most consistent zero-shot affinity prediction.

Top Performers: ProteinMPNN achieved the highest mean equal-assay Spearman $\rho$ (0.292), followed closely by ESM-IF1 (0.278).
Mechanism: These models encode the entire Ab-Ag complex as a global representation, capturing long-range residue interactions and contextual features at the binding interface that local-structure token models (e.g., SaProt, ProSST) and sequence-only autoregressive models (e.g., ProGen2) miss.
Structure Prediction: Among structure-prediction models, ESMFold2 (scored by ipSAE) performed best, ranking third overall, outperforming AlphaFold 3 and Boltz-2 in this specific affinity ranking task.

2. Limited Value of Antibody-Specific Fine-Tuning

The study found that antibody-specific fine-tuning does not consistently improve performance and can sometimes degrade it.

CurrAb vs. ESM-2: The antibody-adapted CurrAb showed a marginal gain over the general ESM-2 model.
AntiFold vs. ESM-IF1: Conversely, AntiFold (an ESM-IF1 model fine-tuned on antibody structures) performed significantly worse than the base ESM-IF1 model, suggesting potential catastrophic forgetting or limited gains from specific adaptation in this zero-shot context.

3. Asymmetric Utility: Prioritization vs. Exclusion

Model rankings are significantly more useful for prioritizing likely affinity gains than for excluding weak variants.

Prioritization: At $k=10$ , ProteinMPNN and ESM-IF1 achieved precisions of 0.51 and 0.48, respectively, nearly double the random baseline (0.268).
Exclusion: When used as a filter to discard the lowest-scoring 30% of variants, the top models only achieved a modest enrichment over random ranking (capturing ~40% of "bad" variants vs. 30% for random). A low model rank alone is insufficient evidence to exclude a variant from experimental testing.

4. Prospective Validation (F045-092 Case Study)

In a prospective study, models generated CDR-H3 variants for the F045-092 antibody to target H1N1.

Trade-offs: A clear trade-off was observed between predicted binding potential and evolutionary sequence plausibility. For instance, MEAN produced favorable binding energy predictions but with lower sequence plausibility, while SaProt and ESM-IF balanced both metrics.
Experimental Outcome: All 21 selected candidates (derived from ESM-IF, MEAN, and SaProt) produced detectable H1-reactive ELISA signals, confirming that the computational pipeline successfully identified variants with gain-of-function binding capabilities.

Significance and Claims

The paper positions AbBiBench as a critical resource for the antibody engineering community, offering:

A Function-Aware Benchmark: By treating the Ab-Ag complex as the evaluation unit, AbBiBench aligns computational evaluation with the biological reality of antibody interactions, moving beyond sequence similarity metrics.
Practical Guidance: The results provide actionable insights for method development, highlighting that global structure-conditioned inverse folding is currently the most robust approach for affinity ranking without fine-tuning.
Realistic Expectations: The study clarifies that current zero-shot scores are best utilized for enriching experimental shortlists (finding "winners") rather than filtering out "losers," preventing the premature discarding of potentially viable candidates.
Open Framework: AbBiBench provides a standardized data resource and evaluation pipeline to accelerate the development of function-aware antibody design methods, bridging the gap between retrospective benchmarking and prospective experimental validation.

The authors conclude that while significant progress has been made, future work must address limitations such as the lack of kinetic data ( $k_{on}$ , $k_{off}$ ) and the need for orthogonal validation (e.g., SPR/BLI) to further refine these models for therapeutic discovery.