🧬 biology

Library size and structure, not synthetic labels, explain apparent benchmark--experiment discrepancies in antibody machine learning

This paper demonstrates that apparent discrepancies between synthetic benchmarks and experimental results in antibody machine learning stem from uncontrolled differences in library size, structure, and noise rather than the use of synthetic labels, and shows that matching these conditions resolves most performance disagreements.

Original authors: Dohun Kim, Minho Bae

Published 2026-11-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dohun Kim, Minho Bae

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

In the race to design new medicines, scientists are increasingly turning to antibodies, the body's natural defense proteins, as the blueprint for powerful therapies. To find the best versions of these proteins, researchers must sift through vast libraries of genetic sequences, looking for the few that bind tightly to a specific disease target, such as a virus or a cancer cell. Because testing every possible sequence in a laboratory is slow and expensive, teams now rely on computer models to predict which sequences will work best. These models are trained on data, but the data often comes in two very different forms. One type is real, measured in a lab, but usually limited to a small group of related sequences. The other is synthetic, generated by computers to simulate millions of unrelated sequences, offering a massive scale but lacking the physical reality of a wet-lab experiment. For years, the scientific community has debated whether conclusions drawn from these massive computer simulations actually hold true when applied to real-world experiments.

A team of researchers at the Electronics and Telecommunications Research Institute in South Korea set out to settle this debate by looking closely at why computer simulations and real experiments often seem to disagree. They suspected that the problem was not the computer-generated labels themselves, but rather the way the libraries of sequences were built and compared. In their study, they took a standard computer simulation that had been used for years and rebuilt it to look exactly like a real laboratory dataset. They kept the computer's scoring system the same but changed the library to contain only six thousand sequences, all derived from a single parent antibody, just as a real lab would create. This allowed them to compare the two worlds side-by-side, changing only the size and structure of the data while keeping everything else identical.

When the researchers first compared the original, massive computer library against the smaller, real-world data, they found four major disagreements. The computer suggested that a guided search for the best antibodies was nearly a thousand times more effective than random guessing, while the real data showed only a modest improvement. The computer favored a simple way of representing protein sequences, while the real data seemed to prefer a more complex, language-based approach. The computer also suggested that a specific type of neural network was superior, whereas the real data pointed to a simpler statistical method. Finally, the computer suggested that the difficulty of the task was split evenly between measurement errors and the inherent complexity of the protein, a conclusion that did not match the real-world view.

However, once the researchers rebuilt the computer library to match the size and structure of the real data, three of these four disagreements vanished. The massive advantage of the guided search collapsed from a factor of over a thousand down to a modest three-fold improvement, which aligned perfectly with what was seen in the real experiments. The preference for the complex language-based model over the simple one also flipped, showing that the earlier conclusion was driven by the structure of the library, not the nature of the labels. The debate over which computer architecture was best also resolved; it turned out the two sides had simply stopped looking at the data at different points, and when they looked at the same range, they agreed on the crossover point. The fourth disagreement regarding the split between noise and complexity could not be measured because the two libraries still differed in how many positions varied, but the authors noted this was a limitation of the specific data rather than a fundamental flaw in the method.

The study did find a few effects that remained consistent regardless of how the data was structured. Even with matched libraries, a guided search still offered a small but reliable advantage over random guessing, confirming that active learning is a useful tool. The researchers also quantified how much measurement noise alone reduces the accuracy of predictions, finding it lowers the correlation by about ten percent, a floor that no model can cross. They further discovered that using random splits to test models often makes them look much better than they actually are; when tested on truly new, unseen data, the models performed eight times worse than the random splits suggested. This happened because the random splits kept the test data too close to the training data, failing to challenge the model to make genuine predictions.

Ultimately, the paper demonstrates that synthetic benchmarks are not inherently broken or misleading. Instead, the apparent contradictions between computer simulations and real experiments arise when researchers compare apples to oranges. By matching the size, structure, and noise levels of the datasets, the computer models proved to be faithful proxies for reality. The findings suggest that the field does not need to abandon its powerful synthetic tools, but rather needs to be more careful about how it sets up comparisons. When the conditions are aligned, the computer and the lab tell the same story, offering a clearer path forward for designing the next generation of life-saving antibodies.

Technical Summary: Library Size and Structure, Not Synthetic Labels, Explain Apparent Benchmark–Experiment Discrepancies in Antibody Machine Learning

Problem Statement

Machine learning (ML) for antibody discovery increasingly relies on synthetic benchmarks, such as Absolut!, to evaluate methods for ranking candidate sequences. These benchmarks provide ground-truth binding energies at scales unattainable by wet-lab experiments. However, a structural risk exists in comparing conclusions drawn from synthetic data to those from measured experimental data.

Typical synthetic and measured datasets differ simultaneously across five dimensions:

Label Origin: Simulated lattice energies vs. experimental log10 dissociation constants ( $K_D$ ).
Library Size: Synthetic libraries often contain millions of unrelated sequences (e.g., $6.9 \times 10^6$ ), whereas measured libraries are typically mutational neighborhoods of a single parent (e.g., $2\text{--}3.5 \times 10^4$ ).
Library Structure: Synthetic sets are collections of unrelated CDRH3 sequences; measured sets are structured mutational neighborhoods (single, double, and triple mutants).
Alignability: Synthetic sequences vary in length (11–43 residues) and lack positional alignment; measured sequences are aligned to a parent.
Noise: Synthetic labels are noise-free; experimental data contain assay noise.

The central problem is that observed discrepancies between synthetic and experimental results (e.g., regarding search efficiency, preferred sequence representations, or model architectures) have been attributed to the synthetic labels themselves. This paper argues that these discrepancies are actually driven by the uncontrolled covariation of library size, structure, and noise, rather than the realism of the labels.

Methodology

The authors employed a "matched lattice" experimental design to isolate the effect of library structure from label origin.

Fixed Oracle, Variable Library: The study held the Absolut! oracle (the scoring function and antigen structure) constant. Instead of using the standard, massive, unrelated synthetic library, they reconstructed a new library that mimicked the structure and size of a typical measured dataset.
Construction of the Matched Library:
- A strong-binding parent CDRH3 sequence (length 17) was selected from the synthetic oracle.
- All 286 possible single mutants were enumerated.
- Random double and triple mutants were added to reach a total of 6,000 sequences, matching the design recipe of the AlphaSeq experimental libraries.
- All sequences were scored using the same Absolut! binary, ensuring the labels remained synthetic lattice energies.
Comparative Framework:
- Arm A (Unmatched): Standard synthetic library (large, unrelated) vs. Standard measured library (small, mutational).
- Arm B (Matched): Rebuilt synthetic library (small, mutational, same parent) vs. Standard measured library.
- Arm C (Independent Validation): An independent prospective campaign (AlphaBind/AlphaSeq) involving 119,828 measurements was used to validate representation choices.
Evaluation Metrics: The study compared methods based on:
- Search Advantage: The ratio of top-fraction recovery by active learning vs. random sampling.
- Representation: One-hot encoding vs. Protein Language Model (PLM) embeddings (ESM-2).
- Architecture: Ridge regression vs. Convolutional Neural Networks (CNN).
- Learnability Decomposition: Partitioning error into noise vs. intrinsic smoothness.
- Prospective Performance: Random splits vs. prospective splits (training on one campaign, testing on a later, more distant campaign).

Key Results

1. Discrepancies Disappear Upon Matching

When library size and structure were matched, three of the four major discrepancies between synthetic and experimental regimes vanished:

Search Advantage: The apparent 1,620-fold advantage of active learning over random sampling on the unmatched synthetic library collapsed to 3.3-fold on the matched lattice. This aligned closely with the 3.0-fold advantage observed on measured data. The massive advantage in the unmatched regime was an artifact of the library size and lack of structure.
Representation Preference: On the unmatched synthetic library, one-hot encoding outperformed ESM-2 embeddings at all training sizes. On the matched lattice (and measured data), ESM-2 embeddings outperformed one-hot encoding at 3 out of 5 training sizes, reversing the conclusion. This suggests the preference for PLMs is driven by the alignability of mutational neighborhoods, not the label type.
Architecture Crossover: The apparent disagreement regarding whether Ridge regression or CNNs were superior was resolved as an artifact of the training set size range examined. Both regimes showed a crossover point at approximately 2,000 labels: Ridge performed better below this threshold, while CNNs performed better above it.

2. Effects That Survived Matching

Several findings remained consistent regardless of library matching, indicating they are robust properties of the problem space:

Modest Search Advantage: Even with matched conditions, active learning retains a single-digit advantage (approx. 3-fold) over random sampling.
Measurement Noise Floor: Injecting experimental noise levels ( $\sigma = 0.27$ in log10 $K_D$ ) into the matched synthetic data reduced Spearman correlation by 0.105. This establishes a performance floor attributable solely to assay noise.
Random Splits Overstate Performance: Evaluating models via random splits on mutational neighborhoods overstates prospective performance by 8.1-fold. When tested prospectively (training on an early campaign, testing on a later, more distant campaign), recovery of the top 100 candidates dropped significantly. This is due to the "extrapolation gap" where later campaigns explore sequence space further from the parent than earlier ones.
Combined Representations: A model combining ESM-2 embeddings with one-hot features consistently outperformed other architectures (Ridge, CNN, Random Forest) across four measured targets.
Read-Count Models: On panning data, models predicting final abundance from early read counts did not outperform a trivial baseline of ranking by the last available read count. Learning value is found in scoring unmeasured sequences, not re-predicting counts.

3. Independent Validation

An independent prospective campaign (AlphaBind) involving 119,828 measurements confirmed the representation findings. Candidates generated using ESM-2 embeddings showed significantly higher improvement rates over parent antibodies compared to those generated using one-hot encodings across three different parental antibodies.

Significance and Claims

The paper claims that synthetic benchmarks are faithful proxies for experimental data, provided that the comparison conditions are matched. The apparent contradictions in the literature are not due to the synthetic labels being "unrealistic," but rather to confounding variables (library size, structure, and noise) that vary simultaneously between the two regimes.

Attribution: The study demonstrates that the "synthetic label" is not the primary source of discrepancy; rather, the library design (size and mutational structure) dictates the observed performance characteristics.
Protocol for Future Benchmarks: The authors propose a protocol for future synthetic benchmarking to ensure interpretability. Before selecting a method, researchers should match the synthetic benchmark to the intended application in five specific properties:
1. Library size.
2. Library structure (mutational neighborhood vs. unrelated).
3. Number of varying positions.
4. Measurement noise (injected at the assay's noise-to-signal ratio).
5. Range of training-set sizes (to capture architecture crossovers).

The paper concludes that a substantial part of the disagreement between synthetic and measured evidence is a property of how the comparison is set up, and is therefore fixable by design rather than requiring new machine learning methods. The findings reinforce the broader principle in protein engineering that the design of the training set often matters more than the choice of model.