Can Folding Models Tell Binders from Bluffers? Evidence from POISK: The Patent-Derived Antibody Dataset

Technical Summary: Can Folding Models Tell Binders from Bluffers? Evidence from POISK

Problem Statement
Structure prediction models are increasingly utilized as zero-shot digital screens in antibody design, operating under the assumption that high structural confidence (e.g., high interface predicted template modelling, or ipTM) implies true binding. However, rigorous validation of this premise has been constrained by small test sets, narrow target diversity, and the risk of data leakage from training-adjacent structures found in public repositories like the Protein Data Bank (PDB) and the Structural Antibody Database (SAbDab). Existing benchmarks often rely on temporal holdouts from these same sources, which may not adequately test a model's ability to generalize to novel, experimentally validated interactions outside the training distribution.

Methodology
To address these limitations, the authors introduce POISK (Patent-extracted Organization of Immunoglobulin Sequence Knowledge), a large-scale dataset derived from US patent literature. The methodology involves four primary stages:

1. Data Extraction: Using an LLM-based pipeline (Gemini 3 Pro), the authors extracted 89,074 antibody-antigen records from 10,110 US patents. The pipeline targeted 36 distinct fields, including epitope details and binding affinities. An automated auditor flagged ungrounded information, and a manual audit of 40 records confirmed high extraction accuracy (95% for benchmark records).
2. Benchmark Curation: A balanced benchmark set of 22,494 complexes was curated. This included 11,247 patent-validated binders with no known resolved structure (filtered to share <95% sequence identity with SAbDab entries). Each binder was paired with a "decoy" (putative non-binder) constructed from sequence-similar human repertoire antibodies (via KA-Search against Observed Antibody Space) or generated via germline reversion with randomized CDRH3 loops.
3. Folding and Scoring: Six leading structure prediction models were benchmarked: AlphaFold-Multimer, Boltz-2, IntelliFold v2, OpenFold3, Protenix-v1, and Protenix-v2. All models were run without templates using MSAs computed via ColabFold. The primary evaluation metric was ipSAE (interaction prediction Score from Aligned Errors), which focuses on interchain residue pairs below a specific Predicted Aligned Error (PAE) threshold, rather than scoring entire chains like ipTM.
4. Analysis: Models were evaluated on their ability to discriminate binders from decoys using pairwise accuracy, 1-in-50 screening simulations, and cross-model structural consensus (DockQ).

Key Contributions

• POISK Dataset: A novel, large-scale resource of 89,074 antibody-antigen records extracted from patents, offering a stringent test bed that minimizes data leakage from standard structural databases.
• Benchmarking Framework: A rigorous evaluation of six state-of-the-art folding models on their ability to distinguish true binders from sequence-similar decoys in a zero-shot setting.
• Metric Analysis: Demonstration that ipSAE outperforms standard ipTM for binding discrimination by isolating the interface signal from disordered regions.
• Consensus Signal: An analysis showing that structural agreement across multiple models provides an independent and powerful discriminative signal.

Results

• Discrimination Performance: The best-performing model, Protenix-v2, achieved a pairwise accuracy of 72% across all pairs. When restricted to the 50% of pairs with the highest structural confidence, accuracy rose to 87%.
• Screening Utility: In a "1-in-50" zero-shot screening simulation (identifying one true binder among 49 decoys), Protenix-v2 correctly ranked the true binder first in 29% of trials. This represents a 14-fold enrichment over the random baseline (2%), though it falls short of reliable single-candidate identification.
• Metric Superiority: ipSAE consistently yielded higher Cohen's d values (separation between binder and decoy populations) than ipTM across all models.
• Structural Consensus: Cross-model consensus significantly improved discrimination. When three or more models agreed on a binding pose (DockQ ≥ 0.23), pairwise accuracy reached 73.1%. Stricter consensus (e.g., 96.8% accuracy at DockQ ≥ 0.80) was possible but covered only a small fraction of pairs (3.0%).
• Epitope Localization: Despite improved binding discrimination, epitope localization remains unreliable. The best model (Protenix-v2) contacted annotated epitope residues in only 62.8% of cases, indicating that high confidence in binding does not guarantee accurate physical placement of the interaction.
• Affinity and Novelty: There was only a weak correlation between structural confidence and binding affinity, and a weak negative correlation between confidence and sequence distance to known structures, suggesting models capture a binary binding/non-binding distinction rather than graded affinity.

Significance and Claims
The paper claims that current folding models provide useful but limited signals for antibody screening. While they can substantially enrich for true binders over random selection (particularly when filtering for high-confidence predictions or using multi-model consensus), they are not yet reliable enough to replace experimental validation or serve as the sole criterion for candidate selection.

The authors emphasize that the POISK dataset is a critical community resource for future benchmarking and model training, as it extends well beyond the target space of public structural databases. They conclude that folding models should be complemented by orthogonal experimental or computational methods, particularly because high confidence scores do not necessarily equate to correct epitope localization or high-affinity binding. The work highlights the gap between statistical discrimination and reliable hit identification in de novo antibody design.