Technical Summary: Capabilities, Specificity Gaps, and Training-Data Dependence of AlphaFold3

Problem Statement

While AlphaFold2 (AF2) revolutionized single-protein structure prediction and demonstrated broad generalizability to protein complexes, the field has shifted toward modeling diverse biomolecular assemblies including RNA, DNA, lipids, and post-translational modifications (PTMs). AlphaFold3 (AF3) introduced an all-atom diffusion-based framework to address these expanded capabilities. However, the extent to which AF3's new features translate into reliable, generalizable performance across diverse biological application areas remains unclear. Specifically, there is a need to evaluate whether AF3's confidence metrics can distinguish genuine interactions from decoys (specificity) and whether its accuracy relies heavily on the presence of homologous structures in its training data (training-set dependence).

Methodology

The authors conducted a community-style assessment of AF3 across five distinct application areas, utilizing experimental structures and large-scale datasets as benchmarks. The study employed the following approaches:

1. Ubiquitinated Protein Structures:

   • Approach: Since AF3 cannot natively model covalent protein-protein bonds, the authors developed a workflow converting the C-terminal glycine of ubiquitin into a small-molecule ligand to specify covalent isopeptide bonds.
   • Dataset: A benchmark of 299 experimental structures from the PDB, including polyubiquitin chains, enzymatic intermediates, and mono-ubiquitinated proteins.
   • Evaluation: Root-mean-square deviation (RMSD) and DockQ scores were calculated against experimental structures. The impact of explicitly defining covalent bonds was tested by running predictions with and without them. A subset of "novel" structures (no homologs >30% identity in the pre-training PDB) was analyzed to assess generalization.

2. TCR-Epitope Recognition:

   • Approach: AF3 was used to model T-cell receptor (TCR) complexes with peptide-MHC (pMHC) molecules.
   • Scenarios: (1) Identifying epitope-specific TCRs from a repertoire; (2) Identifying cognate epitopes for a specific TCR from a peptide library; (3) Assessing the impact of single amino acid variants in TCRs or epitopes.
   • Datasets: Yellow Fever (YF) vaccinated donor repertoires and cancer neo-epitope datasets (Gumpert et al.).
   • Evaluation: Area Under the Curve (AUC) for enrichment of true positives, ranking of cognate pairs, and correlation with experimental binding affinity/variant scans.

3. Antibody-Antigen Complexes:

   • Approach: Prediction of antibody-SARS-CoV-2 Receptor Binding Domain (RBD) interfaces.
   • Datasets: A comprehensive set of antibodies against D614G and Omicron BA.1 strains (Cao et al.).
   • Evaluation: Comparison of "top-ranked" models (by AF3's internal score) versus "oracle best" models (selected by ground-truth accuracy). Analysis of interface recovery rates, epitope-likeness dependence, and the ability to predict binding affinity changes or mutation escape.

4. Protein-RNA Interactions:

   • Approach: Modeling RNA-binding proteins (RBPs) with RNA targets.
   • Datasets: Integrated CLIP-seq (POSTAR), cross-linking mass spectrometry (RNPxl, XRNAX), and multi-omics cancer data (CPTAC) to define true positive, random, and negative control pairs.
   • Evaluation: Interface predicted Template Modeling (ipTM) scores to distinguish true RBPs from non-RBPs and random RNA. Spatial accuracy was measured by the distance between predicted RNA and experimentally validated cross-link sites.

5. Protein-Lipid Co-folding:

   • Approach: Co-folding of lipid transport proteins (LTPs) with cognate lipids versus non-cognate lipids and non-LTPs.
   • Datasets: Structurally resolved LTPs (DB1), non-LTPs with cavities (DB2), and large-scale mass spectrometry datasets (DB4, DB5) partitioned by training-set overlap.
   • Evaluation: ipTM, min-iPAE, and avg-pLDDT metrics. Performance was measured via Area Under the Precision-Recall Curve (AUPRC) and Normalized Enrichment Factors (NEF) at various thresholds.

Key Results

1. Ubiquitinated Proteins

• Accuracy: AF3 achieved medium-to-high accuracy for many structures (169/299 with RMSD < 5 Å), particularly for enzymatic intermediates and polyubiquitin chains with binders.
• Specificity: Explicitly defining covalent bonds was essential; removing them caused widespread deterioration in accuracy, indicating AF3 cannot reliably infer ubiquitin orientation from context alone.
• Confidence: Model confidence (ipTM) was negatively correlated with RMSD, allowing for the enrichment of accurate predictions.
• Generalization: A small subset of novel structures showed similar or worse accuracy, but the sample size was too small for definitive conclusions on generalization.

2. TCR-Epitope Interactions

• Enrichment: AF3 showed strong enrichment (AUC=0.88) for identifying YF-specific TCRs from a repertoire, even without structural templates. However, enrichment was weak (AUC < 0.6) for cancer neo-epitopes.
• Prioritization: AF3 successfully ranked the cognate epitope highly (4th out of 251) for a known TCR.
• Variants: AF3 failed to accurately predict the impact of single amino acid variants on binding, showing weak correlation with experimental data even when the wild-type complex was modeled correctly.

3. Antibody-Antigen Complexes

• Interface Recovery: AF3 outperformed AF2 in interface recovery for many targets and was less sensitive to "epitope-likeness."
• Ranking Failure: While accurate interfaces were often present in the top 50 seeds, AF3's confidence metrics (ipSAE) frequently failed to select the correct model as the top rank.
• Affinity & Mutations: Predicted scores did not correlate with binding affinity. AF3 could not predict the loss of binding due to point mutations.

4. Protein-RNA Interactions

• Specificity Gap: AF3 could distinguish RBPs from non-RBPs but failed to distinguish genuine RNA targets from random RNA sequences. True positive pairs and random RNA pairs had statistically similar ipTM scores.
• Spatial Accuracy: Despite the lack of sequence specificity, AF3 placed RNA chains within 3–10 Å of experimentally validated cross-link sites in 77–91% of predictions.
• Training Dependence: Performance did not drop in the absence of homologous templates; in fact, confidence scores slightly increased for random pairs in the low-homology subset, suggesting the model relies on learned RNA-binding fold representations rather than template copying.

5. Protein-Lipid Co-folding

• Favorable Conditions: AF3 accurately reconstructed known protein-lipid complexes (RMSD ~0.08 Å) when the complex was in the training set.
• Specificity: AF3 showed limited specificity. Approximately 7% of predictions for non-LTPs (true negatives) yielded high confidence scores (ipTM > 0.8).
• Training Dependence: Performance dropped significantly in out-of-distribution settings (DB5, no training overlap). AUPRC fell from 0.309 (DB4, with overlap) to 0.189 (DB5, no overlap), indicating a strong dependence on training data exposure.

Significance and Claims

The authors conclude that while AF3 expands the scope of structure prediction to include diverse biomolecules and covalent modifications, its performance is uneven and less generalizable than the gains observed with AF2 for monomers and protein-protein complexes.

• Nuanced Utility: AF3 is a powerful tool for hypothesis generation and prioritization (e.g., identifying potential binders or ranking candidates) but is not yet a replacement for experimental validation in routine screening.
• Specificity Limitations: A recurring theme is the lack of specificity in confidence metrics. High scores often fail to distinguish true interactions from decoys, particularly in protein-RNA and protein-lipid contexts.
• Training-Set Dependence: Accuracy in several domains (notably protein-lipid and antibody-antigen) is heavily influenced by the presence of related complexes in the training data. The model struggles to generalize to out-of-distribution targets.
• Responsible Use: The paper advocates for a cautious approach where AF3 predictions are interpreted with respect to training-set overlap and are always accompanied by appropriate negative controls and orthogonal validation. The provided benchmarks are intended to serve as a standard for testing future developments in the field.