Structural Plausibility Without Binding Specificity: Limits of AI-Based Antibody-Antigen Structure Prediction Confidence Scores

This study demonstrates that while state-of-the-art AI methods can generate geometrically plausible antibody-antigen structures, their internal confidence scores fail to reliably distinguish correct binding pairs from incorrect ones, highlighting a critical need for explicit negative controls and realistic benchmarking in therapeutic discovery.

The Gist

Imagine you are a detective trying to find a specific key that fits a specific lock. You have a bag of 106 real keys and 106 real locks. You know exactly which key opens which lock (the "Real" pairs). But to test your detective skills, you also mix them up randomly, creating thousands of fake pairings where a key is held up to a lock it doesn't belong to (the "Shuffled" pairs).

Your goal is to use a high-tech AI scanner to look at these pairs and say, "Yes, this key fits this lock!" or "No, this is a mismatch."

This paper is about testing three of the most advanced AI scanners available today (AlphaFold3, Boltz-2, and Chai-1) to see if they can actually tell the difference between a real match and a fake one.

The Big Surprise: The AI is "Polite" but Wrong

The authors found that these AI tools are incredibly good at making things look like they fit.

• The Analogy: Imagine you have a square peg and a round hole. The AI is so good at geometry that it can twist the square peg into a shape that looks like it fits the round hole perfectly. It creates a "plausible" structure.
• The Problem: Just because the peg looks like it fits doesn't mean it actually turns the lock. The AI generates a beautiful, geometrically sound structure for the fake pairs just as often as it does for the real pairs.

The "Confidence Score" Trap

When you ask these AI tools, "How sure are you that this is a match?" they give you a confidence score (like a grade from 0 to 1).

• The Reality: The paper shows that these scores are not reliable for telling real matches from fake ones.
• The Metaphor: It's like a weatherman who is 90% confident it will rain, but it's actually sunny. The AI says, "I'm very confident this key fits!" even when it's holding a key to a completely different house.
  • AlphaFold3 was the "best" of the bunch, but it still failed to distinguish real from fake most of the time.
  • Boltz-2 was "overconfident," giving high scores to almost everything, even the mismatches.
  • Chai-1 was "underconfident," sometimes missing good matches because it didn't trust its own predictions.

The "More Sampling" Myth

A common idea in AI is: "If we run the simulation 100 times instead of once, we'll get a better answer."

• What the paper found: Running the AI 100 times does make the shape of the key and lock look slightly better (more polished). However, it does not help the AI realize if it's holding the wrong key in the first place.
• The Analogy: Imagine you are trying to solve a maze. If you run the maze 100 times, you might draw the walls a bit straighter and the path a bit smoother. But if you started in the wrong room, drawing the walls better won't get you to the exit. The AI gets stuck in the "wrong room" (the wrong binding mode) and just refines that mistake.

The Cost of Computing

The researchers also measured how much electricity these AI tools use.

• The Finding: Running the AI 100 times uses a lot of energy (like leaving a high-powered computer running for hours).
• The Advice: The paper suggests that running the AI 10 to 25 times is usually enough to get a "good enough" shape. Running it 100 times is mostly a waste of energy because the AI isn't learning anything new about which key fits; it's just polishing the same wrong answer.

The Bottom Line for Drug Discovery

Scientists use these AI tools to design new medicines (antibodies) to fight diseases. They hope to generate thousands of potential drug candidates and use the AI's confidence score to pick the best ones.

• The Warning: This paper warns that you cannot trust the AI's confidence score alone. If you pick the top 100 "most confident" predictions, you will likely get a mix of real winners and a huge number of "hallucinations" (fake matches that look real but don't work).
• The Solution: Instead of just trusting the AI's internal score, scientists need to use "negative controls." This means testing the AI against fake, shuffled pairs to see if it can tell the difference. If the AI can't tell the difference between a real match and a fake one, its high confidence score is meaningless.

In short: The AI is a master architect that can build beautiful, plausible-looking castles. But it is currently terrible at knowing which castle is actually built on solid ground and which one is just a mirage. We need better ways to check the foundation before we start building our medicines.

Technical Summary

1. Problem Statement

The central challenge addressed is the inability of current AI-driven protein structure prediction tools to distinguish between cognate (biologically correct) antibody-antigen interactions and non-cognate (incorrect but structurally plausible) pairings.

• Context: While tools like AlphaFold3 (AF3), Boltz-2, and Chai-1 have revolutionized protein structure prediction, their application to antibody discovery is hindered by high false-positive rates.
• The Gap: In therapeutic discovery, researchers must screen thousands of candidates to find specific binders. Current workflows rely on internal confidence metrics (e.g., ipTM, pLDDT) to filter candidates. However, it is unclear if these metrics reflect binding specificity (correct paratope-epitope pairing) or merely structural self-consistency (geometric plausibility).
• Hypothesis: The authors hypothesize that current models generate geometrically plausible structures for incorrect pairings ("shuffled" complexes) and that their internal confidence scores are not calibrated to discriminate these from true biological interactions.

2. Methodology

The study employs a rigorous, controlled benchmarking framework designed to test specificity under realistic conditions.

• Dataset Curation:

  • Real Complexes: 106 experimentally determined single-chain antibody (nanobody/VHH)-antigen complexes extracted from the PDB (91 unique PDB entries).
  • Shuffled Complexes (Negative Controls): 11,130 non-cognate pairs generated by systematically recombining VHHs and antigens from different experimental complexes. These serve as ground-truth decoys.
  • Total Scale: Approximately 1.8 million AI-generated structures were produced (106 real + 11,130 shuffled $\times$ 50 stochastic replicates each).

• Tools Evaluated:

  • AlphaFold3 (AF3)
  • Boltz-2 (and Boltz-1 for comparison)
  • Chai-1

• Evaluation Metrics:

  • ipTM (interface predicted TM-score): The primary internal confidence metric for interface quality.
  • DockQ: A structural accuracy metric comparing predicted complexes to experimental references (calculated only for real/cognate pairs).
  • Clash Score: Measures steric clashes to assess geometric plausibility.
  • Epitope Recall: The fraction of experimentally defined epitope residues recovered by the prediction.
  • Energy Consumption: GPU energy usage (Wh) to assess computational cost.

• Experimental Design:

  • All-vs-All Screening: Every VHH was paired with every antigen to create a matrix of real (diagonal) and shuffled (off-diagonal) predictions.
  • Saturation Sampling: Models were run with varying numbers of diffusion samples ($N=1, 10, 25, 50, 100$) and independent random seeds to test if increased sampling improves discrimination or structural quality.
  • Data Leakage Analysis: Systems were categorized as "train," "test," or "mixed" based on the training cutoffs of each model to rule out memorization as the sole cause of performance.

3. Key Results

A. Failure to Discriminate Specificity

• Confidence Scores Lack Specificity: Internal confidence metrics (ipTM) failed to distinguish real complexes from shuffled ones.
  • Heatmaps: Interaction landscapes showed no systematic enrichment of high scores along the diagonal (real pairs). Shuffled pairs frequently achieved ipTM scores comparable to or higher than real pairs.
  • Precision-Recall (PR) Analysis: All tools performed poorly. AF3 achieved the highest Average Precision (AP = 0.187), but this is barely above the random baseline (~0.011). Boltz-2 and Chai-1 performed significantly worse (AP = 0.026 and 0.067, respectively).
  • Score Distributions: The distributions of ipTM scores for real and shuffled complexes heavily overlapped across all tools and dataset splits (train/test/mixed).

B. Structural Plausibility vs. Biological Correctness

• Geometric Plausibility: Shuffled complexes often produced geometrically plausible structures with low clash scores, indistinguishable from real complexes in many cases.
• Epitope Recovery: Models frequently recovered epitope residues even in shuffled (incorrect) pairings. Epitope recall did not reliably differentiate real from shuffled interactions, suggesting models have a generic "epitope-seeking" bias rather than specific recognition.

C. Confidence Calibration and Sampling

• Misalignment of Confidence and Quality:
  • Overconfidence: Boltz-2 frequently assigned high ipTM scores to structurally poor (low DockQ) predictions.
  • Underconfidence: Chai-1 often assigned low scores to high-quality structures.
  • Decoupling: There was a near-zero correlation between changes in structural quality ($\Delta$DockQ) and changes in confidence ($\Delta$ipTM) during saturation sampling. Increasing sampling improved geometry but did not update the confidence score to reflect this improvement.
• Sampling Efficiency:
  • Diminishing Returns: Most structural quality gains occurred between $N=1$ and $N=10$ samples. Beyond $N=25$, gains were marginal.
  • Seed Sensitivity: The choice of random seed had a larger impact on final quality than deep sampling within a single seed. Some seeds explored "wrong" solution landscapes that no amount of sampling could rescue.
  • Cost: Chai-1 was the most energy-intensive at high saturation, while AF3 had a high baseline cost (MSA generation) but lower marginal costs.

D. Cross-Tool Agreement

• Low Consensus: There was weak agreement (Pearson $r \approx 0.13-0.18$) between tools regarding which pairs were high-confidence.
• Idiosyncratic Failures: "Overconfident failures" (high ipTM, low DockQ) were tool-specific and rarely overlapped between different models.

4. Key Contributions

1. Novel Benchmarking Framework: Introduced a "real-vs-shuffled" paradigm that explicitly tests binding specificity, moving beyond standard docking benchmarks that only assess geometric accuracy on known pairs.
2. Empirical Evidence of Limitations: Provided comprehensive data showing that state-of-the-art models (AF3, Boltz-2, Chai-1) cannot reliably distinguish correct antibody-antigen pairs from incorrect ones using internal confidence scores alone.
3. Confidence Calibration Analysis: Demonstrated that current confidence metrics are not calibrated to biological specificity and do not track structural improvements gained through sampling.
4. Resource Optimization Guidelines: Established that moderate sampling ($N \approx 10-25$) and diverse seed selection are more cost-effective than deep saturation sampling for improving structural quality, as deeper sampling does not improve specificity.
5. Open Data Release: Released ~1.8 million AI-generated complex structures and associated scripts to enable community benchmarking.

5. Significance and Implications

• For Drug Discovery: The study warns against relying solely on internal confidence scores (like ipTM) to filter antibody candidates. High-confidence predictions may be "hallucinations"—structurally plausible but biologically incorrect. This explains high experimental failure rates in AI-driven discovery pipelines.
• For Model Development: The results suggest that current architectures (relying on MSA-based trunk networks) may be "locked in" to early structural hypotheses. Future models need:
  • Training objectives that explicitly include hard negative controls (shuffled pairs).
  • Confidence metrics calibrated to specificity rather than just structural self-consistency.
  • Integration of dynamics (e.g., Molecular Dynamics) to filter out static, geometrically plausible but functionally implausible complexes.
• Operational Strategy: The authors propose a shift in workflow:
  • Use AI for generating plausible hypotheses (structural self-consistency).
  • Use ensemble consistency and negative controls (comparing candidates against shuffled decoys) for specificity filtering.
  • Avoid "brute force" deep sampling as a proxy for confidence; instead, prioritize diverse seed exploration.

In conclusion, while AI tools have mastered the generation of geometrically plausible antibody-antigen interfaces, they currently lack the ability to determine which interface is biologically correct. Bridging this gap requires moving beyond static confidence scores toward biophysically grounded, specificity-aware filtering strategies.