HonestAffinity: Leak-Aware Evaluation of Protein and Pocket Priors for Binding Affinity Prediction

Technical Summary: HonestAffinity

Problem Statement
Sequence-based deep learning offers a scalable alternative to structure-based scoring for protein–ligand binding affinity prediction, enabling library-scale screening for targets without solved structures. However, progress in this field is difficult to interpret due to two primary sources of uncertainty:

1. Protein Representation: It remains unclear whether pretrained protein language models (PLMs) like ESM-2, which encode evolutionary and structural cues, improve prediction under strict "no-leak" evaluation compared to learned residue embeddings.
2. Evaluation Splits: Standard PDBbind splits (general/refined/core) often leak protein–ligand similarity classes between training and testing sets. Models overfitting to these classes may report strong performance on canonical benchmarks (e.g., CASF-2016) while failing on dissimilar targets. The leak-proof LP-PDBBind reorganization addresses this but has seen limited adoption in sequence-only studies.

Methodology
The authors introduce HonestAffinity, a compact 1D-input affinity predictor designed as a controlled testbed to isolate the effects of two common architectural priors:

• Frozen ESM-2 Embeddings: 1280-dimensional per-residue embeddings from a 650M parameter model.
• Learned Pocket-Position Marker: A binary mask indicating pocket residues.

The architecture utilizes a multi-scale 1D-convolution encoder, a residual block, a single Transformer layer, and a matrix-product compatibility score followed by an MLP head. The ligand branch processes SMILES strings using a similar template.

Three specific variants of the same template are evaluated:

1. HonestAffinity-Pocket: Uses ESM-2 + pocket marker.
2. HonestAffinity-NoPocket: Uses ESM-2 only (no pocket annotation).
3. HonestAffinity-Pocket-NoESM: Uses a learned 21-vocabulary residue embedding + pocket marker (replacing ESM-2).

Experimental Protocol

• Training: All variants train on 11,513 LP-PDBBind complexes in approximately 3 GPU-hours on a single Tesla V100. ESM-2 weights are frozen.
• Evaluation: Models are benchmarked against five reproduced baselines (DeepDTA, Pafnucy, DeepDTAF, DEAttentionDTA, Multi-scale matrix-product) across:
  • LP-PDBBind 3-tier hold-out (test_cl1, test_cl2, test_cl3), which progressively exclude similar protein/ligand pairs.
  • CASF-2016 (canonical scoring-power benchmark).
  • A CASF-2016 non-train subset (161 entries not in the LP training split).
• Reporting: Every method reports mean ± standard deviation over three random seeds. No single "best run" is hand-picked; model selection is based on validation RMSE within each seed.

Key Results
The central finding is a split-conditioned reversal, where architectural components that improve performance on familiar/canonical splits degrade performance on strict no-leak tiers:

1. Pocket Marker Effect:

   • Canonical Splits (val, CASF-2016): Adding the pocket marker improves Pearson R (e.g., +0.042 on validation).
   • Strict LP Splits (test_cl1–cl3): Adding the pocket marker reduces Pearson R (e.g., -0.018 on test_cl1). The variant without the marker (HonestAffinity-NoPocket) outperforms the pocket-equipped variant on strict tiers.

2. ESM-2 Effect:

   • Canonical Splits: Using frozen ESM-2 embeddings improves Pearson R (e.g., +0.019 on validation).
   • Strict LP Splits: Using ESM-2 reduces Pearson R (e.g., -0.024 on test_cl1). The variant replacing ESM-2 with learned embeddings (HonestAffinity-Pocket-NoESM) achieves the highest mean Pearson R on all three strict LP tiers.

3. Overall Performance:

   • HonestAffinity-Pocket achieves the best mean Pearson R on validation, CASF-2016, and the CASF-2016 non-train subset.
   • HonestAffinity-Pocket-NoESM achieves the best mean Pearson R on every strict LP no-leak tier (test_cl1–cl3), outperforming both ESM-equipped variants and the best baseline (DeepDTAF) on test_cl2 and test_cl3.

Significance and Claims
The paper argues that binding-affinity models should not be judged by a single global default configuration. Instead, the authors propose:

• Paired Ablations: Reporting both canonical and leak-proof evaluations is a necessary sanity check. Single-regime reporting conceals the fact that components like pocket markers and PLM features can become counterproductive when the test distribution is dissimilar from the training distribution.
• Deployment-Regime Matching: The optimal model variant depends on the deployment context:
  • Use HonestAffinity-Pocket for familiar/annotated targets where training distribution overlap is high.
  • Use HonestAffinity-Pocket-NoESM for strict LP-style targets with pocket annotations.
  • Use HonestAffinity-NoPocket for targets without pocket annotations.
• Mechanism: The authors suggest that PLM features and pocket markers act as inductive biases toward training-distribution structures (family signatures, pocket geometry). These biases are informative when the test target overlaps the training family but become misleading or uninformative when strict similarity filtering removes that overlap.

The paper concludes that 1D-input PLM-augmented scoring and structure-aware scoring are complementary regimes rather than competing ones, with HonestAffinity offering a fast (∼10 ms inference), low-resource alternative for targets lacking high-quality 3D poses.