Two Comparators May Be All We Need

Technical Summary: Two Comparators May Be All We Need

Problem Statement
In drug discovery, a compound within a cell interacts with a spectrum of proteins across multiple families simultaneously, yet standard screening and modeling typically interrogate one target at a time. While small molecules are inherently promiscuous, and off-target activities often drive safety attrition, most predictive models score one protein and one compound individually. Consequently, deriving selectivity or preference requires subtracting two separate affinity predictions, a process that struggles with the heterogeneity of public potency data (which can vary by nearly a log unit across different assays and laboratories) and fails to directly answer the comparative questions that drive decision-making: "Which of these two targets does a compound prefer?" and "Which of these two compounds does a target prefer?"

Methodology
The authors constructed two reciprocal, structure-free models trained exclusively on public data from ChEMBL 37, covering 2,279 human proteins across 34 protein families. Neither model utilizes protein structures, binding sites, docked poses, or conformational analysis.

• Data Curation: The dataset comprises 1,263,626 measurements (Ki, IC50, Kd, EC50, Kb) for 794,638 compounds. Data were restricted to human single-protein targets with level-2 ChEMBL class labels. Readings were converted to pActivity (negative log of concentration), and ties were discarded.
• Feature Representation:
  • Proteins: Represented as 480-dimensional residue-wise mean vectors derived from the ESM2-t12-35M protein language model.
  • Compounds: Represented as 1,024-value radius-2 Morgan count fingerprints plus 14 whole-molecule descriptors (e.g., molecular weight, atom counts).
• Model Architecture: Both comparators are Random Forests (300 trees each) that take a pairwise input format.
  • Target Comparator: Inputs are (Sequence A, Compound, Sequence B). It predicts which of the two targets the compound prefers. This model handles two scenarios: comparisons across different families and comparisons within a single family.
  • Compound Comparator: Inputs are (Compound A, Sequence, Compound B). It predicts which of the two compounds a single target prefers.
• Training Strategy: To ensure strict generalization, the models were evaluated on compound-disjoint holdouts. For the compound comparator, a comparison was only included in the test set if both compounds were unseen during training; comparisons straddling the split were discarded. Every comparison was entered twice with swapped positions and inverted labels to enforce symmetry.
• Output: The models return a probability (preference strength) ranging from 0.50 (undecided) to 1.00 (certain), rather than a predicted affinity value.

Key Results
The models were evaluated on held-out comparisons where neither the target nor the compound (depending on the comparator) appeared in the training set.

• Target Preference (Across Families): On 8,689 held-out comparisons between targets of different families, the model correctly identified the preferred target 75% of the time. Accuracy rose to 94% for the most confident predictions (strength ≥ 0.80).
• Target Preference (Within Family): On 32,738 comparisons between targets of the same family, accuracy was 78%, rising to 95% for high-confidence calls. Kinase families presented the most difficulty (73% accuracy), likely due to high structural similarity among kinases.
• Compound Preference: On 65,725 held-out comparisons where neither compound was seen during training, the model correctly identified the preferred compound 71% of the time. This accuracy reached 96% for the most confident predictions.
• Impact of Difference Magnitude: Accuracy is strongly correlated with the magnitude of the true activity difference. When the two measurements differed by more than two log units, accuracy ranged from 90% to 92%. When they were within half a log unit, accuracy dropped to near-chance levels (0.57–0.64).
• Coverage Limitations: The performance ceiling for cross-family target comparisons is limited by the available data record, not the model architecture. Only 4.6% of classified compounds in the database have measurements across more than one protein family.

Significance and Claims
The paper posits that direct pairwise comparison is a more practical and robust approach than predicting absolute affinity values for prioritization workflows. The authors claim that these two comparators provide a complete set of ranking tools for a typical drug discovery cascade: the compound comparator helps decide what to synthesize or buy next, while the target comparator helps decide what to counter-screen against.

Key contributions include:

1. Breadth: Extending pairwise comparison beyond single families (like kinases or GPCRs) to cover 34 diverse protein families.
2. Structure-Free Prediction: Demonstrating that sequence and 2D chemical descriptors alone can effectively rank targets and compounds without structural data.
3. Strict Evaluation: Establishing a rigorous benchmark using compound-disjoint splits, ensuring the models generalize to unseen chemistry.
4. Operational Utility: The models output a preference probability that correlates with accuracy, allowing users to filter for high-confidence decisions. The authors note that while the models rank compounds and targets well within the represented chemical and target space, they do not estimate potency and cannot predict toxicity or safety outcomes directly.

The work is presented as a foundational step ("Family Foundation Model") that can be extended by users with their own proprietary data, as the Random Forest architecture allows for the concatenation of new trees without retraining the entire model. The models and code are released under open licenses to facilitate community adoption and extension.