Published benchmark AUROC does not predict generalisation: an independent evaluation of protein solubility predictors on native and heterologous E. coli proteins

Technical Summary: Published Benchmark AUROC Does Not Predict Generalisation

Problem Statement
Sequence-based predictors for recombinant protein solubility are widely used to triage expression targets in Escherichia coli. While deep-learning models frequently report strong headline benchmark AUROCs (Area Under the Receiver Operating Characteristic curve), it remains unclear whether these reported metrics predict how a model will rank on an independent data distribution. Furthermore, the variance between two state-of-the-art (SOTA) models when evaluated on the exact same proteins is rarely tested directly. The paper addresses whether a model's published benchmark AUROC serves as a reliable guide for its performance on new distributions and how much performance varies between top-tier models within the same class.

Methodology
The study conducted an independent evaluation of protein solubility predictors across two distinct E. coli distributions:

1. Native Distribution (eSOL): 3,131 native E. coli K-12 ORFs measured in a chaperone-free reconstituted translation system.
2. Heterologous Distribution (SoluProt): 3,100 balanced heterologous constructs, representing a harder, more realistic recombinant expression triage scenario.

Predictors Evaluated:

• Deep Models: RP3Net (ESM-2 650M based), NetSolP (ESM1b based), and PLM_Sol (ProtT5 based).
• Baselines: Two dataset-naive composition scores: the published Solubility-Weighted Index (SWI) and an in-house deterministic linear heuristic (used only as a corroborating baseline).
• Calibrated Model: ProteinSol (calibrated on native eSOL, evaluated only on heterologous data).

Leakage Control:
A critical component of the methodology was rigorous per-model train/test leakage control using MMseqs2.

• Near-duplicate screening: Sequences with ≥90% identity over ≥80% query coverage were removed from test sets if they appeared in a model's public training data.
• Homology screening: A secondary screen clustered sequences at ≤30% identity to rule out residual homology artifacts.
• Specific Corrections: Notably, 35.7% of the SoluProt test set overlapped with PLM_Sol's training data (UESolDS), and 4.06% of eSOL overlapped with NetSolP's training data. These overlaps were removed to ensure fair evaluation.

Key Results
The performance of solubility predictors was found to be highly dependent on both the specific model and the target distribution, rather than the model class (e.g., "deep learning").

1. On Native Proteins (eSOL):

   • Two SOTA deep models occupied opposite extremes. NetSolP was the best predictor (AUROC 0.792), while RP3Net was the worst (AUROC 0.709).
   • The gap between these two deep models (0.083) was larger than the gap between any model and a simple baseline.
   • RP3Net, despite a reported benchmark AUROC of 0.83 on its own prospective set, was significantly outperformed by the simple SWI score (0.745, p=0.006) and tied with the naive heuristic.
   • NetSolP significantly outperformed both baselines.
   • This ordering (NetSolP > SWI > RP3Net) persisted even after homology-level deduplication.

2. On Heterologous Proteins (SoluProt):

   • The trend inverted. The two leading deep models (NetSolP and RP3Net) clustered at the top and were statistically indistinguishable (NetSolP 0.633 vs. RP3Net 0.619, p=0.23).
   • Both deep models numerically led the simple baselines, though all predictors fell within a narrow, low-discrimination band (0.58–0.63).
   • PLM_Sol demonstrated a severe distribution mismatch: despite reproducing its own benchmark AUROC of ~0.83, it collapsed to 0.564 on the leak-free SoluProt set, performing below every baseline and indistinguishable from a naive composition score. This collapse was confirmed to be a genuine distribution effect, not an artifact of training data overlap, as performance was identical on overlapping and non-overlapping proteins.
   • ProteinSol, calibrated on native eSOL, collapsed to near-chance (0.542) on the heterologous set.

Key Contributions

• Demonstration of Distribution Dependence: The study provides empirical evidence that a model's headline benchmark AUROC does not predict its ranking on an independent distribution. A model can be SOTA on one distribution and perform worse than a simple heuristic on another.
• Variance Between SOTA Models: The authors show that the performance gap between two equally "state-of-the-art" deep models can exceed the gap between a deep model and a simple interpretable baseline.
• Role of Baselines: The paper establishes that dataset-naive composition baselines serve as a useful "sanity floor." A SOTA model falling below a simple baseline on a target distribution signals a generalization gap that headline numbers conceal.
• Rigorous Leakage Control: The study highlights the prevalence of hidden train/test overlaps (e.g., 35.7% for PLM_Sol) and demonstrates that even after removing these, distribution mismatches can cause significant performance drops.

Significance and Claims
The paper frames its findings not as a criticism of specific models (which may be well-suited to their intended design domains) but as a caution regarding the selection of solubility predictors based solely on reported benchmark numbers.

• Evaluation Context: Solubility predictors should be evaluated on the specific distribution of interest (e.g., native vs. heterologous) rather than trusted by a single headline number.
• Design Domain Mismatch: RP3Net's low ranking on native proteins is interpreted as a reflection of distribution mismatch (evaluating a recombinant-focused model on native data) rather than a model deficiency. Conversely, ProteinSol's failure on heterologous data mirrors this effect in the opposite direction.
• Practical Recommendation: The authors recommend that future evaluations of solubility predictors explicitly include dataset-naive baselines to provide a reference floor for generalization performance.

The study concludes that "model choice, not model class, dominates" performance on native proteins, and that the field's shorthand "solubility predictor" spans distinct targets (intrinsic thermodynamic solubility vs. recombinant expression outcomes) which drives the observed distribution dependence.