Decomposing the Generalization Gap in PROTAC Activity Prediction: Variance Attribution and the Inter-Laboratory Ceiling

This paper decomposes the generalization gap in PROTAC activity prediction, identifying inter-laboratory measurement variance as the dominant limiting factor that creates a performance ceiling around 0.67 AUROC, while introducing the PROTAC-Bench dataset and calibration protocols to address these evaluation challenges.

The Gist

The Big Picture: The "New Student" Problem

Imagine you are a teacher trying to test how well a student understands a subject.

• The Old Way (Random Split): You give the student a test where 80% of the questions are about topics they've already studied in class, just with slightly different numbers. They score 90%. You think, "Great, they're a genius!"
• The New Way (Leave-One-Target-Out): You give the student a test on a completely new topic they have never seen before. Suddenly, they score only 67%.

This paper investigates PROTACs (a type of drug that acts like a "molecular trash can" to destroy bad proteins). Researchers had built AI models that scored 85–91% on the "Old Way" tests but dropped to about 67% on the "New Way" tests.

The big question was: Is the AI just bad at learning new things? Or is the test itself flawed because the data is messy?

The Investigation: It's Not the AI, It's the Noise

The authors acted like detectives to figure out why the score dropped so much. They didn't just blame the AI; they looked at the data itself.

The Analogy: The "Noisy Room" Experiment
Imagine you are trying to hear a friend whisper a secret in a quiet room. You hear it perfectly (Score: 90%). Now, imagine you try to hear that same whisper in a room where three different people are shouting different versions of the story, and the microphone is broken. You can't hear clearly anymore (Score: 67%).

The paper concludes that the drop in score isn't because the AI is "stupid." It's because the "room" (the scientific data) is incredibly noisy.

• The Main Culprit: Different labs measure the same drug against the same protein and get different results. It's like three different weather stations in the same city reporting three different temperatures for the same hour.
• The Finding: The authors calculated that this "lab-to-lab noise" accounts for about 80% of the reason the AI scores drop. The AI can't predict what it can't measure consistently.

The "Ceiling" Effect

The researchers tested many different types of AI models, from simple ones to massive, complex ones (like the huge ESM-2 protein language models).

• The Result: No matter how smart or complex the AI was, they all hit the same "ceiling" of about 67%.
• The Analogy: Imagine trying to see a mountain peak through thick fog. You can use a better telescope (a smarter AI), but if the fog (the noisy data) is too thick, you still can't see the peak any clearer. The fog is the limit, not the telescope.

They also tried to "tweak" the AI by changing its settings thousands of times (Hyperparameter Optimization).

• The Trap: When they picked the "best" settings based on just one test run, the AI looked amazing. But when they tested it again with different random seeds (like running the test on a different day), the score crashed. This is a classic case of "overfitting" or getting lucky with a specific test setup.

The Solution: The "Tutor" Approach

If the AI can't learn from the whole class because the class is too noisy, what can it do? The authors found a clever workaround called Few-Shot Calibration.

• The Analogy: Instead of trying to learn the rules of a new game by reading a thick manual (the whole dataset), the AI asks a local expert for just 5 examples of how the game is played in this specific town.
• The Result: By giving the AI just 5 specific examples for each new target protein (a "few-shot" approach) and adding some extra safety data (ADMET features), the score jumped from 66.8% to 70.5%.
• Why it works: It's like the AI realizing, "Oh, in this specific lab, they measure things slightly differently. Let me adjust my guess based on these 5 local examples."

The Toolkit: PROTAC-Bench

The authors didn't just solve the problem; they built a new playground for everyone else to use.

• They created PROTAC-Bench, a curated collection of 10,748 drug measurements.
• Unlike other databases that are huge but messy, this one is smaller but deep. It focuses on having many measurements for the same targets so scientists can actually study the "noise" and the "variance."
• They also released the code and the "variance decomposition" framework, which is a method for other scientists to use to figure out if their own AI problems are caused by bad models or just bad data.

Summary of Claims (What the paper actually says)

1. The Gap is Real: There is a huge difference between how well AI predicts drugs on familiar targets vs. new targets.
2. The Cause is Data, Not AI: The main reason for the drop in performance is inter-laboratory measurement variance (different labs getting different results for the same thing), not a failure of the AI to learn.
3. The Ceiling is Hard: Even the biggest, most complex AI models cannot break past a ~67% score on new targets because the data itself is inconsistent.
4. Hyperparameter Tuning is Tricky: Trying to find the "perfect" settings on a single test run leads to false confidence. The AI needs to be tested multiple times to be reliable.
5. Small Adjustments Help: Using a "few-shot" approach (learning from just 5 examples per new target) can squeeze a little extra performance out of the system, pushing the score to about 70.5%.
6. Calibration Matters: The AI's raw confidence scores are often wrong (overconfident). Using a simple math trick (Platt scaling) fixes this, making the probabilities more trustworthy.

In short: The AI isn't broken; the data is just messy. Until scientists can measure drug activity more consistently across different labs, the AI will hit a wall. The best strategy right now is to use simple, robust models and give them a few local examples to help them adjust to the specific "noise" of the new target.

Technical Summary

Technical Summary: Decomposing the Generalization Gap in PROTAC Activity Prediction

Problem Statement

Machine learning predictors for biochemical activity, specifically for PROTACs (proteolysis-targeting chimeras), exhibit a significant "generalization gap." While models report high AUROC values (0.85–0.91) under random-split cross-validation, performance drops precipitously to approximately 0.67 under leave-one-target-out (LOTO) protocols. This gap is well-documented but previously undecomposed. The central question is whether this performance collapse reflects a fundamental inability of models to extrapolate to novel targets (learning failure) or if it is driven by irreducible noise in the data, specifically inter-laboratory measurement variance and label heterogeneity.

Methodology

The authors introduce PROTAC-Bench, a curated benchmark of 10,748 degradation measurements across 173 targets, prioritizing within-target measurement density and multi-publication replicate structures over raw corpus size. The study employs a variance-decomposition framework to attribute the generalization gap to specific components:

1. Evaluation Protocols: The study compares random-split cross-validation, scaffold-split, and the canonical LOTO protocol. It also introduces a "within-target cross-lab" evaluation, holding out one publication's data for targets with multiple publications to isolate inter-laboratory variance.
2. Architecture and Scale Invariance: The authors evaluate eight distinct architectures (including DeepPROTACs, DegradeMaster, PROTAC-STAN, and standard baselines like Random Forest with Morgan fingerprints) and scale protein language models (PLMs) from 8M to 3B parameters (ESM-2).
3. Hyperparameter Optimization (HPO): A 21-dimensional HPO search across 2,000 trials is conducted to test if aggressive optimization can break the performance ceiling. The results are analyzed against the Bailey-López de Prado closed-form prediction for selection bias.
4. Variance Decomposition: The gap is triangulated using four independent bounds:
   • Inter-laboratory measurement variance (via cross-lab cascade).
   • Binarization-threshold choice across four labelling schemes.
   • Cross-DOI conflict removal.
   • Residual distributional shift.
5. Few-Shot Calibration: The study tests whether per-target retraining with few-shot data ($k=5$) combined with ADMET features can recover performance lost to measurement noise.

Key Results

1. The Universal Performance Ceiling

Across all eight evaluated architectures and ESM-2 PLM scales, LOTO performance plateaus within a narrow band of 0.668 to 0.678 AUROC.

• PLM Scaling: Larger PLMs (up to 3B parameters) do not improve LOTO performance; in fact, performance is non-monotonic, peaking at 150M parameters (0.691) and regressing at larger scales. This suggests larger models act as "accidental taxonomists," identifying target families they have seen verbatim during pretraining rather than learning generalizable degradation mechanisms.
• HPO Regression: The top-ranked configuration from 2,000 HPO trials (single-seed objective 0.764) regressed to 0.603 under multi-seed validation, a drop of 0.161 AUROC. This matches the theoretical prediction for selection bias (Bailey-López de Prado), indicating that single-seed HPO overfits to noise.

2. Variance Attribution

The study decomposes the random-split-to-LOTO gap (approx. 0.18–0.20 AUROC) and identifies inter-laboratory measurement variance as the dominant component:

• Inter-Laboratory Bound: A within-target cross-lab cascade (Random-CV 0.802 $\to$ Cross-Lab 0.678 $\to$ LOTO 0.653) anchors the inter-laboratory contribution at 0.124 AUROC.
• Empirical Anchor: Identical compounds measured across different publications show a median 3.7-fold DC50 variation, translating to an equivalent label-flip rate of ~23% under synthetic noise calibration.
• Other Factors: Binarization choice contributes ~0.05 AUROC, while other factors (conflict removal, residual shift) are negligible. The gap is primarily a measurement-variance issue, not a learning-failure issue.

3. Recovery via Few-Shot Calibration

While the "ceiling" is fixed for zero-shot prediction, the study demonstrates that few-shot per-target retraining can recover a portion of the lost performance:

• Combining Morgan fingerprints, ADMET features, and stratified $k=5$ retraining lifts the 65-target LOTO AUROC from 0.668 to 0.7050.
• This approach outperforms meta-learning baselines (MAML, ProtoNet) and suggests that the inter-laboratory variance component is recoverable if a small number of target-specific measurements are available.
• Calibration: Post-hoc Platt scaling recovers raw output Expected Calibration Error (ECE) from 0.150 to 0.031, bringing it below the 0.05 well-calibrated threshold.

4. Structural and Geometric Approaches

Evaluation of 3D structural features (EGNN, Boltz-2, AlphaFold-predicted pockets) shows they contribute at most 0.013 AUROC beyond 2D Morgan fingerprints. Pocket-shuffle controls (randomly permuting pocket residues) degrade performance by less than 0.013 AUROC, indicating that current structural prediction tools do not provide a signal strong enough to overcome the inter-laboratory noise floor.

Significance and Claims

The paper claims that the apparent generalization gap in PROTAC activity prediction is not a failure of model architecture or capacity but a reflection of the inter-laboratory reproducibility floor of the underlying experimental data.

• Methodological Contribution: The work provides a variance-decomposition framework and a benchmark (PROTAC-Bench) that shifts the focus from "better models" to "better evaluation of measurement noise." It argues that the operational ceiling for held-out-target prediction is currently bounded by data quality (0.668–0.678 AUROC) rather than algorithmic limitations.
• Practical Implication: For deployment, the paper recommends a three-tier protocol: a Morgan baseline (0.668), augmented by ADMET features (0.687), and further improved by few-shot per-target retraining (0.705). It emphasizes that predictors should be used for "active-versus-inactive enrichment" rather than fine-grained potency ranking, as the data noise precludes precise ranking.
• Generalizability: The authors posit that this decomposition methodology—attributing generalization gaps to measurement variance rather than learning failure—serves as a template for other small-data therapeutic settings where inter-laboratory or inter-site variance is a dominant source of noise.

The authors explicitly state that no model in this work exceeds the predictive accuracy achievable by domain experts using standard cheminformatics tools under current data constraints, and that the primary contribution is methodological transparency regarding evaluation protocols and variance attribution.