Disagreement among variant effect predictors guides experimental prioritization of target proteins

This study demonstrates that because agreement among computational variant effect predictors does not correlate with their accuracy against experimental data, prioritizing proteins with high inter-predictor disagreement is an effective strategy for selecting targets for resource-intensive experimental characterization to maximize informational value.

The Gist

The Big Picture: The "Weather Forecast" Problem

Imagine you are trying to predict the weather for a specific town. You have 71 different weather forecasters (these are the Variant Effect Predictors, or VEPs). Some use satellites, some use ground sensors, and some use complex AI models.

Usually, if 70 out of 71 forecasters say, "It's going to rain," you can be pretty confident it will rain. You trust the consensus.

But what if the forecasters disagree?

• Forecaster A says: "Heavy rain!"
• Forecaster B says: "Sunny with a chance of snow."
• Forecaster C says: "It's going to be a tornado."

The old way of thinking: "Oh, the forecasters are confused. We probably don't need to check the sky; the situation is too messy to understand."

This paper's new idea: "Wait a minute! If the experts are arguing this much, it means we don't actually understand the weather yet. This disagreement is a giant red flag telling us: 'Go outside and look at the sky yourself!'"

The Core Discovery: Consensus $\neq$ Truth

The researchers tested this idea by comparing the computer forecasts (VEPs) against real, physical experiments (called MAVEs). Think of MAVEs as sending a drone up into the sky to actually measure the wind and rain.

The Shocking Result:
They found that agreement among the computers did not guarantee accuracy.

• Sometimes, all 70 computers agreed perfectly, but the drone data showed they were all wrong. (They were all using the same bad logic).
• Sometimes, the computers were screaming at each other, but the drone data showed that the "disagreement" was actually highlighting a complex, real phenomenon that the computers just couldn't agree on how to describe.

The Analogy:
Imagine a group of chefs trying to guess the ingredients in a secret soup.

• Scenario A: All chefs guess "Chicken and Carrots." They are in total agreement. But when they taste the soup, it's actually "Spicy Beef." They were all wrong because they all assumed the same thing.
• Scenario B: One chef says "Beef," another says "Lamb," and a third says "Fish." They are in total disagreement. This disagreement tells the head chef: "We don't know what's in here! Let's go taste it (do an experiment) to find out."

Why Do the Computers Disagree?

The researchers looked at why the computers argued. They found that the computers get confused in specific situations:

1. The "Messy Room" Effect (Disordered Proteins):
   Some proteins are like a neatly folded suit (structured and rigid). The computers can easily predict how a stain (mutation) will ruin the suit.
   Other proteins are like a pile of tangled headphones (disordered and flexible). The computers can't figure out how a knot in the headphones affects the sound. Because the "rules" of physics are fuzzy here, the computers give wildly different answers.

2. The "Foreign Language" Effect:
   Some proteins come from viruses or ancient genetic elements. The computers were trained mostly on "human" proteins. When they see a "foreign" protein, they don't have the right dictionary, so they guess wildly.

3. The "Crowded Party" Effect:
   Some proteins are part of huge, complex machines. If you change one tiny part, it might break the whole machine. The computers usually look at the protein in isolation, so they miss the big picture.

The New Strategy: Follow the Arguments

Because the computers are often wrong even when they agree, the authors propose a new strategy for scientists:

Don't prioritize the proteins where everyone agrees. (We probably already know those).
Prioritize the proteins where the computers are fighting.

If the computers are arguing, it means:

1. We are missing a piece of the puzzle.
2. This is the perfect place to spend money and time on real experiments (MAVEs).

The "Funnel" for Choosing What to Test

The paper suggests a step-by-step filter to find the best proteins to study in the lab:

1. Step 1: Find the Arguers. Look for proteins where the 10 best computer programs disagree the most.
2. Step 2: Check the Blueprint. Make sure the protein isn't just "messy" (disordered). If it's messy, experiments are hard to do. We want proteins that look structured (like a neat suit) but still confuse the computers. This means the computers are missing a specific biological rule, not just struggling with a messy shape.
3. Step 3: Check the Importance. Is this protein important for human health? Does it have many "Variants of Uncertain Significance" (VUS)—basically, genetic typos that doctors don't know if they are dangerous or safe?

Why This Matters for You

If you or a family member has a genetic mutation that doctors can't explain (a "Variant of Uncertain Significance"), this research offers hope.

Currently, doctors might say, "The computers disagree, so we can't tell you if this is dangerous."
This paper says: "That disagreement is the clue!"

It tells researchers: "Go test this specific gene in the lab. The fact that the computers are confused means this is a high-value target. Solving this will help us understand the disease and give clear answers to patients."

Summary in One Sentence

When computer models argue about how a genetic mutation works, it's not a sign that the problem is unsolvable; it's a sign that we need to stop guessing and start experimenting, because that's where the most important discoveries are hiding.

Technical Summary

1. Problem Statement

Interpreting the functional consequences of genetic variants, particularly rare missense variants, is a critical challenge in human genetics. While Variant Effect Predictors (VEPs) offer scalable computational predictions and Multiplexed Assays of Variant Effects (MAVEs) provide high-resolution empirical data, a significant gap exists:

• MAVEs are resource-intensive and cannot yet be applied exhaustively across the entire human proteome.
• VEPs often exhibit inconsistent performance, particularly in complex biological contexts (e.g., intrinsically disordered regions).
• The Core Question: Does consensus among different VEPs reliably indicate biological truth (i.e., agreement with experimental MAVE data)? Or do regions of VEP disagreement highlight "blind spots" where computational models fail and experimental investigation is most needed?

Current strategies often prioritize proteins where predictors agree, assuming consensus equals accuracy. This study challenges that assumption, hypothesizing that inter-VEP disagreement may better identify proteins where experimental mapping will yield the highest informational return.

2. Methodology

The authors conducted a large-scale, multi-layered analysis involving over 13,000 human proteins.

• Dataset Construction:
  • Aggregated predictions from 71 distinct VEPs (including tools like AlphaMissense, ESM-1v, GEMME, and MetaRNN) from public resources (dbNSFP, APIs, local runs).
  • Created a harmonized dataset of 6.8 million missense SNVs across 3,846 proteins where all predictors provided scores.
  • Selected a representative subset of 10 diverse VEPs for deep analysis, balancing methodological diversity (embedding-based, alignment-driven, clinically trained) and experimental accuracy.
• Concordance Quantification:
  • Calculated Spearman rank correlations between all pairs of the 10 selected VEPs for each protein.
  • Defined Inter-VEP Concordance as the median pairwise correlation for a given protein.
  • Stratified proteins into three tiers: Low, Intermediate, and High agreement.
• Feature Analysis:
  • Structural Context: Mapped proteins to AlphaFold2 models to extract residue-level features: model confidence (pLDDT), solvent accessibility (rASA), weighted contact number (WCN), and secondary structure.
  • Functional Classification: Annotated proteins using PANTHER database classes.
  • Population Genetics: Integrated gnomAD constraint metrics (pLI, pRec, missense Z-score) and heterozygous selection coefficients ($s_{het}$).
• Experimental Benchmarking:
  • Compared the 10 VEPs against a curated set of 36 high-quality MAVE datasets (reduced to 29 proteins after filtering for variant overlap).
  • Tested the correlation between Inter-VEP Concordance and VEP-MAVE Prediction Accuracy.

3. Key Contributions & Results

A. Methodological Lineages Drive Agreement

Global clustering of 71 VEPs revealed that agreement is not random but structured by methodological lineage. Models using similar architectures (e.g., protein language models like AlphaMissense and ESM-1v) cluster together, as do alignment-based methods (GEMME, ESCOTT). This suggests that "consensus" often reflects shared inductive biases rather than independent validation of biological truth.

B. Structural and Functional Determinants of Disagreement

Inter-VEP concordance varies significantly across the proteome and is strongly associated with specific biological features:

• Protein Classes: Low agreement is enriched in viral/transposable element proteins, defense/immunity proteins, storage proteins, and structural proteins. These classes often evolve rapidly, lack deep conservation, or rely on quaternary structures difficult to model with monomer-centric predictors.
• Residue Context: Low concordance is strongly correlated with intrinsically disordered regions (IDRs) and flexible surface loops.
  • Low Agreement: Characterized by low AlphaFold2 pLDDT (<70), high solvent exposure, and high coil content.
  • High Agreement: Characterized by high pLDDT, buried residues, and regular secondary structures (helices/strands).
• Genetic Constraints: While gene-level metrics (pLI) showed weak correlation with concordance, the missense Z-score (indicating depletion of missense variants) was higher in high-concordance genes, suggesting that strong evolutionary conservation in ordered domains drives predictor agreement.

C. The Decoupling of Consensus and Accuracy (Key Finding)

The study found no correlation between Inter-VEP Concordance and agreement with experimental MAVE data.

• High Consensus $\neq$ High Accuracy: Many proteins with high inter-VEP agreement showed poor correlation with MAVE results (e.g., HMGCR). This indicates a "false consensus" where models share biases but fail to capture the true functional mechanism.
• Disagreement as a Signal: Proteins with low inter-VEP agreement were just as likely to have high experimental validity as those with high agreement.
• Implication: Consensus does not guarantee correctness; it often reflects shared priors (conservation, physicochemical properties) that may be insufficient for specific assay contexts.

D. A New Prioritization Strategy

Based on these findings, the authors propose a prioritization funnel to optimize experimental resource allocation:

1. Select Low-Concordance Targets: Start with proteins where predictors disagree (Low Agreement tier).
2. Filter for Structural Tractability: Exclude proteins where disagreement is solely due to disorder (low pLDDT). Prioritize low-concordance proteins that are structurally well-resolved (high pLDDT). These represent "true" blind spots where current models fail despite having clear structural data.
3. Refine by Constraint and Clinical Relevance: Further filter for genes under strong selective constraint and those with high burdens of Variants of Uncertain Significance (VUS) in ClinVar.

• Example: The protein OSCP1 was identified as a high-value candidate: it has low VEP agreement, high structural confidence (pLDDT ~92), and a high burden of VUS, making it an ideal target for MAVEs.

4. Significance and Impact

• Paradigm Shift: The paper challenges the prevailing heuristic that "agreement equals accuracy." It demonstrates that computational consensus can be a "false positive" for biological truth.
• Resource Optimization: By targeting proteins where predictors disagree (specifically those that are structurally tractable), researchers can maximize the informational yield of expensive MAVE experiments, uncovering mechanisms that current models miss.
• Clinical Utility: This strategy directly addresses the problem of Variants of Uncertain Significance (VUS). Prioritizing targets with high disagreement in clinically relevant genes can resolve uncertainty faster than random or consensus-based selection.
• Model Improvement: Identifying specific contexts (e.g., surface loops in ordered proteins) where models diverge provides clear targets for the next generation of VEPs to improve training data and feature engineering.

In summary, the authors provide a data-driven framework to convert computational uncertainty into experimental action, arguing that the "blind spots" of current AI models are the most valuable targets for empirical investigation.