Calibration of in-frame indel variant effect predictors for clinical variant classification

This study addresses the clinical interpretation gap for in-frame indels by calibrating eight computational predictors using a high-confidence dataset and a statistical framework to establish ACMG/AMP-compliant score thresholds, demonstrating their measurable utility while highlighting their currently lower performance compared to missense variant tools.

The Gist

Imagine your DNA is a massive instruction manual for building a human being. Most of the time, the instructions are written in three-letter words (codons) that tell the body which amino acids (the building blocks of proteins) to use.

Sometimes, typos happen in this manual.

• Missense variants are like changing one letter in a word (e.g., "cat" becomes "bat"). We know a lot about these.
• Frameshift indels are like deleting a whole letter, which scrambles every word after it. These are usually catastrophic and easy to spot as "broken."
• In-frame indels (the focus of this paper) are like adding or deleting a whole word, but keeping the sentence structure intact. For example, changing "The cat sat" to "The big cat sat." The sentence still makes sense, but the meaning might be slightly off, or it might be completely fine.

The Problem:
Doctors and geneticists need to know if these "word additions" or "word deletions" are dangerous (pathogenic) or harmless (benign). While we have very smart computer programs to judge the single-letter typos (missense), the programs for judging these "word changes" (in-frame indels) have been a bit like uncalibrated scales. They give a number, but we didn't know exactly what that number meant in terms of "danger."

The Solution (The Paper's Mission):
The researchers in this paper decided to calibrate these computer scales. Think of it like taking a bunch of different thermometers, testing them against a known standard, and drawing new lines on them so that "70 degrees" actually means "room temperature" and not "hot soup."

Here is how they did it, using some everyday analogies:

1. Building the "Gold Standard" Library

To calibrate a scale, you need to know what "heavy" and "light" actually look like. The researchers gathered a huge library of genetic variants from public databases (ClinVar and gnomAD).

• They separated the "sick" variants (known to cause disease) from the "healthy" ones (found in healthy people).
• They made sure they didn't cheat by using the same data the computer programs were originally trained on. It's like giving a student a new test, not the practice questions they memorized, to see if they really learned the material.

2. The "Prior Probability" (The Baseline Guess)

Before looking at the computer's score, the researchers asked: "How likely is it that a random word change in a protein is actually dangerous?"

• They found that for deletions (removing words), about 4.6% are dangerous.
• For insertions (adding words), it's much rarer, only about 0.8%.
• Analogy: If you find a typo in a recipe, it's more likely to be a missing ingredient (deletion) that ruins the cake than an extra ingredient (insertion) that ruins it. This baseline guess helps the computer interpret the scores correctly.

3. Setting the "Traffic Light" Thresholds

The computer programs output a score (like a number from 0 to 100). The researchers used math to figure out exactly where to draw the lines to create "Traffic Lights":

• Green Light (Benign): The score is low enough to say, "This is almost certainly safe."
• Yellow Light (Uncertain): The score is in the middle. "We don't know yet."
• Red Light (Pathogenic): The score is high enough to say, "This is likely dangerous."

They created specific "Red Light" and "Green Light" numbers for 8 different computer programs (like CADD, VEST-Indel, PROVEAN, etc.).

4. The Results: Good News, But Not Perfect

• The Good News: All 8 programs could now be used in a clinical setting. They can provide evidence to help doctors decide if a patient's genetic variant is the cause of their illness.
• The Catch: These programs are still not as good as the ones for single-letter typos.
  • Analogy: The "single-letter" detectors are like high-tech metal detectors at an airport that catch almost everything. The "in-frame indel" detectors are like the old-school metal detectors; they catch the big stuff, but they miss some smaller threats or sometimes get confused.
  • Specifically, the programs were better at spotting dangerous deletions than dangerous insertions.

5. Why This Matters for Patients

Imagine a patient comes in with a rare genetic disease. The doctor finds a "word deletion" in their DNA but doesn't know if it's the culprit.

• Before this paper: The doctor looks at the computer score and says, "Hmm, the score is 15. Is that bad? I'm not sure."
• After this paper: The doctor looks at the score, checks the new "Traffic Light" chart, and says, "Ah, a score of 15 is a 'Moderate Red Light.' This counts as strong evidence that the variant is dangerous."

This helps doctors make faster, more accurate diagnoses, leading to better treatment plans for patients.

Summary in a Nutshell

The researchers took 8 different computer tools that guess if "word changes" in our DNA are bad, and they gave them a rigorous test. They created a new rulebook (calibration) so doctors can trust the scores these tools give. While the tools aren't perfect yet, they are now reliable enough to be used as part of the official process for diagnosing genetic diseases.

Technical Summary

1. Problem Statement

• The Gap: While computational predictors for missense variants (single nucleotide changes) are rigorously calibrated for clinical use under ACMG/AMP guidelines, tools for in-frame insertions and deletions (indels) remain understudied and uncalibrated.
• Clinical Challenge: In-frame indels (≤50 bp) preserve the reading frame but alter protein length. Their functional impact is often subtle and context-dependent, making them difficult to interpret compared to frameshift or nonsense variants.
• Limitations of Previous Work: Existing evaluations (e.g., Cannon et al., 2023) often suffered from training data contamination (using the same data for training and testing) and relied on default thresholds rather than thresholds calibrated to specific clinical evidence standards (e.g., "supporting," "moderate," or "strong" evidence).
• Goal: To establish a rigorous statistical framework to calibrate in-frame indel predictors, defining specific score thresholds that correspond to distinct levels of evidence for pathogenicity and benignity according to ACMG/AMP guidelines.

2. Methodology

The authors applied a Bayesian framework previously developed for missense variants (Pejaver et al., 2022) to in-frame indels.

• Datasets:

  • Calibration Set: 3,625 high-confidence in-frame indels from ClinVar (Dec 2023) in disease-associated genes (GenCC), filtered for non-conflicting classifications and low population frequency (gnomAD AF ≤ 1%).
  • Test Sets:
    • ClinVar 2025: An independent set of 1,131 variants to validate thresholds.
    • gnomAD: 26,014 rare in-frame indels used as a reference set for human variation to estimate prior probabilities.
    • Rare Genomes Project (RGP): 300 probands used to assess the real-world "evidence yield" per individual.
  • Training Data Filtering: For supervised tools, variants overlapping with the training data of specific tools were removed from the calibration set to prevent bias.

• Tools Evaluated: Eight diverse predictors were selected:

  • Evolutionary Conservation: PROVEAN.
  • Machine Learning: VEST-Indel, MutPred-Indel, FATHMM-indel, INDELpred.
  • Protein Language Models (pLMs): ESM1b, ProGen2.
  • General Score: CADD v1.7.
  • Note: Tools using allele frequency as a feature were excluded to avoid circularity.

• Calibration Framework:

  1. Prior Probability Estimation: Using the DistCurve algorithm on gnomAD data, the prior probability of pathogenicity was estimated.
     • Overall in-frame indels: 4.0%.
     • Deletions: 4.6% (similar to missense variants).
     • Insertions: 0.8% (significantly lower).
  2. Threshold Determination: Local posterior probabilities were calculated to map tool scores to ACMG/AMP evidence points (+1 to +4 for pathogenic, -1 to -4 for benign).
  3. Separate Calibration: Insertions and deletions were calibrated independently due to their distinct prior probabilities and performance characteristics.

3. Key Contributions

• First Clinical Calibration: This is the first study to systematically calibrate in-frame indel predictors against ACMG/AMP evidence standards, moving beyond default thresholds.
• Quantitative Evidence Mapping: The study provides specific score thresholds for eight tools that correspond to "supporting," "moderate," and "strong" evidence levels for both pathogenicity and benignity.
• Differentiation of Variant Types: It highlights the critical need to treat insertions and deletions separately, as they have different prior probabilities of pathogenicity and require different thresholds.
• Real-World Application: The study quantifies how many variants in a typical patient genome (RGP cohort) would receive actionable evidence from these tools.

4. Key Results

• Evidence Levels Achieved:
  • All eight tools reached at least supporting evidence (+1 or -1) for either pathogenicity or benignity.
  • Deletions: Generally achieved higher evidence levels than insertions. Several tools (MutPred-Indel, VEST-Indel, ESM1b, ProGen2, PROVEAN) reached +3 (moderate) evidence for pathogenicity. FATHMM-indel, INDELpred, and PROVEAN reached -4 (strong) evidence for benignity.
  • Insertions: Performance was lower due to the low prior probability (0.8%). No tool reached "strong" evidence (+4) for pathogenicity. The highest was +3 for VEST-Indel.
• Performance Disparities:
  • PROVEAN: Despite being an older, conservation-based method, it performed comparably to newer tools, underscoring the value of evolutionary data.
  • Protein Language Models (ESM1b, ProGen2): Showed imbalanced performance. They were effective at identifying pathogenic variants but struggled to assign benign evidence (scores clustered near zero for both benign and pathogenic).
  • Comparison to Missense: Indel predictors generally achieved lower maximum evidence levels than missense predictors (e.g., AlphaMissense, REVEL), which can reach "strong" evidence (+4) more consistently.
• Threshold Discrepancies: Calibrated thresholds often differed significantly from default tool thresholds. For example, FATHMM-indel's default threshold (0.5) was much lower than the calibrated +1 pathogenic threshold (0.961 for deletions), indicating that default settings would lead to overestimation of pathogenicity.
• Evidence Yield: In the RGP cohort, approximately 1 in 10 probands harbored at least one indel receiving +1 pathogenic evidence, and 1 in 25 received +2 or +3 evidence.

5. Significance and Recommendations

• Clinical Utility: The calibrated thresholds allow clinical laboratories to use computational tools as valid evidence (PP3/BP4 criteria) for in-frame indel classification, improving diagnostic rigor.
• Tool Selection:
  • The authors recommend selecting a single tool that achieves at least moderate pathogenic evidence (+2/+3) and supporting benign evidence (-1).
  • MutPred-Indel, VEST-Indel, and PROVEAN are highlighted as the most balanced options meeting these criteria.
  • Laboratories should consider using separate tools for insertions and deletions if one performs significantly better for a specific type.
• Cautionary Notes:
  • Conservatism: Due to the low prior probability of insertions, the thresholds are conservative, potentially underestimating the strength of benign evidence for insertions. This is intentional to avoid false positives in clinical settings.
  • Version Specificity: Thresholds are valid only for the specific versions of the tools tested; future updates may require re-calibration.
  • Data Scarcity: The study notes that current datasets are too small for length-based calibration (e.g., 1-aa vs. multi-aa indels), though this is a target for future work.

Conclusion:
This work bridges a critical gap in genomic medicine by providing the necessary statistical calibration to use in-frame indel predictors in clinical settings. While these tools currently lag behind missense predictors in maximum evidence strength, they offer measurable and actionable value for classifying variants of uncertain significance (VUS), thereby improving patient diagnostic outcomes.