Clinical validation of large-scale functional assays: insights from 2,120 gene-truthset-assay evaluations

Technical Summary: Clinical Validation of Large-Scale Functional Assays

Problem Statement
The clinical validation of functional assays for genetic variant classification relies on "truthsets"—collections of variants with established pathogenic or benign status. However, clear guidance is lacking regarding how these truthsets should be assembled, specifically concerning the selection of variant types (e.g., missense vs. protein truncating), the stringency of classification (e.g., number of ClinVar stars, allowance of soft-conflicts), and the phenotypic context of submissions. This lack of standardization leads to significant inconsistency in the evidence points (EPs) allocatable to functional assays under the ClinGen assay-level clinical validation framework. Consequently, the same assay may yield vastly different evidence strengths depending on the arbitrary choices made during truthset construction, hindering the consistent application of functional data in clinical variant classification.

Methodology
The study employed a multi-step analytical approach to quantify the impact of truthset assembly rules on clinical validation outcomes:

1. Survey of Current Practice: The authors reviewed 112 ClinGen gene-specific classification specifications (CSPECs) from 39 Variant Curation Expert Panels (VCEPs) to assess existing methodologies for truthset assembly and assay validation.
2. Truthset Permutation Generation: The researchers systematically constructed truthsets for four cancer susceptibility genes (VHL, BRCA1, BRCA2, and RAD51C) using ClinVar data. They varied multiple parameters to generate 70 distinct "basic" approaches:
   • Variant Type: All variants, Protein Truncating Variants (PTVs) + Synonymous, or Missense only.
   • Classification Stringency: Pathogenic/Likely Pathogenic (P/LP) vs. P only; inclusion/exclusion of "soft-conflicts" (e.g., VUS vs. LP); and varying star ratings (any, ≥1, ≥2, ≥3).
   • Phenotype: For VHL, they tested restrictions to specific phenotypes (e.g., Type 1 vs. Type 2 VHL syndrome, Chuvash Polycythaemia).
   • Augmentation: They augmented ClinVar-classified benign missense variants with "proxy-clinical" benign variants systematically assembled using ACMG/AMP v3.0 rules (Level 1 to Level 4 stringency based on allele frequency and in silico predictions).
3. Clinical Validation Application: These 70+ truthset permutations were applied to five large-scale multiplexed functional assays (MAVEs) for the four genes.
4. Metric Calculation: For each of the 2,120 gene-truthset-assay combinations, the study calculated:
   • Allocatable ACMG/AMP Evidence Points (EPs) for pathogenicity (PS3) and benignity (BS3).
   • Concordance metrics: False Positive Rate for pathogenicity (reflecting sensitivity) and benignity (reflecting specificity).
   • Balanced accuracy.

Key Contributions and Results

• Inconsistency in Current Guidelines: The review of 112 CSPECs revealed that only 61% of guidelines where functional evidence is allowed elucidate the truthsets used for validation. Among those, only 52% provided provenance details, and merely 2% specified variant type or classification stringency, highlighting a critical lack of transparency and standardization.
• Wide Variability in Evidence Points: Applying different truthset permutations to the same assay resulted in dramatic fluctuations in allocatable EPs.
  • For the VHL assay, EPs for pathogenicity ranged from 0.0 (nil) to 5.6 (strong), and for benignity from -1.5 to -6.4.
  • For the BRCA1 assay, pathogenicity EPs ranged from 1.9 to 6.0.
  • This variability demonstrates that the choice of truthset is a primary driver of the evidence strength assigned to an assay.
• Impact of Variant Type: Clinical validation using truthsets comprising only ClinVar-classified missense variants typically yielded lower EPs than those including PTVs and synonymous variants. This reduction was primarily attributed to the paucity of available ClinVar-classified missense truthset variants (particularly benign ones) rather than poorer assay concordance.
• Impact of Stringency: Increasing the stringency of truthset selection (e.g., requiring ≥3 stars or excluding soft-conflicts) generally improved concordance metrics but drastically reduced the size of the truthset. Due to the ClinGen framework's reliance on truthset size (specifically the Haldane correction in the denominator), increased stringency often resulted in lower allocatable EPs despite higher precision.
• Benefit of Systematic Augmentation: Augmenting ClinVar-classified truthsets with systematically generated "proxy-clinical" benign missense variants (using ACMG/AMP rules) typically improved the allocatable EPs for pathogenicity. The increased statistical power from larger truthset sizes often negated modest reductions in concordance.
• Phenotype and Threshold Effects: Restricting truthsets to specific phenotypes (e.g., Type 1 VHL) improved concordance but reduced truthset size, often lowering EPs. Similarly, tightening the deleterious score threshold (defining a more extreme "deleterious" zone) improved concordance but reduced the proportion of variants classified as deleterious, affecting EP calculations.

Significance and Claims
The paper claims that the current heterogeneity in truthset assembly creates a "rich substrate" for inconsistent clinical validation, where the same functional assay can be assigned vastly different evidence strengths based solely on the truthset construction rules. The authors argue that:

1. Explicit Guidance is Urgently Needed: There is a critical need for prescriptive clinical guidance to standardize truthset assembly to improve consistency in evidence application.
2. Power vs. Precision Trade-off: While high-stringency truthsets offer better concordance, the loss of statistical power (due to small sample sizes) often results in lower allocatable evidence. The study suggests that augmenting truthsets with systematically classified "proxy-clinical" variants is a viable strategy to improve evidence strength, particularly for pathogenicity.
3. Missense Focus: While PTV-synonymous validation is a useful first step, the clinical utility of functional assays lies in classifying missense variants. Therefore, validation strategies must address the specific challenges of missense truthset assembly, including the scarcity of benign missense variants in public databases.

The study concludes that insights from these 2,120 permutations will facilitate the development of more prescriptive guidelines for the assembly of truthset variants, ultimately leading to more reliable and consistent clinical variant classification.