'Truthsets' for clinical validation of large-scale functional assays: Practice recommendations from Cancer Variant Interpretation Group UK (CanVIG-UK)

Technical Summary: 'Truthsets' for clinical validation of large-scale functional assays: Practice recommendations from Cancer Variant Interpretation Group UK (CanVIG-UK)

Problem Statement
The classification of Variants of Uncertain Significance (VUS), particularly rare missense variants, remains a critical challenge in clinical genomics. While large-scale multiplex assays of variant effect (MAVEs) offer transformative potential for resolving these variants, a significant implementation gap exists between the publication of functional data and its routine adoption in clinical diagnostic services. This gap is driven by a lack of consensus on how to construct "truthsets"—reference sets of clinically classified pathogenic and benign variants used to validate assay performance.

Current practices vary widely, with many assayists utilizing mixed variant types (e.g., combining protein-truncating variants [PTVs], synonymous variants, and missense variants) for validation. This heterogeneity creates ambiguity and reduces confidence among clinical diagnostic scientists. Furthermore, the existing ClinGen assay-level clinical validation framework (Brnich et al., 2020) provides baseline guidance but lacks specific mandates regarding truthset composition, leading to potential over-estimation of assay performance and inconsistent evidence point (EP) allocation.

Methodology
CanVIG-UK (Cancer Variant Interpretation Group UK) developed consensus recommendations through an iterative national consultation process involving the CanVIG Steering Advisory Group (CStAG), wider membership, and international functional genomics experts. The methodology comprised three primary components:

1. Survey of Current Practices: An analysis of CSpec pages for 112 genes across 39 Variant Curation Expert Panels (VCEPs) revealed inconsistent application of functional data and wide variation in truthset provenance.
2. Truthset Analyses: The authors performed 2,120 distinct clinical validations across four genes and corresponding assays. These analyses systematically varied truthset composition parameters, including:
   • Variant type (missense-only vs. PTV/synonymous vs. all variants).
   • Stringency of ClinVar classification (e.g., pathogenic vs. likely pathogenic, number of submitters, allowance of "soft-conflicts").
   • Augmentation with "proxy-clinical" benign variants (systematically derived using ACMG/AMP rules on population and in silico data).
   • Phenotypic association stringency and deleterious score zones.
3. Consensus Building: Preliminary findings were presented to CStAG, CanVIG-UK, and the ClinGen Functional Working Group. Through multiple iterations and reviews, nine guiding principles and seven best-practice recommendations were established, specifically tailored for cancer susceptibility genes where loss-of-function is the mechanism of pathogenicity.

Key Contributions and Results
The study establishes a standardized framework for truthset construction to ensure independent, consistent clinical validation.

• Nine Guiding Principles:

  • Principle A: Validation must reflect both predictiveness (concordance) and confidence (power/number of variants).
  • Principle B: Mechanistic asymmetry exists; proving pathogenicity requires demonstrating one abrogated mechanism, while proving benignity requires confirming all mechanisms are neutral.
  • Principle C: Classification shifts from VUS to pathogenic carry higher clinical utility but also higher risk of error compared to shifts to benign.
  • Principle D: Missense variants are the critical target. Functional data most influences missense classification. Using PTVs or synonymous variants as primary truthsets may overestimate performance because they do not recapitulate the specific challenges of missense variants (e.g., modest functional perturbations).
  • Principle E: Assayist-defined thresholds (often calibrated on PTVs/synonymous variants) may not align with missense distributions, potentially leading to misclassification near thresholds.
  • Principle F & G: There is a paucity of clinically classified benign and pathogenic missense variants, limiting statistical power.
  • Principle H: Truthset contamination attenuates evidence points (EPs) but does not inflate them.
  • Principle I: Circularity must be avoided where clinical classifications rely on the same functional data used for validation.

• Seven Best-Practice Recommendations:

  1. Missense-Only Truthsets: For assays intended to interpret missense variants, truthsets must comprise missense variants only. Mixing in PTVs or synonymous variants is not recommended as it dilutes signal and risks over-estimation.
  2. Independent Validation: Clinical validation should be performed independently of the assayist's initial calibration, using distinct truthset variants where feasible.
  3. Relaxed Stringency: To improve power (and thus EPs), stringency regarding ClinVar classification (e.g., allowing "likely" classifications or soft-conflicts) can be relaxed, provided the truthset is defined a priori.
  4. Proxy-Clinical Benign Variants: Benign missense truthsets can be augmented with "proxy-clinical" variants—variants not clinically reported but systematically classified as benign using population data (e.g., gnomAD) and in silico predictions.
  5. Evidence for Benignity: If sufficient concordant pathogenic missense variants exist, evidence for benignity (BS3) can be allocated. If fewer than three exist, a single EP may be allocated if assay-defined deleterious variants show a statistically significant association with the disease phenotype en masse.
  6. Score Distribution Analysis: The distribution of assay scores for missense truthsets must be compared against PTV/synonymous distributions to ensure thresholds are appropriate.
  7. Circularity Mitigation: Filtering or censoring ClinVar data (e.g., excluding classifications made after assay publication or those relying on the assay's own data) is recommended to prevent circularity.

• Mathematical Fix: A specific mathematical adjustment (Haldane correction) is proposed for scenarios where zero concordant pathogenic missense variants exist but sufficient benign variants are available, allowing for the calculation of pathogenicity evidence points.

Significance and Claims
The paper claims that these principles and recommendations provide a necessary baseline for the clinical implementation of large-scale functional assays. By mandating missense-only truthsets and offering strategies to overcome data scarcity (via proxy-clinical variants and relaxed stringency), the framework aims to:

1. Address the implementation gap between functional data publication and clinical use.
2. Enable consistent, independent validation that prevents the over-estimation of assay performance.
3. Facilitate the classification of VUS based on functional data alone, which is becoming feasible under the upcoming ACMG/AMP v4.0 guidelines (where functional data can provide up to +8/-8 points).

The authors emphasize that while these recommendations are developed for cancer susceptibility genes acting via loss-of-function, they form a foundational discussion for other functional and disease contexts. The work acknowledges limitations, such as the lack of "proxy-clinical" methods for pathogenic variants and the challenges of validating reduced penetrance or non-coding variants, identifying these as priorities for future work.