AnnotateMissense: a genome-wide annotation and benchmarking framework for missense pathogenicity prediction

Technical Summary: AnnotateMissense

Problem Statement
The interpretation of missense variants remains a significant challenge in genomics because pathogenicity is determined by heterogeneous, partially overlapping evidence layers. These include population frequency, evolutionary conservation, transcript context, amino acid substitution severity, prior pathogenicity predictors, and features derived from protein language models. While individual computational resources exist, there is a lack of reproducible workflows that integrate these diverse sources at a genome-wide scale, systematically benchmark their contributions, and provide accessible outputs for downstream research. Furthermore, integrating these resources is complicated by differences in score directionality, coverage, and potential data circularity, particularly when predictors are trained on clinical databases like ClinVar or HGMD.

Methodology
The authors present AnnotateMissense, a scalable framework for genome-wide annotation, feature integration, benchmarking, and prediction. The workflow processes 90,643,830 hg38 missense single-nucleotide variants derived from dbNSFP v5.1.

• Feature Integration: The framework integrates a 303-feature space comprising:

  • ANNOVAR-based gene, region, and filter annotations.
  • dbNSFP transcript and protein descriptors.
  • AlphaMissense scores and ESM-derived features (protein language models).
  • Conservation metrics (e.g., PhyloP, GERP++).
  • Population frequency variables (e.g., gnomAD).
  • Established pathogenicity predictors (e.g., SIFT, PolyPhen-2, REVEL, CADD).
  • Engineered biological features (e.g., amino acid physicochemical changes, codon composition, BLOSUM62 scores).
  • Direct ClinVar fields intended for prediction were excluded to minimize target leakage.

• Benchmarking and Model Evaluation:

  • Dataset: A supervised benchmark was constructed using 132,714 ClinVar-labelled missense variants (76,804 pathogenic/likely pathogenic; 55,910 benign/likely benign). Variants with uncertain significance or conflicting labels were excluded.
  • Models: Multiple machine learning (XGBoost, Random Forest, FLAML AutoML) and deep learning models (TabNet, PyTorch DNN, TensorFlow DNN) were evaluated using stratified five-fold cross-validation.
  • Metrics: The primary metric was the Matthews Correlation Coefficient (MCC), supplemented by ROC-AUC, accuracy, and F1-score.
  • Ablation Studies: To assess feature contributions and control for circularity, the authors tested restricted feature sets:
    • Naive features: Conservation and functional annotations only.
    • Location-oriented features: Local sequence and functional context.
    • Circularity-controlled: Excluding prior predictors, population frequencies, or tools trained on clinical databases (ClinVar/HGMD).
    • Protein-model exclusion: Removing AlphaMissense and ESM features.

• Validation:

  • Temporal Validation: The model was tested on newly observed pathogenic/benign variants in a newer ClinVar release that were absent from the training set.
  • Biological Use Cases: Analysis of Variants of Uncertain Significance (VUS) stratified by gnomAD missense constraint Z-scores and pairwise discordance analysis against established predictors.

Key Results

• Performance: The full 303-feature benchmark set achieved the highest performance using XGBoost, with a mean MCC of 0.9411 and ROC-AUC of 0.9950 across cross-validation folds.
• Feature Importance: Restricted feature sets performed substantially worse. The naive feature set (41 features) achieved a best MCC of 0.4989, and the location-oriented set (56 features) achieved 0.5113.
• Ablation Insights:
  • Removing prior pathogenicity predictors and population-frequency metrics (Dataset 5) reduced the best MCC to 0.7582.
  • Removing features derived from tools trained on clinical databases (Dataset 6) resulted in a similar reduction (MCC 0.7326).
  • Excluding AlphaMissense and ESM-derived features alone (Dataset 7) had minimal impact (MCC 0.9409), suggesting these features did not provide independent signal beyond the broader annotation set in this specific configuration.
• Temporal Validation: On 298,353 newly observed variants, the model achieved an MCC of 0.7613, accuracy of 0.8798, and F1-score of 0.8750.
• Biological Utility: VUS in missense-intolerant genes (gnomAD Z-score > 3) received significantly higher predicted pathogenicity scores (mean 0.631) compared to those in tolerant genes (mean 0.548).
• Discordance Analysis: AnnotateMissense showed broad concordance with established predictors but provided complementary classifications for a subset of variants. Notably, on discordant variants with ESM1v, AnnotateMissense agreed with ClinVar labels 67.9% of the time.

Significance and Claims
The authors position AnnotateMissense as a scalable annotation, benchmarking, and research-prioritisation framework, explicitly stating it is not a standalone clinical classification system.

• Integration is Key: The primary finding is that high-performing missense classification requires the integration of heterogeneous evidence sources. Models trained on the full multi-source feature set substantially outperformed those relying solely on conservation, functional annotation, or local sequence features.
• Transparency and Reproducibility: The framework provides a transparent method for quantifying feature-set contributions and managing circularity in benchmarking.
• Resource Availability: The authors have released the source code, workflow scripts, and the full genome-wide prediction outputs (90,643,830 variants) as a public research resource via GitHub and Zenodo.
• Clinical Caveat: The paper emphasizes that predictions should be interpreted as prioritisation scores for research, benchmarking, and variant triage. Clinical interpretation requires expert review and integration with disease context, inheritance, segregation, and functional evidence, consistent with current clinical guidelines.