AnnotateMissense: a genome-wide annotation and benchmarking framework for missense pathogenicity prediction

1. Problem Statement

The interpretation of missense variants (single nucleotide changes resulting in an amino acid substitution) remains a critical bottleneck in genomic medicine. The primary challenge lies in the heterogeneity of evidence required to determine pathogenicity. Accurate classification depends on synthesizing diverse data types, including:

• Population frequency and evolutionary conservation.
• Transcript context and amino acid substitution severity.
• Outputs from prior pathogenicity predictors.
• Emerging features derived from Protein Language Models (pLMs).

Existing tools often lack a unified, scalable framework that integrates these disparate data sources while rigorously benchmarking the contribution of specific feature sets to predictive performance.

2. Methodology

The authors developed AnnotateMissense, a comprehensive framework designed for annotation, benchmarking, and genome-wide prediction. The methodology involves three core stages:

• Data Integration & Feature Engineering:

  • Source Data: The framework integrates 132,714 ClinVar-labelled missense variants (hg38) derived from dbNSFP v5.1 and ANNOVAR.
  • Feature Set: A massive 303-feature vector was constructed, encompassing:
    • Transcript/Protein Descriptors: From dbNSFP.
    • AI-Derived Scores: AlphaMissense scores and features extracted from ESM (Evolutionary Scale Modeling) protein language models.
    • Conservation & Frequency: Evolutionary conservation metrics and population frequency variables.
    • Contextual Features: Engineered amino acid and codon-context features.
    • Legacy Predictors: Outputs from established pathogenicity predictors.

• Model Benchmarking:

  • The authors evaluated various Machine Learning (ML) and Deep Learning (DL) architectures.
  • Validation Strategy: Models were trained using stratified five-fold cross-validation.
  • Ablation Studies: To understand feature importance, the authors performed "circularity-controlled" ablations, systematically removing specific data categories (e.g., prior predictors, population frequency, AlphaMissense, ESM features) to measure the impact on model performance.

• Temporal Validation:

  • To ensure robustness against data leakage and overfitting, the models were validated against newly observed pathogenic/benign variants in ClinVar (temporal split), simulating real-world deployment on future data.

3. Key Contributions

• Unified Framework: Creation of a scalable, end-to-end pipeline for missense variant interpretation that harmonizes traditional genomic features with modern pLM-derived embeddings.
• Comprehensive Benchmarking: A rigorous comparison of feature subsets, demonstrating that while AI-derived features (AlphaMissense/ESM) are valuable, they are not the sole drivers of performance when compared against established clinical and population data.
• Genome-Wide Application: The successful application of the final model to 90,643,830 hg38 missense variants, generating a massive, publicly available dataset of pathogenicity scores and binary labels.
• Open Science: Full release of code, model outputs, and documentation via GitHub and Zenodo to facilitate reproducibility and further research.

4. Results

The study yielded several high-impact quantitative findings:

• Peak Performance: The full 303-feature set, modeled using XGBoost, achieved state-of-the-art performance:
  • Mean MCC (Matthews Correlation Coefficient): 0.9411
  • ROC-AUC: 0.9950
• Feature Importance (Ablation):
  • Removing prior-predictor, population-frequency, and clinically overlapping evidence caused a significant drop in performance, highlighting that traditional clinical data remains the backbone of prediction.
  • Excluding AlphaMissense and ESM-derived features alone had a minimal effect, suggesting that while these AI features are additive, the model's core strength relies on the integration of diverse, non-AI evidence.
  • Restricted "naive" and "location-oriented" feature sets performed significantly worse (MCC ~0.50), confirming the necessity of the full feature spectrum.
• Temporal Validation: On newly observed ClinVar variants, the model maintained strong generalization capabilities:
  • MCC: 0.7613
  • Accuracy: 0.8798
  • F1-Score: 0.8750

5. Significance

AnnotateMissense addresses the complexity of missense variant interpretation by demonstrating that the most robust predictions arise from a hybrid approach. It proves that while cutting-edge protein language models (like AlphaMissense and ESM) provide valuable insights, they must be integrated with population genetics, evolutionary conservation, and established clinical evidence to achieve maximum accuracy.

The framework sets a new standard for benchmarking pathogenicity predictors, offering a transparent view of how different evidence types contribute to decision-making. By providing a genome-wide annotation of nearly 91 million variants, the authors have created a vital resource for researchers and clinicians to prioritize variants for further study, ultimately accelerating the diagnosis of genetic diseases.