ProtFinder: An efficient machine learning framework for protein model selection on real data

Technical Summary of ProtFinder: An Efficient Machine Learning Framework for Protein Model Selection on Real Data

Problem Statement
Phylogenetic inference relies fundamentally on selecting the best-fit model of sequence evolution for a given multiple sequence alignment (MSA). Traditional methods, such as ModelFinder, utilize statistical information criteria (e.g., BIC, AIC) to evaluate candidate models. While accurate, these likelihood-based approaches are computationally expensive, requiring repeated maximum-likelihood optimizations that become prohibitive for large datasets. Furthermore, the applicability of information criteria in phylogenetics has been questioned due to potential violations of underlying assumptions.

Existing machine learning (ML) approaches, such as ModelDetector, offer speed but suffer from significant limitations: they are trained exclusively on simulated data, leading to poor performance on real biological datasets, and they lack support for critical model components like rate heterogeneity across sites (RHAS) and amino acid frequency determination.

Methodology
The authors introduce ProtFinder, an efficient ML framework designed to predict three distinct components of protein evolution models:

1. QFinder: Selects the best-fit amino acid substitution matrix (exchangeability rate matrix).
2. FFinder: Determines the amino acid frequency model (either empirical frequencies fixed to the model [-F] or calculated from the input MSA [+F]).
3. RHASFinder: Identifies the rate heterogeneity among sites model (None, +I, +G, or +I+G).

Data Strategy and Transfer Learning:
To bridge the gap between simulation and reality, ProtFinder employs a three-stage transfer learning strategy:

1. Initial Training: Pre-training on a large-scale simulated dataset (268,800 MSAs) generated using AliSim. Simulation parameters were derived from the empirical cumulative distribution functions of real data (EvoNAPS database) to ensure realism.
2. Joint Training: Training on a combination of simulated data (13,440 MSAs) and real data (15,330 MSAs from EvoNAPS).
3. Fine-tuning: Final training exclusively on the real EvoNAPS dataset to improve generalization.

Feature Extraction and Network Architecture:

• QFinder: Uses a Convolutional Neural Network (CNN) with Squeeze-and-Excitation (SE) blocks. It processes a 440-feature matrix derived from 625 randomly sampled sequence pairs per alignment, capturing substitution frequencies, invariability counts, and amino acid frequencies.
• FFinder: Utilizes XGBoost with hyperparameters optimized via grid search. Features include observed amino acid frequencies, Kullback–Leibler and Jensen–Shannon divergences, and alignment statistics.
• RHASFinder: Employs a Transformer architecture. It processes site-specific features (relative frequencies, Shannon entropy, diversity counts, and invariability indicators) alongside global alignment statistics (e.g., proportion of invariable sites, entropy variance).

Training Details:
The framework addresses class imbalance in real data by weighting the loss function by the reciprocal of class sample counts. Training utilizes the AdamW optimizer, L2 regularization, and early stopping.

Key Results

• Accuracy on Simulated Data:
  • Substitution Models: QFinder achieved 87.06% average accuracy, outperforming ModelDetector (48.50%) and approaching the accuracy of ModelFinder (91.75%). Accuracy improved significantly with alignment length and taxon count.
  • Frequency Models: FFinder significantly outperformed ModelFinder (96.93% vs. 87.73%), particularly on short alignments where ModelFinder struggled.
  • RHAS Models: RHASFinder slightly outperformed ModelFinder (76.75% vs. 74.63%), with the advantage being most pronounced on smaller datasets (fewer taxa or shorter sequences).
• Accuracy on Real Data (HSSP Dataset):
  • ProtFinder demonstrated high agreement with ModelFinder on an independent real dataset (HSSP). QFinder achieved 67% top-1 agreement and 90.96% top-2 agreement.
  • FFinder and RHASFinder achieved top-1 agreements of 74.54% and 73.67%, respectively, with top-2 agreements exceeding 99%.
  • In contrast, ModelDetector performed poorly on real data (48.76% top-1 agreement), confirming the necessity of training on real data.
• Computational Efficiency:
  • ProtFinder is up to 1,400 times faster than ModelFinder.
  • While ModelFinder's runtime scales rapidly with the number of taxa and alignment length, ProtFinder's runtime remains nearly constant. For an alignment with 256 taxa and 1,000 sites, ModelFinder took ~10 minutes, whereas ProtFinder took 1.5 seconds.

Significance and Claims
The paper claims that ProtFinder represents a significant advancement by successfully adapting machine learning for phylogenetic model selection on real biological data, overcoming the primary limitation of previous ML methods. Its key contributions are:

1. Comprehensive Scope: It is the first ML framework to simultaneously predict substitution models, frequency models, and RHAS models.
2. Real-World Applicability: Through transfer learning, it achieves accuracy comparable to maximum likelihood methods (ModelFinder) on medium-to-large datasets while being orders of magnitude faster.
3. Practical Utility: The authors propose a two-stage strategy where ProtFinder acts as a fast pre-screening module to identify a small set of high-probability candidate models, which are then rigorously evaluated by ModelFinder. This approach aims to retain the accuracy of likelihood-based methods while drastically reducing computational costs.

The authors note that while ProtFinder excels in speed and general accuracy, distinguishing between highly correlated substitution models (e.g., Q.pfam vs. LG) remains challenging for both ML and likelihood methods due to similar amino acid substitution patterns. They plan to integrate ProtFinder into the IQ-TREE software suite to facilitate this accelerated workflow.