Technical Summary: Pre-AF 13 – An Interpretable Atrial Fibrillation Risk Score

Problem Statement

Atrial fibrillation (AF) is the most prevalent cardiac arrhythmia and a critical determinant of patient prognosis, contributing significantly to stroke, heart failure, and mortality. Existing clinical risk scores (e.g., CHARGE-AF, C2HEST, MHS, HAVOC) often rely on common risk factors such as age, hypertension, and obesity. While effective for the general population, these tools offer limited stratification power in patients with established cardiovascular disease (CVD), where such risk factors are nearly ubiquitous. Furthermore, traditional scores are typically designed for long-term (5–10 year) prediction, which is less suitable for identifying patients requiring immediate screening or intervention in the short-to-medium term (6–24 months). Additionally, many existing tools lack interpretability and fail to incorporate dynamic factors such as ongoing pharmacological treatments, structural remodeling markers, or the impact of prior invasive interventions.

Methodology

Data Sources and Study Population

The study utilized a retrospective, single-center dataset from the National Research Cardiology Center in Russia, covering the period from January 2012 to May 2019. The cohort consisted of 45,000 unique patients (17,562 meeting inclusion criteria) aged ≥18 years with documented CVD but no pre-existing AF, who had been hospitalized more than once. The dataset included 80,576 electronic health records (EHRs).

Natural Language Processing (NLP) Pipeline

A custom NLP pipeline was developed to transform unstructured discharge reports into 73 structured features. This pipeline combined two approaches:

1. Rule-Based Parsing: Utilizing regular expressions and a GLR parser (Yargi Parser) for easily extractable features (e.g., age, gender, BMI, diabetes). This achieved a mean F1 score of 0.98 for binary features.
2. Deep Learning-Based Extraction: Employing transformer-based Named Entity Recognition (NER) models (specifically domain-adapted RuBioRoBERTa) for complex features requiring contextual understanding, such as medication names, left ventricular hypertrophy (LVH), and peripheral artery disease (PAD). Active learning was used to optimize annotation efficiency. This approach achieved an average F1 score of 0.95.

The pipeline extracted features from diagnosis, physical examination, laboratory tests, instrumental tests, and medication recommendations.

Target Variable Definition

The prediction task was formulated as a binary classification of incident AF within specific time windows relative to an index hospital visit:

• 24-Month Target: AF diagnosed between 1 week and 24 months post-visit.
• Entire Follow-Up Target: AF diagnosed at any point after the index visit (excluding the first week).

Model Construction

Three distinct models were developed using the LightAutoML framework:

1. Full Model: Trained on all 73 extracted features using an ensemble of Random Forest, LightGBM, L2-regularized linear regression, and CatBoost.
2. Simple Model (Pre-AF 13): A compact model using a subset of 13 features selected via a consensus of statistical (F-test) and model-based (Recursive Feature Elimination, Sequential Feature Selection) methods.
3. Linear Model (Pre-AF 9): A simplified multivariate linear model derived from the 13 features, further reduced to 9 predictors to create a paper-based risk score with integer weights.

Evaluation and Interpretation

Performance was assessed using ROC AUC, Precision-Recall AUC, balanced accuracy, and F1 scores. Models were compared against four established clinical scores (CHARGE-AF, C2HEST, MHS, HAVOC). Interpretability was achieved using SHapley Additive exPlanations (SHAP) to quantify feature contributions.

Key Results

Feature Extraction Performance

The NLP pipeline demonstrated high efficacy, with the rule-based parser achieving a 0.97 average performance across all features and the transformer-based extractors achieving 0.95.

Predictive Performance

• Full Model: Achieved a test ROC AUC of 0.735 for the 24-month target and 0.696 for the entire follow-up period.
• Simple Model (Pre-AF 13): Performed nearly identically to the full model, with a test ROC AUC of 0.725 (24-month) and 0.696 (entire follow-up).
• Comparison: All non-linear models (Full and Simple) significantly outperformed the four established clinical risk scores, which yielded ROC AUCs ranging from 0.53 to 0.64 on the same test set.
• Linear Model: The linear model (ROC AUC ~0.64) showed reduced performance compared to non-linear models but remained non-inferior to the established clinical scores.

Feature Importance

SHAP analysis identified age and left atrial volume as the dominant predictors. Other significant features included supraventricular ectopic beats, heart failure, creatinine levels, and specific medications. Notably, clopidogrel and percutaneous coronary intervention (PCI) emerged as protective factors (negative SHAP values), likely reflecting better risk factor management in these patient subgroups.

Risk Stratification

The derived Pre-AF 9 scoring system successfully stratified risk. For the 24-month horizon, the observed incidence of AF ranged from approximately 7% in the lowest risk category to 36% in the highest risk category.

Key Contributions and Significance

1. Superior Performance in High-Risk Cohorts: The study demonstrates that machine learning models trained on routinely collected hospital data can outperform traditional clinical risk scores specifically in patients with established CVD, a population where standard scores often fail to provide granular stratification.
2. Interpretability and Clinical Utility: By distilling complex non-linear models into a 13-feature "Simple Model" and a 9-feature "Linear Model" (Pre-AF 9), the authors created tools that are both high-performing and clinically interpretable. The Pre-AF 9 score allows for bedside risk assessment without computational resources.
3. Short-to-Medium Term Prediction: The models are specifically optimized for a 24-month horizon, addressing a gap in existing literature where most tools focus on long-term (5–10 year) risks. This timeframe is more actionable for immediate screening and intervention strategies.
4. End-to-End NLP Pipeline: The research validates a robust pipeline for extracting structured clinical features from unstructured Russian-language EHRs, proving that high-quality automated extraction is feasible without manual chart review, even for complex domain-specific concepts.
5. Identification of Novel Predictors: The models successfully integrated markers of atrial substrate remodeling (e.g., left atrial volume) and treatment effects (e.g., SGLT2 inhibitors, clopidogrel), offering a more dynamic view of AF risk than static demographic scores.

The authors conclude that while the absolute ROC AUC values are modest (consistent with the difficulty of predicting multifactorial cardiovascular events), the relative improvement over established scores and the ability to stratify risk in a high-risk population highlight the potential for these tools to guide more precise screening and preventive strategies. The study acknowledges limitations regarding its single-center, retrospective nature and the need for external validation before clinical deployment.