Chart review and genetic validation of electronic medical record dementia diagnoses in VA: The impact of CMS data

Technical Summary: Chart Review and Genetic Validation of Electronic Medical Record Dementia Diagnoses in VA: The Impact of CMS Data

Problem Statement
Epidemiological studies and genetic research regarding Alzheimer's disease (AD) and related dementias (ADRD) frequently rely on International Classification of Diseases (ICD) codes within Electronic Medical Records (EMR) to identify cases. However, identifying ADRD within the Veterans Affairs (VA) healthcare system presents specific challenges, including the widespread use of non-specific dementia codes and the tendency for initial diagnoses to occur in primary care rather than specialty clinics. Furthermore, Veterans often receive care outside the VA system, potentially creating data gaps and selection bias in VA-EMR-only studies. While Centers for Medicare & Medicaid Services (CMS) data offers a potential solution to capture these external encounters, the impact of integrating CMS data on the performance of diagnostic algorithms—specifically regarding sensitivity, specificity, and genetic validity—remains an open question.

Methodology
The authors conducted a multi-faceted evaluation using data from the VA Million Veteran Program (MVP), a large biobank with over one million enrolled Veterans.

• Algorithm Evaluation: The study benchmarked nine distinct AD/ADRD algorithms: four MVP-derived algorithms (ranging from strict AD to broad dementia), two Chronic Conditions Warehouse (CCW) algorithms, two PheCode algorithms, and one medication-based algorithm. These were tested both with VA EMR data alone and with VA EMR data augmented by CMS data (including Medicare/Medicaid claims and encounters).
• Chart Review Validation: A chart review was performed on a combined cohort of 203 participants (100 new reviews plus 103 from a prior study). Medical records were scrubbed of identifiers and reviewed by two independent reviewers using a "silver standard" classification system ("Likely," "Possible," or "Likely Not" for AD and all-cause dementia). Consensus was reached through discussion. Performance metrics (sensitivity, specificity, positive predictive value [PPV], negative predictive value [NPV]) were calculated against these manual classifications.
• Genetic Validation: To assess the biological validity of the case definitions, the authors examined associations between the algorithm-derived case sets and two genetic markers in a large MVP cohort (n ≈ 396,000, restricted to European ancestry): the APOE ε4 allele (the strongest genetic risk factor for late-onset AD) and an AD polygenic risk score (PRS) excluding the APOE region. Logistic regression models were used to calculate odds ratios (ORs) and Z-scores.

Key Contributions

1. CMS Data Integration: The study provides a rigorous assessment of how adding CMS data to VA EMR data alters the performance of dementia algorithms.
2. Algorithm Refinement: The authors updated their MVP-derived algorithms to reflect changes in ICD code usage and compared them against established CCW and PheCode standards.
3. Genetic Benchmarking: By correlating algorithm performance with APOE ε4 and PRS associations, the study offers a method to evaluate the "purity" of case sets regarding late-onset AD, distinguishing between algorithms that capture true AD cases versus those that capture broader, less specific dementia phenotypes.

Results

• Case Yield and Sensitivity: The addition of CMS data significantly increased the number of detected cases across all algorithms, often doubling the count for strict AD algorithms (e.g., MVP-AD increased from ~11,345 to ~22,537). Consequently, sensitivity and negative predictive value (NPV) improved.
• Specificity and Predictive Value: The increase in case yield came with trade-offs. Specificity and positive predictive value (PPV) decreased when CMS data was added.
• Genetic Associations:
  • Strict AD Algorithms: When using VA data alone, strict AD algorithms (MVP-AD, Phe-AD) yielded the highest odds ratios (ORs) for APOE ε4 (OR ≈ 2.40–2.41), indicating a high proportion of true late-onset AD cases. However, adding CMS data to these strict algorithms reduced the ORs (to ≈ 1.95–1.97), suggesting the new cases included a lower proportion of true AD.
  • Broad ADRD Algorithms: Broader algorithms (MVP-ADRD, Phe-Dementia) showed strong genetic associations but lower ORs overall. Interestingly, the performance of broad algorithms using only VA data was comparable to strict algorithms using both VA and CMS data.
  • Medication Data: Including AD medication data generally increased sample sizes but, similar to CMS data, tended to lower the ORs for strict AD algorithms.
  • Age Cutoffs: Applying an age cutoff (65+) uniformly increased the significance of genetic associations and OR estimates across algorithms.

Significance and Claims
The paper concludes that the choice of algorithm and data source depends heavily on the study's specific goals:

• For Genetic Discovery: To maximize the proportion of true late-onset AD cases (indicated by higher ORs), the authors recommend using strict AD algorithms based solely on VA EMR data without CMS or medication data. If CMS data is available and necessary for sample size, the authors suggest using strict AD algorithms (MVP-AD, CCW-AD, or Phe-AD) supplemented with CMS and medication data, though this results in a slightly lower OR than VA-only strict algorithms.
• For Epidemiological Studies: For studies prioritizing sensitivity and case capture (e.g., monitoring disease rates), the authors recommend using broader ADRD algorithms (MVP-ADRD or Phe-Dementia) based on VA EMR data alone. They note that adding CMS data to broad algorithms does not significantly improve performance and may dilute the phenotype.
• General Best Practices: The study reinforces the need for requiring multiple ICD codes to qualify as a case, integrating medication data where appropriate, and applying age cutoffs (≥60 or ≥65) to focus on late-onset disease.

The authors emphasize that ICD coding practices vary by institution and data source (VA vs. CMS). Therefore, the inclusion of CMS data fundamentally changes the composition of the case set, making it more "ADRD-like" rather than purely "AD-like." Consequently, researchers must carefully evaluate algorithm performance within their specific EMR systems rather than assuming uniform applicability across different healthcare databases.