📄 public and global health

Predicting Subjective Cognitive Decline on Future BRFSS Survey Years: An Open Multi-Language Machine Learning Benchmark

This study presents an open, multi-language machine learning benchmark using Behavioral Risk Factor Surveillance System (BRFSS) data that demonstrates stable, well-calibrated prediction of future Subjective Cognitive Decline (SCD) across temporal splits while identifying key predictors like decision-making difficulty through convergent interpretability methods.

Original authors: Nguyen, T. T., Nguyen, T. D.

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nguyen, T. T., Nguyen, T. D.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

For many people, the first sign that their mind is changing is not a medical test result, but a quiet, personal worry. They might feel that their memory is slipping or that they are more confused than usual. This feeling, known as subjective cognitive decline, is a common experience among adults, yet it is difficult to study because it relies entirely on what a person says they feel rather than what a doctor can see. While this feeling can sometimes be a very early warning sign of serious conditions like Alzheimer's disease, it can also stem from stress, poor sleep, or other health issues. Because the feeling is so personal and the causes are so varied, tracking it across large groups of people has been a challenge for public health officials. They need a way to understand who is at risk and how these feelings change over time, but traditional methods often rely on small groups of patients or expensive brain scans that are not available to everyone.

A team of researchers has tackled this problem by building a new kind of digital tool to track these feelings across the entire United States. Instead of looking at patients in a hospital, they turned to a massive, ongoing telephone survey called the Behavioral Risk Factor Surveillance System, which asks millions of Americans about their health every year. The researchers wanted to see if they could use the answers people give about their daily lives—such as how they feel about their general health, whether they have trouble making decisions, or if they struggle with depression—to predict who is likely to report memory problems in the future. To do this, they created a computer program that learns from past survey data to make predictions about future years, a method that acts like a weather forecast for public health, using today's patterns to guess tomorrow's trends.

The researchers divided the survey data from 2015 through 2024 into three distinct time periods to test their system fairly. They used the earliest years to teach the computer program, the middle years to check if it was learning correctly, and the most recent years, which the program had never seen before, to see if it could actually predict what was happening. This strict separation ensured that the results were not just a lucky guess based on old data. They tested their approach in two places: across the entire nation and specifically within New York State. In both cases, they compared sixteen different types of computer learning algorithms to find which ones worked best. The goal was not to find a single perfect machine, but to build a reliable, open system that other scientists could use and improve upon.

The results showed that the computer models were quite successful at identifying patterns. When the system looked at the most recent data from 2023 and 2024, it could distinguish between people who reported memory worries and those who did not with a level of accuracy that was consistent across different programming languages and methods. The models performed well enough to suggest that they could be useful for public health surveillance, helping officials spot rising trends in cognitive concerns before they become a crisis. However, the researchers were careful to note that these tools are not designed to diagnose individuals. The system is better at ranking groups of people by risk than at telling a single person if they have a specific disease. The accuracy was high enough to be useful for large-scale monitoring, but not high enough to replace a doctor's evaluation for a single patient.

One of the most striking discoveries was how consistent the signals were across different methods. Whether the researchers used a complex computer model, a standard statistical equation, or a visual map of how different health factors connect, the same few answers stood out as the strongest indicators. The single most important factor was whether a person said they had difficulty making decisions due to health problems. People who answered "yes" to this question were far more likely to report memory issues than those who answered "no." Other strong signals included how a person rated their overall health and whether they were dealing with depression. These findings suggest that the feeling of cognitive decline is deeply tied to a person's broader physical and mental well-being, rather than being an isolated symptom.

The study also highlighted a growing trend in the data. The number of people reporting memory and confusion problems increased noticeably in the later years of the survey, particularly in 2023 and 2024. The computer models were able to track this rise and continue to separate those with concerns from those without, even as the overall numbers changed. This ability to adapt to new data is crucial, as it means the tool can remain useful even as the health landscape shifts. The researchers made their entire process, including the code and the data, available to the public so that other scientists can verify the results or apply the same methods to different regions.

Ultimately, this work provides a new way to listen to the public's voice on a massive scale. By turning simple survey answers into a predictive tool, the researchers have created a system that can help health officials understand the scope of cognitive concerns across the country. The tool does not solve the mystery of why people feel this way, nor does it offer a cure, but it offers a clear, data-driven picture of who is struggling and how those struggles are changing over time. It turns a collection of individual worries into a map that can guide future research and public health efforts, ensuring that the needs of the population are seen and understood as they evolve.

Technical Summary: Predicting Subjective Cognitive Decline on Future BRFSS Survey Years

Problem Statement
Subjective Cognitive Decline (SCD), defined as self-reported worsening of memory or confusion, is a critical early marker for Alzheimer's disease and population health concerns. While machine learning (ML) has been applied to SCD prediction using clinical cohorts or neuroimaging, population-based benchmarks utilizing large-scale public health surveillance data remain limited. Existing studies often lack rigorous temporal validation on future survey cycles and do not provide exportable, multi-language software workflows. Furthermore, the challenge of "temporal drift"—where model performance degrades as survey years and population characteristics change—has not been systematically addressed in a reusable framework for SCD.

Methodology
The authors developed a reusable, multi-language (Python and R) machine learning benchmark using data from the Behavioral Risk Factor Surveillance System (BRFSS) Cognitive Decline Module.

Data Sources: Two independent cohorts were analyzed: a national cohort ( $n = 298,944$ ) and a New York State cohort ( $n = 30,366$ ). Data spanned survey years 2015–2024.
Temporal Partitioning: To simulate real-world deployment and prevent data leakage, the study employed a strict chronological split:
- Training: 2015–2019 (used for feature selection and model fitting).
- Validation: 2020–2022 (National) / 2020–2021 (New York) (used for algorithm selection, calibration, and threshold tuning).
- Locked Test: 2023–2024 (used for a single, final performance evaluation).
Preprocessing: The pipeline included train-fitted median imputation and ComBat-lite batch effect correction to address survey-year variations. Synthetic Minority Oversampling Technique (SMOTE) was restricted strictly to training folds during cross-validation to avoid biasing the validation and test sets.
Feature Selection: Nested LASSO (L1-penalized logistic regression) with stratified group cross-validation (grouped by survey year) identified stable predictors. Predictors selected in $\ge$ 50% of outer folds formed the final feature set (24 features for the national cohort; 30 for New York).
Modeling Pipelines: Six analytic pipelines were constructed:
1. Python single-model ML.
2. Python soft-voting ensemble (top 3 classifiers).
3. R tidymodels single-model ML.
4. R tidymodels soft-voting ensemble.
5. ML-integrated knowledge-graph analysis.
6. Data-only knowledge-graph analysis.
  Sixteen supervised classifiers (ranging from linear models to neural networks) were compared. Primary pipelines were prespecified before the test set was unlocked: R gradient boosting for the national cohort and Python AdaBoost for New York.
Evaluation Metrics: Performance was assessed using ROC-AUC, PR-AUC, accuracy, sensitivity, specificity, F1 score, and Brier score. Calibration was evaluated using isotonic regression on validation data.
Interpretability: The study utilized permutation importance, SHAP values, post-hoc Generalized Linear Models (GLMs), and knowledge graphs to identify key predictors.

Key Results

Predictive Performance: All four primary prediction pipelines demonstrated consistent performance on the locked 2023–2024 test set.
- National Cohort: ROC-AUC of 0.770 (95% CI: 0.767–0.773) using R gradient boosting.
- New York Cohort: ROC-AUC of 0.762 (95% CI: 0.746–0.777) using Python AdaBoost.
- Soft-voting ensembles performed similarly to single models, serving as robust sensitivity benchmarks.
Calibration and Stability: Predicted probabilities were reasonably calibrated (Brier scores: 0.118 nationally; 0.112 in New York). Crucially, the separation of risk scores between SCD-positive and SCD-negative respondents persisted across all survey years in both validation and test sets, despite rising SCD prevalence in later years.
Feature Importance: Across all interpretability methods (permutation importance, SHAP, GLMs, and knowledge graphs), "difficulty deciding" (whether health problems make it hard to make decisions) emerged as the strongest predictor. Respondents reporting difficulty had significantly higher odds of SCD (adjusted odds ratio $\approx$ 0.13 for "No" vs. "Yes"). Other consistent top predictors included general health status, mental health days, depression, and functional limitations.
Knowledge Graphs: Both data-only and ML-integrated knowledge graphs placed "difficulty deciding" nearest to the SCD node, reinforcing its centrality in the associational structure of the data.

Significance and Claims
The paper claims to provide the first open, multi-language benchmark for SCD prediction that respects temporal drift in public health surveillance. Its primary contributions are:

Temporal Rigor: By locking the 2023–2024 test set and prespecifying pipelines, the study demonstrates that stable future-year risk ranking is achievable using routine surveillance items, even in the presence of geographic and temporal shifts.
Reproducibility: The release of an end-to-end Python/R workflow allows agencies and researchers to refit, recalibrate, and update models as new BRFSS cycles become available.
Convergent Interpretability: The study identifies a stable "signature" for SCD in population data—centered on decision-making difficulty, depression, and self-rated health—that is consistent across machine learning, statistical regression, and network analysis.
Surveillance Utility: The authors position the tool for population-level screening and surveillance rather than clinical diagnosis. The models provide calibrated probability scores that can track SCD trends over time, offering a complementary signal to brief cognitive screens which often lack sensitivity for early decline.

The authors explicitly note limitations, including the cross-sectional nature of the data (preventing causal inference), the lack of survey weights in the ML pipelines (though used in post-hoc GLMs), and moderate sensitivity at Youden-optimized thresholds, emphasizing that the tool is intended for risk ranking and hypothesis generation rather than individual clinical diagnosis.

Technical Summary: Predicting Subjective Cognitive Decline on Future BRFSS Survey Years

More like this