Performance-based assessments of cognition are less susceptible to demographic effects than traditional memory tests: Evidence from MindCrowd

Using data from the MindCrowd cohort, this study demonstrates that performance-based assessments of cognition are significantly less susceptible to demographic influences such as race, ethnicity, and education compared to traditional verbal memory tests, suggesting they offer a more equitable approach for evaluating cognitive function in diverse populations.

The Gist

Imagine you are trying to judge how well a group of people can solve a puzzle. In the past, doctors and researchers have used standard tests to measure memory and thinking skills in older adults. But there's a catch: these tests often act like a rigged game.

If you grew up with a lot of money, went to great schools, or speak the "right" language, you tend to score higher. If you didn't have those advantages, you might score lower, even if your actual brain power is just as sharp. It's like asking someone to run a race where some people start 50 feet ahead of the finish line because of their education or background. To fix this, scientists usually try to "adjust the score" later, but that's like trying to fix a broken photo after you've already taken the picture—it's messy and often inaccurate.

This new study asks a simple question: Can we design a test that is fair to everyone, regardless of their background?

The Two Contenders

The researchers compared two different ways of testing the brain:

1. The "School Test" (Paired Associates Learning):

   • What it is: This is a classic memory game. You see pairs of words (like "Dog" and "Shoe") and have to remember which word goes with which.
   • The Problem: This test is heavily influenced by your life story. The study found that people with more education, women, and non-Hispanic white people did significantly better. It's like a test that rewards you for having a big vocabulary or knowing how to take a test, rather than just measuring your raw memory.

2. The "Bean Game" (Performance-Based Test):

   • What it is: This is the star of the show. Participants sit at a table with a spoon and some beans. They have to scoop up two beans at a time and move them from a central cup to three different target cups in a specific pattern, using their non-dominant hand. They do this over and over.
   • How it's scored: They don't count how many beans you get right. Instead, they measure how steady your hand is. If your time to move the beans wobbles a lot from one try to the next, that's a sign of cognitive trouble. If your time is steady, your brain is working well.
   • The Magic: This test is like a universal language. It doesn't care if you went to college, what your race is, or what your ethnicity is. It just looks at how smoothly your brain controls your hand.

What They Found

The researchers tested over 1,300 people from all over the US. Here is what happened:

• The "School Test" was biased: It showed big differences based on gender, education, race, and ethnicity. For example, men scored lower than women, and people with less education scored lower than those with degrees. The "gap" between groups was huge.
• The "Bean Game" was fair: It was almost completely immune to these demographic factors. The only tiny difference was that men were slightly less steady than women, but the gap was so small it barely mattered. Race, education, and ethnicity didn't change the scores at all.

The Big Picture

Think of the traditional memory test as a luxury car that runs great on premium fuel (high education) but sputters on regular fuel. The "Bean Game" is like a rugged off-road vehicle. It works just as well on a smooth highway as it does on a bumpy dirt road.

Why does this matter?

• Fairness: It gives everyone a fair shot at being diagnosed with memory issues or cognitive decline, without needing to "correct" their score based on their background.
• Accessibility: You can do this test at home with a cheap kit (beans, a spoon, and cups) mailed to your door. You don't need a fancy computer or a doctor in the room.
• The Future: This could change how we screen for Alzheimer's and other brain diseases. Instead of using tests that accidentally filter out diverse groups of people, we can use a tool that works for everyone, from the CEO to the retiree who never finished high school.

In short, this study suggests that sometimes, the best way to measure a person's mind isn't by asking them to recite facts, but by watching how smoothly they can move a spoonful of beans. It's a simple, fair, and powerful new way to see the brain at work.

Technical Summary

1. Problem Statement

Traditional cognitive assessments used to screen for impairment in older adults (e.g., MMSE, MoCA, RBANS) are heavily influenced by demographic variables such as education level, race, ethnicity, and sex.

• Limitations of Current Methods: To mitigate bias, clinicians often apply demographic corrections. However, these corrections are flawed; for instance, "years of education" fails to capture the quality or accessibility of education. Furthermore, using race as a proxy variable can lead to harmful misinterpretations of test results.
• The Gap: There is a critical need for novel assessment tools that evaluate cognition and daily functioning with minimal susceptibility to demographic factors, thereby allowing for more equitable screening across diverse populations without requiring complex score adjustments.

2. Methodology

The study utilized data from the MindCrowd electronic cohort, an internet-based platform for neuroaging research.

• Participants:
  • Sample Size: $N = 1,326$ adults (Mean age = 61.9 ± 10.9 years).
  • Demographics: 80% Female; 94% White (with smaller representation of Black/African American, Asian, and Multi-race groups); 8.2% Hispanic/Latino; 25% with less than a college degree.
  • Inclusion Criteria: No self-reported history of mental illness, Alzheimer's, Parkinson's, or stroke; normal visual acuity; no peripheral sensory/motor loss; residing in the US.
• Assessments:
  1. Performance-Based Test ("Bean Game"):
     • Task: Participants used a plastic spoon in their non-dominant hand to transport two kidney beans from a central cup to three distal cups arranged at specific angles (-40°, 0°, 40°). The sequence was repeated 5 times (15 reaches total).
     • Scoring: Four trials were completed. The dependent variable was the intraindividual variability (standard deviation) of the trial time across the four trials. Higher variability indicates worse performance.
     • Setting: Conducted remotely at home using mailed kits.
  2. Verbal Memory Test (Paired Associates Learning - PAL):
     • Task: A classic Wechsler Memory Scale paradigm. Participants learned 12 word-pairs per trial over three trials (36 total pairs). They were then cued with the first word and asked to type the second word.
     • Scoring: Total number of correct word pairs recalled (0–36).
• Statistical Analysis:
  • Two separate linear regression models were generated to predict test scores based on demographic factors: Age, Sex, Education, Race, and Ethnicity.
  • The performance-based test scores were log-transformed to meet normality assumptions.
  • Effect sizes (Cohen's $d$ for categorical variables, partial $R^2$ for continuous) were calculated for significant predictors.

3. Key Contributions

• Large-Scale Validation: Unlike previous small-scale studies (<200 participants), this study leveraged a large, heterogeneous cohort ($N > 1,300$) to robustly test demographic effects.
• Remote/At-Home Feasibility: The study validates the "Bean Game" in an unsupervised, remote setting, demonstrating its utility for teleneuropsychology and reducing the cost/barriers of clinical trials.
• Equity in Assessment: It provides empirical evidence that performance-based motor-cognitive tasks may serve as "universal design" tools that do not require demographic corrections, unlike traditional verbal memory tests.

4. Results

The linear regression analyses revealed stark differences in how demographic factors influenced the two tests:

A. Performance-Based Test (Bean Game)

• Significant Predictors: Only Sex and Age showed statistically significant effects.
  • Sex: Males had 9% higher (worse) variability scores than females ($\beta_{exp} = 1.09$, $p = .03$). The effect size was small (Cohen's $d = -0.15$).
  • Age: Every one-year increase in age correlated with a 0.3% increase in score (worse performance) ($p = .04$).
• Non-Significant Predictors: Education ($p = .86$), Race ($p = .59$), and Ethnicity ($p = .67$) had no significant impact on performance.

B. Paired Associates Learning (PAL)

• Significant Predictors: Sex, Education, Race, and Ethnicity all significantly affected scores.
  • Sex: Males scored 4.25 points lower than females (Moderate effect size: $d = 0.58$).
  • Ethnicity: Hispanic/Latino participants scored 3.02 points lower than non-Hispanic/Latino participants ($d = 0.41$).
  • Education: Significant effect ($p < .001$).
  • Race: Significant effect ($p = .03$). Post-hoc analysis showed White participants scored 6.14 points higher than Black/African American participants ($d = -0.83$).
  • Age: Strong negative association with performance ($p < .001$).

5. Significance and Implications

• Equitable Screening: The performance-based "Bean Game" demonstrates that cognitive and functional decline can be assessed with minimal demographic bias. This suggests that such tests could be used to screen diverse populations without the need for demographic corrections, which are often imperfect or controversial.
• Clinical and Research Utility:
  • Teleneuropsychology: The test is suitable for remote administration, potentially lowering the cost of clinical trials and improving recruitment/adherence in diverse groups.
  • Biomarker Correlation: Previous work links this test to medial temporal lobe volume and Alzheimer's biomarkers. Its demographic neutrality makes it a superior tool for pre-screening candidates for expensive biomarker testing (e.g., PET scans, CSF analysis).
• Limitations: The sample was not fully representative of the general US population (over-representation of White and highly educated individuals). However, the study was sufficiently powered to detect the large demographic effects seen in the PAL test, suggesting the lack of effect in the Bean Game is robust. Future research is needed to validate these findings in more diverse global populations and clinical samples with diagnosed cognitive impairment.

Conclusion: The study concludes that performance-based assessments offer a more equitable approach to evaluating cognition in older adults, particularly for diverse populations, by reducing the reliance on demographic corrections that often obscure true cognitive status.