Can we trust AI to detect healthy multilingual English speakers among the cognitively impaired cohort in the UK? An investigation using real-world conversational speech

1. Problem Statement

The rapid rise in dementia prevalence among ethnic minority populations in the UK (particularly Black and Asian communities) highlights a critical gap in current diagnostic tools. Existing AI models for detecting cognitive decline (such as Mild Cognitive Impairment [MCI] and Alzheimer's Dementia) are predominantly trained on monolingual, native English speakers.

• The Core Issue: There is a lack of evidence regarding whether these AI systems are robust enough to handle multilingual English speakers (individuals speaking English as an additional language) and speakers with specific regional accents (e.g., South Yorkshire vs. West Yorkshire).
• The Risk: If AI models exhibit bias against these groups, they may lead to false positives (misclassifying healthy multilingual individuals as cognitively impaired) or false negatives, exacerbating health inequalities and limiting the clinical utility of remote screening tools like CognoMemory.

2. Methodology

A. Data Collection (The CognoMemory Dataset)

The study utilized a novel dataset comprising 1,395 participants contributing over 263 hours of conversational speech.

• Monolingual Group: Recruited nationally across the UK. Includes Healthy Controls (HC), MCI, and Dementia patients.
• Multilingual Group: Recruited from four community centers in Sheffield and Bradford. Includes speakers of Somali, Chinese, and South Asian languages (Hindi, Urdu, Punjabi, Mirpuri, Arabic).
  • Note: Due to language barriers, the cognitively impaired multilingual cohort could not complete the English-based assessments. Therefore, the multilingual test set consisted exclusively of Healthy Controls (HC).
  • Accent Stratification: Multilinguals were further categorized by accent: South Yorkshire (Sheffield) and West Yorkshire (Bradford).
• Assessment Protocol: Participants interacted with a virtual agent answering 14 clinically validated memory-probing prompts (covering memory recall, fluency, picture description, and reading tasks).
• Transcription: Audio was transcribed using three state-of-the-art ASR systems: Whisper-medium, Wav2Vec 2.0, and NVIDIA NeMo.

B. Experimental Setup

The researchers employed four complementary evaluation techniques to assess bias:

1. ASR Performance: Measured via Word Error Rate (WER) comparing ASR transcripts against manual transcripts (available for 349 participants).
2. Classification: Trained Support Vector Machines (SVM) and Large Language Models (LLMs: BART, RoBERTa, DistilBERT) to classify speakers into Dementia, MCI, or HC.
   • Training: Used a fixed training set (Monolingual Dementia/MCI/HC) combined with folds of multilingual data.
   • Testing: Tested on held-out monolingual and multilingual HC groups.
   • Datasets: Evaluated on both the CognoMemory dataset and the public DementiaBank dataset (binary labels only).
3. Regression: Used Support Vector Regression (SVR) to predict Mini-Mental State Examination (MMSE) scores based on acoustic and linguistic features.
4. Qualitative Analysis: Applied TF-IDF (Term Frequency-Inverse Document Frequency) to analyze linguistic patterns and identify specific words driving classification decisions.

C. Feature Extraction

• Acoustic Features: eGeMAPS and ComParE feature sets.
• Linguistic Features: TF-IDF and Bag-of-Words (BoW) derived from ASR transcripts.
• Fusion: Acoustic and linguistic features were fused to train classifiers and regressors.

3. Key Results

A. ASR Performance (Word Error Rate)

• Overall Bias: There was no statistically significant difference in WER between monolingual and multilingual speakers across the three ASR systems (p > 0.05).
• System Comparison: NeMo performed best overall (lowest WER and standard deviation), followed by Whisper. Wav2Vec 2.0 showed significantly higher error rates.
• Task Specificity: All ASR systems struggled more with fluency tasks (Q10, Q11), regardless of language background.

B. Classification Performance (Bias Detection)

• Acoustic vs. Linguistic: Models using acoustic features showed consistent performance across groups with no significant bias. However, models using linguistic features exhibited significant bias against multilingual speakers (p < 0.01).
• Task Specificity: The bias was most pronounced in memory, fluency, and reading tasks.
• Misclassification Severity: Healthy multilingual speakers, particularly those with South Yorkshire accents, were significantly more likely to be misclassified as having Dementia (severe decline) rather than MCI or HC.
  • Statistic: Only 16% of younger monolinguals were misclassified as impaired, compared to 35% of South Yorkshire Asian multilinguals (25% misclassified as Dementia).
• Dataset Impact: Bias was stronger when models were trained on the public DementiaBank dataset compared to the larger, more diverse CognoMemory dataset.

C. Regression (MMSE Prediction)

• Models predicted MMSE scores with significantly higher error (RMSE) for multilingual speakers compared to monolinguals (p ≈ 0), indicating systematic underestimation of cognitive health in multilingual groups.

D. Qualitative Analysis (TF-IDF)

• Linguistic Markers: The analysis revealed that healthy multilingual participants frequently referenced their countries of origin (e.g., Pakistan, Lahore) and places of origin when answering memory prompts.
• Contrast: Healthy monolinguals referenced UK-specific locations and prominent political figures (e.g., British Prime Ministers).
• Implication: The AI models likely interpreted these culturally specific references as "irrelevant" or "confused" content, leading to false positive diagnoses of cognitive decline.

4. Key Contributions

1. First UK-based Investigation: This is the first study to explicitly investigate and quantify bias in AI-driven cognitive decline detection specifically for multilingual ethnic minority populations within the UK context.
2. Novel Dataset: Creation of a large-scale, real-world conversational speech dataset (CognoMemory) including diverse accents (South/West Yorkshire) and languages (Somali, Chinese, South Asian), addressing a major data gap in the field.
3. Bias Identification: Demonstrated that while ASR systems are generally robust, downstream classification models using linguistic features are highly biased against multilingual speakers, often misdiagnosing them with severe dementia.
4. Accent Sensitivity: Highlighted that specific regional accents (South Yorkshire) combined with multilingualism create a "double burden" of bias in AI diagnostics.

5. Significance and Conclusion

• Clinical Reliability: The study concludes that current off-the-shelf AI tools are not yet reliable for diagnostic use in multilingual ethnic minority populations in the UK. Using them could lead to significant over-diagnosis of dementia in healthy individuals.
• Root Cause: The bias stems primarily from the models' inability to distinguish between cognitive decline and cultural/linguistic differences (e.g., referencing foreign countries or using non-standard vocabulary).
• Future Directions: The authors advocate for:
  • Recruiting cognitively impaired multilingual participants to create balanced training data.
  • Developing bias-mitigated models that are culturally sensitive.
  • Integrating culturally appropriate screening tools (e.g., RUDAS, MCE) alongside AI.
  • Fine-tuning ASR systems specifically for these dialects and accents.

In summary, while AI shows promise for remote dementia screening, this study serves as a critical warning that without addressing linguistic and cultural bias, these tools risk widening health disparities rather than solving them.