Technical Summary: PROCESS-2 Benchmark Speech Corpus

Problem Statement

The detection of early cognitive decline, including Mild Cognitive Impairment (MCI) and dementia, is a critical global health challenge. While speech production engages multiple cognitive systems (semantic retrieval, working memory, executive control) and offers a non-invasive biomarker for early detection, progress in automatic cognitive assessment has been constrained by data limitations. Existing datasets often suffer from:

• Low Ecological Validity: Many rely on controlled clinical environments with standardized elicitation protocols, limiting generalizability to real-world telehealth deployments.
• Narrow Scope: Early resources often focus on binary distinctions (Alzheimer's vs. Healthy Controls) or single tasks, lacking representation of intermediate states like MCI or diverse cognitive domains.
• Accessibility and Reproducibility: Large-scale, clinically annotated cohorts are frequently restricted, preventing reproducible machine learning research.
• Lack of Multimodal Diversity: Few datasets balance cohort scale, task diversity, and clinically grounded diagnostic annotations while capturing the acoustic variability of remote, consumer-device-based recording.

Methodology

The authors introduce PROCESS-2, a large-scale speech dataset collected via the CognoMemory digital assessment platform (formerly CognoSpeak). The methodology emphasizes remote, real-world data acquisition to preserve ecological validity.

Data Collection and Participants

• Cohort: 400 older adults recruited across the UK, comprising 50 participants with dementia, 150 with MCI, and 200 cognitively healthy controls (HC).
• Recruitment: Participants were recruited through NHS memory services, primary/secondary care referrals, and research registries (e.g., Join Dementia Research). Diagnoses were verified by qualified clinicians using standard clinical procedures (history, cognitive testing, imaging) independent of computational analysis.
• Environment: Data was collected remotely in participants' natural environments (homes, community locations) using their personal consumer devices (laptops, tablets) via a browser-based interface. This intentionally preserves acoustic variability (background noise, device heterogeneity) typical of real-world telehealth.

Speech Elicitation Tasks

The dataset includes three complementary tasks targeting distinct cognitive processes:

1. Semantic Fluency Task (SFT): Participants list as many words as possible in a specific category (animals) within 60 seconds.
2. Phonemic Fluency Task (PFT): Participants generate words starting with a specific letter ("P") within 60 seconds.
3. Cookie Theft Description (CTD): Participants describe a complex visual scene without a strict time limit, capturing spontaneous narrative speech (typically 60–75 seconds).

Data Processing and Curation

• Recording: Audio was captured via WebRTC, transcoded to mono .wav format (16 kHz, 128 kbps), and normalized to -23 LUFS (EBU R128 standard).
• Transcription: All recordings were manually transcribed by professional annotators and study authors. Transcripts preserve conversational features, including disfluencies, speaker identifiers, and pauses.
• Metadata: A structured metadata table (meta-info.csv) links recordings to participant demographics (age, gender), diagnostic labels, and available cognitive scores (MMSE for a subset).
• Privacy: Data is pseudonymized; personally identifiable information is removed. Access is controlled via Hugging Face under a data use agreement to protect participant privacy.

Technical Validation

The dataset underwent rigorous validation across five dimensions:

1. Demographic Integrity: Statistical analysis (ANOVA, Kruskal-Wallis) confirmed that age distributions are comparable across diagnostic groups, minimizing age as a confounding factor. Gender distribution showed a small but significant association with diagnosis (higher male proportion in dementia), but training and test splits remained balanced.
2. Clinical Validity: MMSE scores showed clear, statistically significant separation between diagnostic groups (Dementia < MCI < HC), confirming the clinical grounding of the labels.
3. Recording Stability: Signal-to-Noise Ratio (SNR) and task durations were consistent across groups and splits, indicating stable acquisition despite heterogeneous devices.
4. Embedding-Space Structure: Geometric analysis of embeddings (distance to the HC centroid) revealed that linguistic representations (from manual transcripts) showed significant separation between groups, particularly for SFT and CTD tasks, whereas raw acoustic embeddings showed less distinct separation.
5. Benchmark Performance: Baseline modeling experiments compared classical models (Logistic Regression, MLP) against Transformer-based LLMs (DistilBERT, RoBERTa).

Key Results

• Dataset Scale: The release contains 1,200 audio recordings (approx. 21 hours total) and corresponding transcripts, organized into predefined 80% training and 20% test splits.
• Model Performance:
  • Linguistic vs. Acoustic: Linguistic features (manual transcripts) consistently outperformed acoustic features (ComParE, Wav2Vec 2.0, Whisper) in classification tasks.
  • Model Architecture: Transformer-based LLMs significantly outperformed classical models. In 2-way classification (Case vs. Control), LLMs achieved a peak Macro F1 of 0.85 (DistilBERT on SFT manual transcripts), compared to 0.76 for the best classical model.
  • 3-Way Classification: In the more challenging 3-way task (Dementia vs. MCI vs. HC), LLMs achieved a peak Macro F1 of 0.59, demonstrating superior ability to distinguish graded cognitive severity compared to classical approaches.
  • Regression: For MMSE score prediction, DistilBERT achieved the lowest RMSE (3.87).
• Error Analysis: Confusion matrices indicated that misclassifications predominantly occurred between adjacent diagnostic categories (e.g., MCI predicted as Dementia or HC), suggesting the model captures a graded continuum of cognitive decline rather than making extreme errors.
• Robustness: Performance remained stable across training and test sets, confirming the absence of dataset leakage. The dataset supports robust modeling even when using ASR-generated transcripts, despite higher word error rates (~40-60%) due to preserved conversational disfluencies.

Significance and Claims

The paper positions PROCESS-2 as a reproducible benchmark resource designed to bridge the gap between controlled clinical research and real-world digital health deployment. Its primary contributions are:

1. Ecological Validity: Unlike previous corpora, PROCESS-2 captures the natural variability of remote speech (background noise, consumer devices, spontaneous interaction), enabling the development of models robust to real-world deployment conditions.
2. Clinical Grounding: The dataset features clinically verified diagnostic labels (Dementia, MCI, HC) and includes a significant representation of the intermediate MCI stage, addressing a gap in binary-focused datasets.
3. Multimodal and Multi-Task Design: By integrating three distinct speech elicitation paradigms (SFT, PFT, CTD) with manual transcripts and metadata, it supports diverse research into lexical retrieval, executive function, and discourse organization.
4. Reproducibility: The provision of predefined splits, standardized preprocessing pipelines, and open-source code (Apache 2.0) facilitates independent verification and benchmarking.
5. Responsible Access: The controlled access framework via Hugging Face balances the need for open research with the ethical imperative to protect participant privacy, acknowledging that raw voice data cannot be fully anonymized.

The authors conclude that PROCESS-2 provides a reliable, high-quality foundation for advancing speech-based biomarkers for early cognitive impairment, demonstrating that clinically meaningful group separation and stable modeling performance are achievable even with ecologically valid, remote data.