Unifying Acoustic Features and Text with Multimodal LLMs for Neurodegenerative Screening

Technical Summary: NeurMLLM for Neurodegenerative Screening

Problem Statement
Neurodegenerative diseases, specifically Alzheimer's disease (AD) and Parkinson's disease (PD), pose a significant burden on global healthcare systems, often remaining underdiagnosed until irreversible damage occurs. While neuroimaging and clinical tests are definitive, their high cost, invasiveness, and limited accessibility hinder large-scale screening. Voice-based screening offers a scalable, non-invasive alternative, as neurodegeneration disrupts speech motor control and cognitive language planning, manifesting in measurable acoustic and linguistic changes (e.g., slowed articulation, reduced pitch variation).

However, current approaches face two primary challenges:

1. Data Integration: Many existing Large Language Model (LLM) studies are transcript-centric, failing to integrate heterogeneous acoustic representations (spectrograms, MFCCs) with linguistic and demographic contexts within a single unified model.
2. Decision Rigidity: Many LLM-based systems rely on a classification head that maps pooled representations to a fixed label space. This rigid decision layer is difficult to generalize across tasks and is less aligned with token-level multimodal interactions, particularly in resource-constrained settings with small cohorts.

Methodology: NeurMLLM
The authors propose NeurMLLM, an efficient multimodal generative framework designed for fine-grained neurodegenerative staging. The architecture operates as follows:

• Input Modalities: The model processes three distinct inputs:
  • Acoustic Features: Spectrograms and Mel-frequency cepstral coefficients (MFCCs) derived from patient voice data.
  • Linguistic Context: Aligned speech transcripts.
  • Demographic Context: Attributes such as age and gender.
• Architecture:
  • Acoustic Encoding: Two modality-specific Vision Transformers (ViT) encode the spectrogram and MFCC maps.
  • Projection: Lightweight linear layers project these acoustic embeddings into the embedding space of a backbone LLM.
  • Unified Sequence: The projected acoustic embeddings are concatenated with the text embeddings (derived from transcripts and demographic instructions) and special modality tags (e.g., [SPEC], [MFCC], [TEXT]) to form a single token sequence.
  • Fusion: The backbone LLM utilizes self-attention mechanisms to enable cross-modal interaction among all tokens within the unified sequence.
• Training Strategy:
  • Fine-tuning: The model is fine-tuned using Low-Rank Adaptation (LoRA) on the query and value projection matrices of the transformer blocks, keeping the majority of backbone weights frozen to ensure parameter efficiency.
  • Generative Objective: Instead of using a classification head, the staging task is formulated as constrained label-token generation. The model is instruction-tuned to autoregressively predict a specific label token (e.g., "MCI", "AD", "CN") from a predefined set, guided by task-specific prompts.

Key Contributions

1. Multimodal Architecture: Introduction of a framework that jointly models paralinguistic (acoustic), linguistic (transcript), and population (demographic) factors within a single architecture, addressing the gap in transcript-centric LLM studies.
2. Generative Staging Paradigm: Demonstration that instruction-tuned LLMs performing constrained label-token generation outperform both classical machine learning methods and LLM-based approaches utilizing classification heads.
3. Extensive Validation: Comprehensive evaluation on the Bridge2AI-Voice dataset (v3.0.0) for multi-class staging of AD and PD, showing consistent superiority over baselines.

Experimental Results
The framework was evaluated on the Bridge2AI-Voice dataset, involving 156 participants for AD (stratified into CN, MCI, AD) and 167 for PD (stratified into Healthy Control, Early Stage, Advanced Stage).

• Performance: NeurMLLM achieved the strongest results across both diseases. For AD staging, it improved macro-AUROC from 0.852 (ClsHead baseline) to 0.917 and Accuracy from 0.740 to 0.823. For PD staging, it achieved a macro-AUROC of 0.872 and Accuracy of 0.735.
• Modality Analysis: While acoustic-only inputs performed strongly, the multimodal fusion of acoustic, textual, and demographic data yielded the best results, indicating that linguistic information provides complementary context that improves recall, particularly in borderline cases.
• Backbone Sensitivity: Experiments with different LLM backbones (Llama-3.2-3B, Qwen2.5-3B) indicated that instruction-tuned models (e.g., Llama-3.2-3B-Instruct) generally align better with the staging objective than base models, particularly for AD.

Significance and Claims
The paper claims that reframing neurodegenerative staging as a constrained label-token generation task, combined with parameter-efficient instruction tuning, offers a simple and effective alternative to traditional classification heads. The authors argue that generative decoding reuses pre-trained output projections, preserving consistency with the LLM's pretraining objective and avoiding the challenges of learning a new classification head from scratch on small cohorts.

The results suggest that multimodal LLMs have high potential for accessible, scalable neurodegenerative screening. By integrating acoustic, linguistic, and demographic data, NeurMLLM supports fine-grained staging with improved accuracy and sensitivity to minority classes. The authors note that while the small cohort size introduces variance, the findings serve as encouraging signals for the viability of this approach, with future work potentially extending to additional modalities (e.g., neuroimaging) and more complex generative outputs (e.g., clinical summaries).