MEMOR-E: In-Context and Fine-Tuned LLM Personalization for Alzheimer's Assistive Robotics

Technical Summary: MEMOR-E – In-Context and Fine-Tuned LLM Personalization for Alzheimer's Assistive Robotics

Problem Statement
Alzheimer's disease (AD) causes progressive declines in memory, language, and executive functions, increasing reliance on caregivers and placing significant burdens on families and healthcare systems. While socially assistive robotics (SAR) have shown promise in reducing stress and improving engagement, existing systems often lack mobility, personalization, or integration with modern language-based reasoning. Furthermore, current systems rarely provide transparent, interpretable insights into cognitive performance to support caregiver oversight without attempting medical diagnosis.

Methodology
The paper presents MEMOR-E, a mobile quadruped robot equipped with a head-mounted tablet interface. The system operates on a two-step, privacy-preserving framework designed to detect cognitive severity and plan assistive interactions without transmitting raw patient data.

1. Hardware and Architecture:

   • Platform: Built on a Unitree Go2 quadruped robot for stable indoor mobility and social embodiment.
   • Interface: A touchscreen tablet delivers reminders, cognitive games, and visual prompts.
   • Processing: All computation occurs locally using ROS 2 for navigation and an Intel RealSense camera for obstacle avoidance. The system runs a local Qwen 2.5 (7B) Large Language Model (LLM) and Longformer classifiers, ensuring no patient data leaves the device.

2. Step 1: Explainability-Driven Cognitive Signal Extraction:

   • Data: The system utilizes transcripts from the TalkBank DementiaBank Pitt corpus, covering four neuropsychological tasks: Cookie Theft Picture Description, Story Recall, Verbal Fluency, and Sentence Construction. Synthetic healthy controls were generated via LLM for Recall, Fluency, and Sentence tasks to balance datasets.
   • Classification: Independent binary classifiers (using allenai/longformer-base-4096) were trained for each task to estimate the probability of AD ($P(AD)$) from transcripts.
   • Explainable AI (XAI): To ensure privacy and interpretability, raw transcripts are not passed to the LLM. Instead, Integrated Gradients are used to extract token-level attributions. These attributions are aggregated into linguistically motivated "buckets" (e.g., disfluencies, lexical content, punctuation) to generate privacy-preserving statistics (e.g., disfluency-to-content ratio, entropy).
   • Severity Index: A deterministic severity index is calculated based on the mean predicted probability, cross-fold vote rate, and probability variance, avoiding LLM-driven variability in risk assessment.

3. Step 2: LLM-Based Assistive Feature Planning:

   • Input: Only the structured numerical summaries (XAI statistics and severity metrics) are passed to a local Qwen 2.5 (7B) LLM.
   • Mechanism: The LLM is not fine-tuned on medical data. Instead, it utilizes In-Context Learning (ICL) with prompt-based conditioning to map severity signals to non-diagnostic assistive recommendations (e.g., specific reminders, cognitive games, memory cues).
   • Persona Simulation: A feasibility study employed a "Patient LLM" and a "Categorizer LLM" to simulate stage-consistent cognitive errors and generate domain-level summaries (episodic, semantic, working memory) for different impairment stages.

Key Contributions

• System Design with XAI: An integrated hardware/software platform combining a mobile quadruped robot with a closed-loop, human-supervised architecture that produces transparent, human-readable interpretations of cognitive task outcomes.
• Stage-Adaptive Interaction: A simplified graphical interface and interaction logic aligned with cognitive challenges across AD stages, supported by interpretable feedback for caregiver oversight.
• LLM-Based Cognitive Analysis: A feasibility study demonstrating the use of LLMs to emulate stage-consistent patient responses and categorize cognitive errors, focusing on interpretable domain-level summaries rather than diagnostic predictions.

Results

• Classification Performance: On the "Cookie Theft" task (using real AD and real healthy controls), the model achieved an AUC of 0.878 with balanced sensitivity (0.741) and specificity (0.856). The model was conservative, reducing false positives.
• Synthetic Data Artifacts: Tasks using synthetically generated healthy controls (Recall, Fluency, Sentence) achieved perfect separation (1.000 AUC/Accuracy). The authors explicitly interpret these results as feasibility validation rather than evidence of real-world generalization, noting potential distributional artifacts.
• Attribution Analysis: XAI revealed that while lexical content tokens were the primary discriminative signal (approx. 70% of attribution mass), AD predictions showed elevated attribution mass on disfluency markers (fillers like "uh/um" and pauses) and reduced specificity in noun-verb relations. This aligns with established linguistic markers of AD.
• Feasibility of Persona Simulation: The "Patient LLM" demonstrated a monotonic increase in errors from Stage 1 to Stage 5, and the "Categorizer LLM" successfully generated interpretable domain summaries consistent with the simulated impairment levels.

Significance and Claims
The paper claims that MEMOR-E successfully demonstrates a privacy-preserving and explainable framework for cognitively adaptive assistive robotics. The system's significance lies in its ability to:

1. Generate stage-aware, non-diagnostic cognitive summaries that support personalized assistive interactions without replacing clinical assessment.
2. Translate model outputs into transparent, human-readable evidence via XAI, enabling caregiver oversight and fostering trust in human-robot interaction.
3. Operate with conservative autonomy, where all actions are non-invasive, reversible, and mediated through a tablet, ensuring safety for cognitively vulnerable users.

The authors emphasize that the system is designed to augment daily support and caregiver awareness rather than provide medical diagnoses. Future work is identified as necessary to validate these findings with real-world longitudinal studies, real control data, and qualitative assessments of caregiver trust and workload.