Wearable and Interview-based Assessment of Psychological Risk in Alzheimers Caregivers: Machine Learning vs. Large Language Models

This study compares traditional machine learning and large language models for assessing psychological risk in Alzheimer's caregivers, finding that multimodal data integration favors traditional models while interview-only text yields superior results with LLMs, with performance heavily influenced by prompting strategies and the specific psychological construct being predicted.

The Gist

Imagine you are trying to figure out if a person caring for a spouse with Alzheimer's is secretly struggling with stress, feeling overwhelmed, or lonely. Usually, we ask them to fill out long, boring questionnaires. But people get tired of filling those out, and they might not answer honestly or accurately in the moment.

This paper is like a detective story where the researchers tried two different "super-sleuths" to solve the mystery of caregiver well-being without relying solely on the questionnaires.

The Two Super-Sleuths

1. The Traditional Machine Learning Model (The "Data Cruncher"): Think of this as a very organized accountant. It looks at hard numbers: how many steps did the person take? How was their heart rate? Did they sleep well? It's great at spotting patterns in numbers but can't really "understand" a story.
2. The Large Language Model (The "Empathic Listener"): This is like a wise, well-read counselor (using AI like GPT-4o or Gemini). It reads the transcripts of interviews where the caregiver talks about their day. It's amazing at understanding tone, emotion, and the "vibe" of what someone is saying, but it sometimes gets confused by raw numbers.

The Three Clues (Data Types)

The researchers gave these sleuths three different types of clues to work with:

• The Fitness Tracker Clues (Wearables): Data from a Fitbit, like heart rate, steps, and sleep patterns.
• The Interview Clues (Text): Transcripts of a 30-minute conversation where the caregiver talks about their life.
• The Mixed Bag (Multimodal): A combination of both the fitness tracker data and the interview text.

The Three Mysteries to Solve

They tried to solve three specific problems:

1. Perceived Stress (PSS): How overwhelmed does the caregiver feel right now?
2. Caregiver Burden (ZBI): How heavy does the responsibility feel?
3. Loneliness (UCLALS): How isolated do they feel?

What Did They Find?

1. The "Stress" Mystery was the Easiest
The researchers found that "Perceived Stress" was the easiest to predict. It's like a loud alarm bell; it shows up clearly in both the numbers (heart rate, sleep) and the words (people saying they are "stressed" or "rushing"). Both the Data Cruncher and the Empathic Listener did a good job here.

2. The "Burden" and "Loneliness" Mysteries were Harder
Figuring out if someone feels "burdened" or "lonely" was much trickier.

• The Data Cruncher worked best when it had both the fitness tracker data and the interview text. It was like trying to solve a puzzle with two different sets of pieces; when you put them together, the picture became clear.
• The Empathic Listener (the AI chatbot) worked best when it only had the interview text. It didn't need the numbers; it just needed to hear the story. When you forced it to look at the numbers, it actually got a bit confused, like a poet trying to read a spreadsheet.

3. The "How You Ask" Matters (Prompt Engineering)
The researchers discovered that how you ask the AI to solve the problem changes the answer.

• If you tell the AI, "Pretend you are the caregiver and tell me how you feel," it sometimes gives a different answer than if you say, "Pretend you are a doctor looking at this patient's file."
• It turns out, the way you phrase the instructions (the "prompt") is like tuning a radio; if you tune it slightly wrong, the signal gets staticky.

4. The Winner Depends on the Job

• Gemini 2.0 was the most stable and reliable AI overall.
• GPT-4o was great at reading the interview text but struggled when given the fitness tracker numbers.
• Llama 4 was okay but generally didn't perform as well as the others.

The Big Takeaway

The paper concludes that there isn't one "magic bullet" AI.

• If you want to use numbers (like heart rate), you need a traditional computer model.
• If you want to use words (like interview transcripts), a modern AI chatbot is your best bet.
• If you want the best possible accuracy, you need to combine the numbers and the words, but you have to use the traditional computer model to do the combining, not the chatbot.

Essentially, the researchers found that to understand a caregiver's hidden struggles, you need the right tool for the right job: a calculator for the numbers and a listener for the stories. Mixing them up requires a specific kind of "translator" (the traditional model) to make sense of both.

Technical Summary

Technical Summary: Wearable and Interview-based Assessment of Psychological Risk in Alzheimer's Caregivers

Problem Statement
Spousal caregivers of individuals with Alzheimer's disease and related dementias (AD/ADRD) frequently experience elevated perceived stress, caregiver burden, and loneliness, which are linked to adverse physiological and psychological health outcomes. Current assessment methods rely heavily on infrequent self-report instruments (e.g., Perceived Stress Scale, Zarit Burden Interview, UCLA Loneliness Scale), which may miss meaningful daily fluctuations. While digital health technologies offer continuous monitoring via wearables and linguistic analysis of interviews, there is a lack of systematic comparison between traditional machine learning (ML) approaches and Large Language Models (LLMs) in this specific multimodal context. Furthermore, it remains unclear how different data modalities (wearable-derived physiological/behavioral data vs. interview transcripts) and prompting strategies influence the detection of these distinct psychological constructs.

Methodology
The study utilized a dataset of 32 spousal caregivers who wore Fitbit devices for seven days and completed a 30-minute semi-structured interview.

• Data Modalities:
  • Wearable Data: Minute-level heart rate and step counts were processed to extract 104 features, including sleep metrics (duration, regularity), daily activity statistics (steps, active/sedentary time), and rhythm features (ultradian, circadian, and infradian patterns via M10/L5, Relative Amplitude, and Cosinor modeling).
  • Interview Data: Audio interviews were transcribed, cleaned, and segmented into 28 structured question-response units.
• Ground Truth: Participants completed the PSS-10, ZBI-13, and UCLALS-3 scales, which were binarized into high-risk and low-risk groups based on established cutoffs.
• Models Evaluated:
  • Traditional ML: Support Vector Machines (SVM), XGBoost, Random Forests, and K-Nearest Neighbors (KNN) were trained using Leave-One-Out Cross-Validation (LOOCV) due to the small sample size ($N=32$). Feature selection was performed within each fold.
  • Large Language Models (LLMs): Gemini 2.0, Llama 4, and GPT-4o were evaluated using zero-shot inference.
• Prompt Engineering: Two primary perspectives were tested: (1) Caregiver-centered (model adopts the caregiver's viewpoint) and (2) Psychometrician-centered (model acts as an expert assessor). These were combined with two task framings: (A) Direct Classification (binary output) and (B) Score Prediction (predicting scale scores then thresholding). "Informed" prompts included full questionnaire details, while "Uninformed" prompts provided only cutoff values.
• Experimental Configurations: Models were tested under three modality settings: Wearable-Only, Interview-Only, and Multimodal (combined wearable features and interview text).

Key Results

• Model Performance by Modality:
  • Traditional ML: Achieved the best performance in the Multimodal setting. For Perceived Stress Scale (PSS), the multimodal TF-IDF + XGBoost configuration reached 0.81 accuracy and 0.96 recall. This suggests that structured physiological features complement linguistic information for traditional classifiers.
  • LLMs: Achieved their strongest performance with Interview-Only inputs. For PSS, GPT-4o achieved 0.79 accuracy in the Interview-Only setting. LLMs generally underperformed in the Wearable-Only setting, particularly GPT-4o and Llama 4, indicating difficulty in interpreting raw numerical feature descriptions without narrative context.
• Construct Predictability:
  • PSS (Perceived Stress) was the most predictable construct across all models and modalities.
  • ZBI (Caregiver Burden) and UCLALS (Loneliness) were more challenging. LLMs showed relatively stronger performance on ZBI compared to traditional models in certain configurations, but both constructs remained harder to detect than stress.
• Feature Importance (SHAP Analysis):
  • PSS: Driven by linguistic features related to time ("time", "long") and activity metrics (lower max step counts, reduced activity during least active hours).
  • ZBI: Primarily associated with heart rate variability (HRV) and rhythm features (morning/afternoon HRV, 16-hour mesor), alongside linguistic markers of strain ("stressed", "think").
  • UCLALS: Driven by sleep disturbance (awake minutes) and circadian rhythm delays, alongside linguistic features related to communication ("things", "say").
• Prompt Engineering: Prompting strategies significantly influenced LLM performance. For ZBI, the Caregiver-centered Direct Classification (C C) strategy yielded the highest accuracy (0.81). For UCLALS under Wearable-Only inputs, only the Caregiver-centered Score Prediction (C S) strategy produced reasonable results; other strategies failed, suggesting that uncertainty-aware reasoning (hedging) aligns better with the subjective nature of loneliness.
• Model Comparison: Gemini 2.0 demonstrated the most stable and consistent overall performance. GPT-4o excelled in PSS prediction but struggled with Wearable-Only inputs. Llama 4 showed slightly lower performance on PSS and ZBI but comparable results on UCLALS.

Significance and Claims
The paper claims to provide a preliminary systematic comparison of traditional ML and LLMs for identifying psychological risk in AD/ADRD caregivers, highlighting that model choice must be aligned with data modality.

• Complementarity: Traditional ML models benefit from the fusion of physiological and linguistic data, whereas LLMs are most effective when leveraging the rich contextual and emotional cues found in narrative interview text.
• Prompt Sensitivity: The study demonstrates that prompt engineering (perspective and task framing) is a critical variable that can substantially alter LLM performance, with no single strategy universally superior across all scales and modalities.
• Construct Specificity: The findings suggest that different psychological constructs manifest differently in behavioral and linguistic data; stress is more globally detectable across modalities, while burden and loneliness require more specific feature sets or modeling approaches.
• Limitations: The authors acknowledge the small sample size ($N=32$) limits generalizability and note that the multimodal fusion method was intentionally simple to maintain interpretability. They also highlight that the study focused on binary classification, which may oversimplify the nuanced experiences of caregivers.

The study concludes that developing digital health tools for caregiver risk identification requires careful consideration of the interplay between the computational model, the data modality, and the prompting strategy, rather than assuming a single approach fits all psychological constructs.