Generative AI Purpose-built for Social and Mental Health: A Real-World Pilot

Technical Summary: Generative AI Purpose-built for Social and Mental Health: A Real-World Pilot

Problem Statement
Mental health challenges remain a leading cause of global disability, with significant barriers to care including geographic remoteness, financial constraints, practitioner shortages, and stigma. While digital interventions have improved accessibility, most research has focused on live video therapy or rule-based chatbots with limited personalization. Although preliminary studies suggest that Large Language Model (LLM) chatbots offer promise for natural, personalized conversations, there is a lack of evidence regarding their feasibility, safety, and effectiveness in real-world, longitudinal settings outside of controlled Randomized Controlled Trials (RCTs). Specifically, there is a need to evaluate whether foundation models trained on clinical data can maintain safety guardrails, sustain user engagement, and produce heterogeneous symptom trajectories in a naturalistic environment.

Methodology
This study was a single-arm, longitudinal naturalistic pilot evaluating "Ash," a foundation model specifically trained for mental health support.

• Participants: The study recruited 1,808 users via Meta ads, resulting in a final analytic sample of $N=305$ adults (ages 18–73, mean age 41.7) located in the United States. Inclusion criteria required English proficiency, smartphone access, and baseline scores of $\ge$10 on either the Patient Health Questionnaire (PHQ-9) or Generalized Anxiety Disorder Scale (GAD-7). Exclusion criteria included active suicidality (C-SSRS items 3–6), psychotic disorders, severe substance use, or medical conditions better explaining symptoms.
• Intervention: Users interacted with Ash via text or voice on mobile devices. The model was pre-trained on over 100,000 hours of de-identified mental health transcripts (clinical interviews, case studies) and aligned via supervised fine-tuning with clinician annotations. The training data encompassed diverse therapeutic modalities (e.g., CBT, DBT, ACT, Motivational Interviewing). The system employed a two-pass safety architecture: an embeddings-based classifier for high-recall detection of harmful content, followed by an LLM safety verification layer to block or replace flagged messages.
• Procedure: Participants provided opt-in consent and completed baseline assessments. Measures were repeated every two weeks for six weeks, with a final follow-up at 10 weeks. Assessments included PHQ-9, GAD-7, Behavioral Activation for Depression Scale (BADS), UCLA Loneliness Scale, Multidimensional Scale of Perceived Social Support (MSPSS), and the Working Alliance Inventory–Short Report (WAI-SR).
• Analytic Strategy:
  • Safety: Clinicians reviewed all flagged sessions (1.02% of total) to verify escalation protocols. Model appropriateness was tested against 10 clinical scenarios (e.g., delusions, mania) with 100 generations each.
  • Trajectory Modeling: Latent Growth Mixture Modeling (LGMM) was used to identify heterogeneous change patterns across depression, anxiety, social support, and behavioral activation simultaneously.
  • Engagement Analysis: Mixed-effects models examined the relationship between engagement metrics (active days, minutes, words exchanged) and symptom reduction ($\Delta$PHQ-9), stratified by trajectory class.
  • Predictors: Multinomial logistic regression identified baseline demographics and Week 2 therapeutic alliance as predictors of trajectory membership.

Key Results

• Clinical Outcomes: Significant reductions in depressive and anxiety symptoms were observed. At the 10-week follow-up, Cohen's $d$ effect sizes were 0.93 for PHQ-9 (95% CI [0.76, 1.13]) and 0.79 for GAD-7 (95% CI [0.65, 0.96]). Deterioration rates were low (1.83% for GAD-7, 2.29% for PHQ-9).
• Trajectory Heterogeneity: LGMM identified three distinct trajectories across all four clinical outcomes:
  1. Rapid Improving: 9.5% ($n=29$)
  2. Improving: 42.3% ($n=129$)
  3. Non-responders: 48.2% ($n=147$)
     Baseline severity was comparable across groups; differences were driven by the rate and synchrony of change.
• Social and Functional Outcomes: Improvements in loneliness, perceived social support, and social interaction counts were sustained at the 10-week follow-up.
• Safety and Appropriateness: Of 7,493 sessions, 76 (1.02%) were flagged for risk; all were correctly escalated to human resources. A random sample of 1,200 sessions revealed no missed crisis messages (false negatives). In stress-testing against 10 clinical scenarios, the model achieved an 87.7% pass rate, comparable to human therapist benchmarks.
• Engagement and Alliance: Average usage was 9.02 hours ($SD=11.5$). Higher engagement (active days, minutes, words) was significantly associated with greater PHQ-9 reductions, particularly for the "Improving" and "Rapid Improving" classes. The therapeutic alliance (WAI-SR) was comparable to traditional care (mean score ~4.0) and predicted membership in the improving trajectories.

Significance and Claims
The authors claim this study provides the first real-world, naturalistic evidence that a clinically grounded generative AI foundation model is feasible, engaging, and acceptable for mental health support. The study demonstrates that:

1. Safety is manageable: Automated guardrails can effectively detect and escalate risk in a live setting without human-in-the-loop intervention for every message.
2. Efficacy is observable: Symptom reductions comparable to RCT findings can be achieved in unstructured, real-world usage without daily reminders or structured monitoring.
3. Mechanisms align with therapy: The relationship between therapeutic alliance, engagement, and symptom improvement mirrors findings in traditional psychotherapy, suggesting that AI models can function as relational bridges rather than mere substitutes.
4. Scalability potential: The high engagement and low deterioration rates suggest that such tools could extend the reach of mental health services, though the authors note that exclusion of high-risk populations (acute suicidality, psychosis) limits generalizability to those groups.

The paper concludes that these findings support further investigation into mental health GAI in real-world settings, positioning the technology as a viable component for expanding access to care while maintaining safety standards.