Technical Summary: MyMentorLLM

Problem Statement

The training of psychotherapists faces a critical scalability bottleneck. While effective psychotherapy requires repeated practice and expert supervision to develop competencies like agenda setting, emotional attunement, and case formulation, access to these resources is limited. Real patients cannot bear the risks of early clinical errors, standardized actors cannot replicate the full variability of psychopathology at scale, and expert supervisors are a scarce resource. Existing solutions, such as rule-based conversational agents (e.g., ELIZA, PARRY), lack the open-ended dialogue and psychological coherence necessary for realistic training. Furthermore, text-only simulations fail to capture the paralinguistic cues (prosody, pacing, hesitation) essential to psychotherapy, and current Large Language Model (LLM) applications often prioritize linguistic fluency over psychological fidelity, risking the generation of convincing but clinically implausible interactions.

Methodology

The authors introduce MyMentorLLM, a multimodal simulation environment designed to reproduce the complete Cognitive Behavioural Therapy (CBT) mentorship cycle. The study generated 2,100 complete training sessions involving three interacting LLM-instantiated roles:

1. Patient: Grounded in specific DSM-5-TR clinical cases for Major Depressive Disorder (MDD), Generalized Anxiety Disorder (GAD), and Borderline Personality Disorder (BPD). Personas were constructed from six-part clinical profiles (demographics, complaint, history, symptoms, communication style, cognitive-emotional patterns) and instructed to disclose information gradually and react to therapist attunement.
2. Trainee Therapist: Calibrated to the median competence profile of developing clinicians using data from a large-scale dataset of 1,264 real CBT sessions (Goldberg et al.). The persona was constrained to exhibit realistic limitations rather than expert-level performance.
3. Expert Mentor: A senior CBT supervisor that reviews the session transcript, scores the trainee using the 11-item Cognitive Therapy Rating Scale (CTRS), provides qualitative feedback, and guides a diagnostic reassessment.

Experimental Design:

• Models: Six LLMs from three families were tested: Gemini 3.1 Flash Live (native speech-to-speech), Gemini 3.5 Flash, Gemma 4 (12B and E2B), and Qwen 3.5/3.6.
• Modalities: The study compared native speech-to-speech (end-to-end audio processing), speech-mediated (audio input/text output synthesized to speech), and text-only interactions.
• Procedure: Each session consisted of a 31-turn dialogue followed by a supervision phase where the trainee provided an initial diagnosis, received feedback, and then revised the diagnosis and identified five key DSM-5-TR symptoms.

Analysis:

• Emotional Profiles: Analyzed using EmoAtlas and Plutchik's model to compare simulated emotional signatures against a human reference corpus (HOPE dataset).
• Competence: Evaluated via CTRS scores assigned by the mentor, compared against human reference distributions.
• Diagnostic Accuracy: Measured before and after mentor feedback to assess the utility of supervision and the presence of "harmful feedback" (where feedback causes a correct diagnosis to become incorrect).
• Symptom Identification: Assessed the ability to correctly identify disorder-specific symptoms from a fixed list.

Key Results

1. Disorder-Congruent Emotional Dynamics
Simulated patients successfully reproduced disorder-specific emotional signatures: MDD was sadness-led with blunted positive affect; GAD was fear- and anticipation-led; and BPD exhibited broad negative affect (fear, sadness, anger). Crucially, the simulated therapists mirrored these affective profiles in an attenuated form, replicating the "affective resonance" observed in human counselling.

2. Native Speech-to-Speech Replicates Human Competence Levels
A significant divergence was found between modalities. The native speech-to-speech condition (Gemini-3.1LA) produced trainee scores (Mean CTRS = 29.97) that closely matched the human reference sample (Mean = 31.04) and remained below the competence threshold (CTRS ≥ 40). In contrast, text-based and re-synthesised audio conditions yielded scores significantly higher than human baselines (ranging from 41.15 to 55.99), often saturating near the maximum scale. This suggests that text fluency creates an illusion of competence, while native audio preserves the hesitation and pacing that reveal the limitations expected of a trainee.

3. Supervision is Beneficial Only for Capable Models
Mentor feedback improved diagnostic accuracy in five of the seven model conditions, particularly helping trainees recover missed BPD diagnoses. However, in the two smallest-model conditions (Gemma-4-E2B), feedback was harmful, causing a drop in accuracy. These smaller models exhibited "uncritical deference," abandoning correct initial diagnoses to follow the mentor's feedback, often shifting from MDD to GAD. This indicates that supervision can amplify errors in models lacking sufficient reasoning capacity.

4. Model Scale Correlates with Clinical Grounding
Symptom identification accuracy was strongly correlated with model size. The largest model (Qwen3.6-35B) achieved ~95.5% accuracy in identifying correct symptoms, while the smallest models (Gemma-4-E2B) fell to near-chance levels (19–26%). Furthermore, symptom identification was consistent with the final diagnosis: correct diagnoses were accompanied by accurate symptom selection, whereas incorrect diagnoses were not.

Significance and Claims

The paper claims that MyMentorLLM demonstrates the feasibility of simulating a complete CBT mentorship cycle for deliberate practice, but with critical caveats regarding fidelity and safety.

• Shift in Focus: The work shifts the focus of AI in mental health from replacing therapists to improving how therapists are trained.
• Modality Matters: The study argues that native multimodal interaction is essential for realistic simulation. Text-based simulations may overestimate therapeutic competence by smoothing over the paralinguistic cues that characterize real clinical encounters.
• Calibration is Critical: The authors emphasize that simulation is not a monolithic solution. The effectiveness of AI mentors depends on the "calibration" of the system: smaller models may require different supervision strategies to avoid harmful deference, and patient fidelity must be evaluated alongside supervisor reliability.
• Safety and Interpretability: The paper concludes that while synthetic patients and mentors can support scalable education, human supervision and stronger safeguards remain indispensable. The system serves as an interpretable testbed to identify when synthetic training is sufficiently faithful and safe, rather than asserting that it is universally ready for deployment.

The authors explicitly state that the study does not prove LLMs can replace human participants, noting that synthetic populations can misrepresent the groups they emulate and that convincing behavior does not equate to clinical validity. The primary contribution is a framework for evaluating the psychological fidelity and educational safety of AI-driven psychotherapy training.