TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed for mental health support, yet current evaluation methods are insufficient for clinical applications.

• Limitations of Existing Metrics: Standard metrics (e.g., BLEU, ROUGE, generic dialogue benchmarks like MT-Bench) measure fluency and coherence but fail to capture clinical fidelity (adherence to evidence-based therapeutic models) and safety (risk of harm, crisis mismanagement).
• Lack of Process-Oriented Evaluation: Therapy is inherently interactive and processual. Existing benchmarks often treat therapy as a knowledge QA task or use pairwise preference tests that lack interpretability and do not ground evaluations in validated clinical constructs (e.g., the Cognitive Therapy Rating Scale).
• The Gap: There is no scalable framework that simultaneously evaluates therapy chatbots on clinical skill (fidelity) and safety while providing a mechanism to improve them via reinforcement learning.

2. Methodology: The TherapyGym Framework

The authors introduce TherapyGym, an end-to-end framework for evaluating and aligning therapy chatbots based on two clinical pillars: Fidelity and Safety. The system operates through three core components:

A. The Evaluator Suite (THERAPYJUDGEBENCH & THERAPYJUDGE)

• THERAPYJUDGEBENCH: A validation dataset consisting of 116 simulated 10-turn dialogues between LLM therapists and patient simulators.
  • Patient Simulation: Uses the Patient-ψ simulator (initialized with CBT cognitive models) to generate diverse symptom profiles and realistic interaction patterns.
  • Expert Annotation: Dialogues are annotated by two licensed CBT practitioners using the Cognitive Therapy Rating Scale (CTRS).
    • Fidelity: 11 CBT skill dimensions (e.g., Agenda, Feedback, Guided Discovery) rated on a 0–6 scale.
    • Safety: 4 binary flags for critical risks (e.g., providing medical advice, failing to address crisis/abuse).
  • Reliability: Human inter-rater reliability (Krippendorff's $\alpha$) averages 0.52, consistent with human-human therapy studies, establishing a credible ground truth.
• THERAPYJUDGE: An LLM-based judge trained to approximate expert ratings.
  • It takes a full dialogue and a structured rubric as input.
  • It outputs 11 CTRS scores and 4 safety flags.
  • Performance: Achieves a Spearman correlation of 0.56 with human experts on CTRS skills and 99% accuracy on safety flags, validating its use as a reward signal.

B. The Alignment Module (RL Fine-Tuning)

• Training Environment: The framework utilizes 13,093 distinct patient profiles (augmented via TreeSynth) to generate diverse training scenarios.
• Reward Model: The frozen THERAPYJUDGE acts as a reward model. The composite reward $R(d)$ is calculated as:
  $$R(d) = \sum_{i \in S} w_i \cdot \widehat{CTRS}_i(d) - \sum_{j=1}^{4} \lambda_j \cdot \mathbb{1}_{Safety_j}(d)$$
  Where $S$ is the set of 9 reliable CTRS dimensions, and the second term penalizes safety violations.
• Optimization: The system uses Group Relative Preference Optimization (GRPO), an extension of PPO, to fine-tune the therapist LLM. GRPO samples multiple dialogues per patient profile, normalizes rewards within the group, and updates the policy to maximize the composite reward.

3. Key Contributions

1. First Clinical Fidelity & Safety Framework: Introduces a framework that operationalizes clinical research standards (CTRS) for LLM evaluation, moving beyond generic conversational metrics.
2. THERAPYJUDGEBENCH: A high-quality, expert-annotated benchmark for auditing LLM judges in multi-turn therapy settings, addressing the "black box" nature of LLM evaluation.
3. Scalable RL Pipeline: Demonstrates a controllable training loop where clinical fidelity and safety directly drive model optimization via GRPO, without requiring real-time human intervention during training.
4. Safety-Aware Alignment: Explicitly models safety as a penalty term, preventing the model from "gaming" skill rewards at the expense of patient safety.

4. Key Results

The authors evaluated a Qwen3-4B model trained within TherapyGym against a baseline (untrained) and compared it to frontier models.

• Improvement in Clinical Fidelity (CTRS):
  • Human Ratings: Average CTRS score increased from 0.10 (baseline) to 0.60 after GRPO fine-tuning.
  • LLM Judge Ratings: Score increased from 0.16 to 0.59.
  • Specific Gains: Significant improvements were observed in Feedback (0.04 $\to$ 0.65) and Focus (0.04 $\to$ 0.67).
• Safety Improvements:
  • Safety violation rates (human-judged) decreased from 0.38 to 0.20.
  • Ablation Study: Removing the safety penalty during training resulted in higher safety violations (0.43), confirming the necessity of the safety term in the reward function.
• Qualitative Shifts:
  • Trained models shifted from generic emotional validation to structured CBT behaviors (e.g., setting agendas, identifying automatic thoughts, assigning concrete homework).
  • The trained model performed comparably to much larger frontier models (e.g., GPT-o3-mini, Llama 4 Scout) despite being smaller.
• Judge Reliability: The LLM judge successfully recovered a substantial fraction of the human signal, validating the feasibility of automated evaluation in high-stakes domains.

5. Significance and Impact

• Bridging the Gap: TherapyGym provides a scalable solution to the "evaluation bottleneck" in mental health AI, enabling the development of chatbots that are not just engaging but clinically meaningful.
• Safety First: By integrating safety constraints directly into the RL reward function, the framework ensures that models do not optimize for helpfulness at the cost of patient safety.
• Future Directions: While currently focused on CBT, the framework establishes a blueprint for expanding to other modalities (e.g., DBT, ACT) and incorporating longitudinal real-world outcomes.
• Research Artifact: The authors emphasize that this is a research tool for characterizing model behavior, not a clinical tool for deployment, highlighting the need for rigorous validation before real-world use.

In summary, TherapyGym represents a paradigm shift in mental health AI, moving from generic conversational benchmarks to clinically grounded, safety-aligned, and process-oriented evaluation and training pipelines.