Technical Summary: Training Therapeutic Judges and Multi-Agent Systems for Human-Aligned Mental Health Support

Problem Statement

Large Language Models (LLMs) show promise for mental health support, yet their therapeutic quality often fails to improve unless evaluation functions as an actionable control signal rather than a passive metric. Current approaches face a critical "evaluation–action gap": scoring or filtering responses does not guarantee improvement unless evaluative signals are explicitly used to guide revision. Furthermore, existing evaluators often rely on proprietary, closed-source models that lack validation against therapeutic rubrics and safety-critical preferences, while open-source models typically underperform in these high-stakes domains. The core challenge is not merely generation capacity, but how evaluation is utilized to drive targeted, interpretable refinements in response to multi-dimensional, human-aligned therapeutic standards.

Methodology

The authors propose a two-stage framework that transforms therapeutic response generation into a decision-refinement problem driven by multi-dimensional human-aligned evaluation.

Stage I: TheraJudge (The Evaluator)

The first component is TheraJudge, an open-source therapeutic evaluator trained to produce reliable judgments across seven psychological dimensions: Guidance, Informativeness, Relevance, Safety, Empathy, Helpfulness, and Understanding.

• Training Data: The model is trained on 10,000 conversations from the Mental-Align-100K dataset, which includes human-annotated responses.
• Optimization: TheraJudge is trained using Group Relative Policy Optimization (GRPO). This preference-based optimization compares groups of responses (specifically groups of 16) to learn fine-grained distinctions in therapeutic quality.
• Reward Structure: The training employs a two-component reward system:
  1. Format Reward: Ensures the output is a valid JSON structure with all seven keys.
  2. Score-based Reward: A weighted sum of the seven dimensions, where clinically critical dimensions like Safety (weight 0.20) and Empathy (weight 0.25) are prioritized over others.
• Base Model: The evaluator is instantiated using Qwen2.5-7B, selected for its strong open-source performance and transparency.

Stage II: TheraAgent (The Refinement System)

The second component, TheraAgent, operationalizes TheraJudge's evaluations through a coordinated, multi-agent refinement process. It decouples evaluation from generation to prevent evaluator bias and uses a role-separated architecture:

1. Evaluator (TheraJudge): Assesses the initial response and provides structured multi-dimensional ratings.
2. Critic: Identifies specific issues in the response (e.g., lack of action, missing safety framing) based on the evaluator's low scores.
3. Coach: Provides concise, actionable suggestions for dimensions rated below 5, explicitly guiding which parts of the response need revision.
4. Therapist (Generator): Rewrites the response by integrating the Critic's identified issues and the Coach's guidance, maintaining clinical appropriateness.

• Process: This cycle iterates up to eight times. The process stops early if scores stabilize or exceed a threshold, and the highest-quality response from the iterations is selected. The generator used in this stage is LLaMA 3.1–8B, distinct from the evaluator to ensure robustness.

Key Contributions

1. Formulation of Decision-Refinement: The paper frames therapeutic response generation as a decision-refinement problem where multi-dimensional, human-aligned judgments serve as explicit control signals, directly addressing the evaluation–action gap.
2. TheraJudge: The introduction of an open-source, preference-based optimized evaluator that achieves high reliability (ICC > 0.87) across seven therapeutic dimensions, outperforming both supervised baselines and strong closed-source judges.
3. TheraAgent: A multi-agent framework that translates evaluative signals into targeted response revisions. Unlike monolithic regeneration, it selectively repairs low-quality responses while preserving high-quality ones.
4. Empirical Validation: The work provides extensive evaluation using blind human assessment by licensed clinicians, measuring not just mean score improvements but also effect sizes, recovery rates for low-quality responses, and inter-rater reliability.

Results

The framework was evaluated on 267 challenging mental health conversations, with responses rated by licensed clinicians across the seven dimensions.

• Evaluator Performance: TheraJudge achieved Intraclass Correlation Coefficients (ICC) between 0.87 and 0.95 when compared to clinician ratings, significantly outperforming the supervised baseline (Qwen-SFT) and closed-source models (e.g., GPT-4o, Claude-3.7, o4-mini). Notably, TheraJudge showed superior performance on critical dimensions like Safety and Empathy, where closed-source models exhibited instability.
• Therapeutic Quality Improvement: Acting on TheraJudge's evaluations, TheraAgent increased the overall mean human-rated therapeutic quality from 4.26 to 4.69 (+0.43 points) on a 5-point scale under blind evaluation.
• Targeted Recovery: The system demonstrated high efficacy in correcting unsafe or low-quality outputs. Low-quality responses (initial score ≤3) improved by an average of +2.45 points (a 110% gain).
  • Safety Recovery: Unsafe responses improved from 1.74 to 4.93, achieving a 100% recovery rate to acceptable safety levels.
  • Overall Recovery: 94% of initially low-quality ratings were successfully elevated above the acceptable threshold.
• Reliability: The improvements were validated by a second round of blind evaluation by independent clinicians, yielding 96% exact agreement and 100% majority agreement on quality thresholds.
• Ablation Studies: Removing the explicit evaluator reduced improvement by 0.21 points, confirming that the structured, multi-dimensional feedback is essential for effective refinement.

Significance and Claims

The paper claims that effective alignment of mental health LLMs stems from acting on human-aligned evaluation rather than relying solely on stronger generation capabilities. By closing the loop between evaluation and action, the proposed framework enables:

• Interpretable Improvement: The system identifies specific deficiencies (e.g., lack of safety framing) and corrects them without degrading other aspects of the response.
• Safety and Consistency: It ensures that unsafe outputs are consistently corrected while maintaining high performance on already acceptable responses.
• Open-Source Viability: It demonstrates that open-source models, when equipped with a specialized, human-aligned evaluator and a structured refinement pipeline, can achieve therapeutic quality that surpasses proprietary closed-source judges in critical safety and empathy dimensions.

The authors conclude that their results underscore the importance of moving beyond simple generation metrics to frameworks where evaluation serves as a dynamic, actionable control signal for iterative, targeted response refinement in high-stakes domains.