Multilingual KokoroChat: A Multi-LLM Ensemble Translation Method for Creating a Multilingual Counseling Dialogue Dataset

1. Problem Statement

The paper addresses two critical challenges in the field of AI-driven mental health support:

• Data Scarcity: There is a severe lack of high-quality, publicly available, human-authored counseling dialogue datasets in languages other than Japanese. Existing datasets are often synthetic (LLM-generated), lacking authenticity, or limited in size.
• Translation Instability: While Large Language Models (LLMs) are powerful translation tools, no single model consistently produces the highest quality translation for every input. In sensitive domains like counseling, where fidelity, tone, and context are paramount, relying on a single LLM introduces significant risk. Different models exhibit varying strengths and weaknesses depending on the specific context, leading to inconsistent output quality that is unacceptable for professional counseling applications.

2. Methodology: Multi-LLM Ensemble Translation

The authors propose a novel two-stage Multi-LLM Ensemble method to translate the existing Japanese corpus, KokoroChat, into English and Chinese. Unlike traditional selection-based ensembles (which pick the best output from a list), this method uses a fusion-based approach to synthesize a superior output.

Stage 1: Diverse Hypothesis Generation

• Input: The entire dialogue session is processed as a single unit (rather than sentence-by-sentence) to preserve conversational flow, speaker roles, and terminology consistency.
• Process: Three distinct, state-of-the-art LLMs are prompted to generate independent translation hypotheses for the same dialogue.
  • English Models: GPT-5, Gemini 2.5 Pro, Grok-4.
  • Chinese Models: GPT-5, Gemini 2.5 Pro, Qwen-Plus.
• Goal: To create a diverse set of high-quality candidates, ensuring that the strengths of different models are captured.

Stage 2: Integration and Refinement

• Refiner Model: A specialized "refiner" LLM (Gemini 2.5 Pro, chosen for its superior text comparison and synthesis capabilities) acts as the integrator.
• Process:
  1. The refiner receives the original Japanese text and the three translation hypotheses.
  2. It performs a diagnostic analysis for each utterance, explicitly identifying the strengths and weaknesses of each candidate (e.g., empathy level, semantic accuracy, naturalness).
  3. Based on this analysis, the refiner synthesizes a new translation that combines the best elements of the hypotheses while correcting identified flaws.
• Key Innovation: The method does not merely select the best existing output; it generates a new output by leveraging the reasoning capabilities of the refiner to resolve inconsistencies and optimize for the specific nuances of counseling (e.g., tone, politeness, and emotional validation).

3. Key Contributions

• Multilingual KokoroChat Dataset: The creation of the first large-scale, multilingual (Japanese, English, Chinese) psychological counseling dialogue dataset. It is based on KokoroChat, the world's largest manually curated Japanese counseling corpus (6,589 sessions, ~600k utterances), ensuring high authenticity and professional quality.
• Novel Ensemble Framework: A new translation methodology that moves beyond simple selection or voting. It introduces an analysis-driven refinement process where a refiner LLM explicitly critiques and synthesizes multiple hypotheses to produce a result superior to any single source.
• Comprehensive Evaluation: The paper provides a rigorous validation of the method using both automated metrics and human preference studies, highlighting the limitations of current reference-free automated metrics in evaluating dialogue-level coherence.

4. Experimental Results

The authors evaluated the method against single-LLM baselines using 200 sampled dialogues.

Automated Evaluation

• Metrics: Used reference-free metrics XCOMET (higher is better) and MetricX (lower is better).
• Findings:
  • The proposed method achieved the best MetricX scores for both English and Chinese.
  • In XCOMET, the method tied for first place in Chinese and showed statistically significant improvements over GPT-5 in English.
  • Statistical significance was confirmed using the Wilcoxon Signed-Rank Test with Bonferroni correction.

Human Evaluation

• Method: Pairwise comparison where native speakers (English and Chinese) judged translations based on naturalness and contextual appropriateness within a dialogue history.
• Findings:
  • Overwhelming Preference: The ensemble method was preferred over every single-LLM baseline in both languages.
  • Win Rates: In English, the proposed method won 46–54% of comparisons against individual models. In Chinese, the win rate was even higher (72–84%).
  • Qualitative Success: Human evaluators noted the ensemble outputs were significantly more natural and contextually coherent.
  • Case Study Analysis:
    • Success: The refiner successfully identified and adopted the most empathetic phrasing from a candidate (e.g., using modal particles for a gentle tone).
    • Failure Mode: In some cases, the refiner prioritized strict semantic fidelity over conversational naturalness, leading to translations that were technically accurate but felt unnatural to human readers. This highlights a trade-off between fidelity and fluency.

5. Significance and Discussion

• Advancement in Mental Health AI: By providing a high-quality, multilingual dataset, this work enables the development of culturally and linguistically diverse AI counseling systems, addressing the shortage of human counselors globally.
• Insight on Evaluation: The study reveals a discrepancy between automated and human evaluation. Reference-free metrics often penalize contextually appropriate but semantically sparse utterances (e.g., "It is" as a response to "That must be difficult"). This suggests that current automated metrics are ill-equipped to evaluate dialogue-level coherence and role-appropriateness.
• Future Directions: The authors suggest that the Multi-LLM Ensemble approach is particularly effective for language pairs where single-model performance is less mature (e.g., Japanese-Chinese) and could be even more impactful for low-resource languages. Future work aims to better balance semantic fidelity with conversational naturalness in the refinement process.

Conclusion

The paper successfully demonstrates that a fusion-based multi-LLM ensemble method, which analyzes and synthesizes multiple translation hypotheses, produces significantly higher quality counseling translations than any single state-of-the-art model. The resulting Multilingual KokoroChat dataset serves as a foundational resource for advancing cross-lingual research in AI-driven mental health support.