Technical Summary: Limits of Persona-Based Localization via Nationality and Language

Problem Statement

While Artificial Intelligence (AI) and Large Language Models (LLMs) offer potential solutions for global mental health challenges, particularly in regions with limited medical infrastructure, their deployment is hindered by a critical shortage of high-quality, multilingual datasets. Existing synthetic data generation methods, specifically persona-based approaches, rely heavily on English-centric contexts. A prevailing assumption in the field is that modifying demographic parameters (such as nationality and language) within validated English clinical personas can effectively generate equivalent multilingual datasets. However, it remains unclear whether such "parameter-based localization" preserves the clinical integrity of symptoms, particularly depression severity, when translated into non-English contexts like Mandarin, Bengali, and Hindi. This study investigates whether simple parameter substitution is sufficient to maintain clinical consistency or if it introduces systemic biases and inaccuracies in multilingual mental health assessments.

Methodology

The authors conducted a controlled experiment to evaluate the clinical consistency of synthetic personas adapted from a validated English baseline (Wang et al., 2025).

1. Persona Construction:

• Baseline: The study utilized 12 clinically validated English personas containing dimensions such as age, gender, BDI-II (Beck Depression Inventory II) scores, depression symptoms, communication style, life history, and social context.
• Localization Strategy: The researchers modified only two variables: Nationality and Language Use. They created parallel personas for four regions: United States (English), China (Mandarin), Bangladesh (Bengali), and India (Hindi).
• Output: This resulted in 48 total personas (12 per language), maintaining the original English prompt structure while altering the target demographic and output language.

2. Dialogue Generation:

• An independent LLM-based therapist agent (ChatGPT-4o-mini) conducted semi-structured clinical interviews with each persona.
• For each persona, five dialogue sessions were generated: four focusing on specific target symptoms and one mixed-symptom session.
• This produced a multilingual corpus of 60 dialogue sessions per language (240 total).

3. Evaluation Protocol:

• Blind Pairwise Judging: Independent LLM judges (including proprietary models like GPT-4o-mini and DeepSeek-v3.2, and open-weight models like LLaMA3-8B, Qwen3-8B, and DeepSeek-R1-8B) were presented with pairs of dialogues from personas with different ground-truth BDI-II severity levels.
• Task: The judges determined which patient exhibited higher depression severity (A > B, B > A, or Tie) without access to the underlying persona configurations.
• Metrics:
  • Overall Accuracy: Correct identification of the higher severity case (excluding ties).
  • Same-Level Error Rate: The percentage of incorrect predictions where the error occurred between personas of the same severity tier (indicating boundary confusion).
  • Tie Distance: The absolute difference in BDI-II scores when the model output a "Tie," measuring the model's uncertainty threshold.

Key Findings

The study revealed significant disparities in performance across languages and models, challenging the efficacy of minimal parameter-based localization.

1. Cross-Lingual Performance Disparity:

• English Advantage: All models performed best in English, which served as the upper bound for accuracy.
• Performance Drop: Non-English languages (Bengali, Hindi, Mandarin) showed consistent declines in accuracy. For instance, DeepSeek-R1-8B achieved 98.21% accuracy in English but dropped to 63.33% in Bengali and 67.24% in Hindi.
• Model Variance: Smaller open-weight models (8B parameters) exhibited higher volatility and larger performance gaps between English and non-English contexts compared to larger proprietary models.

2. Clinical Consistency and Error Types:

• Cross-Severity Errors: In English, models tended to make errors within the same severity tier (less clinically critical). In non-English languages, errors frequently crossed severity boundaries, indicating a loss of clinical signal clarity.
• Same-Level Error Rates: DeepSeek-R1-8B maintained a 100% same-level error rate in English (errors were only between identical tiers) but dropped to 22.73% in Bengali, meaning errors often misclassified severity levels entirely.
• Uncertainty and Ties: Non-English evaluations triggered significantly more "Tie" responses, often with large BDI-II score differences (high tie distance). This suggests models were unable to distinguish severity levels in non-English dialogues even when the ground-truth difference was substantial.

3. Root Causes of Failure:
The authors attribute these failures to the fact that nationality and language are not neutral parameters. Changing them alters the cultural context and expression patterns embedded in the LLM's generation. Consequently, the same BDI-II severity level manifests differently across cultures, and the shallow localization process fails to preserve the specific clinical cues required for accurate severity assessment.

Key Contributions

• Empirical Evidence of Inadequacy: The paper provides empirical data demonstrating that modifying nationality and language parameters in English-centric personas leads to disparities in depression level representation and weakens alignment with clinical severity levels in Mandarin, Bengali, and Hindi.
• Systemic Limitations Identified: The work highlights that applying synthetic personas to multilingual AI systems via simple parameter substitution is insufficient. It argues that such adapted personas should be treated as distinct clinical artifacts requiring rigorous output-level validation rather than direct extensions of English templates.
• Evaluation Framework: The study establishes a methodology for evaluating cross-lingual clinical consistency using pairwise severity comparison and tie-distance analysis, revealing that both generation and evaluation processes contribute to observed gaps.

Significance and Claims

The paper claims that the current practice of "shallow localization"—simply swapping nationality and language tags in English prompts—is inadequate for generating valid multilingual mental health datasets. The findings suggest that this approach introduces clinical ambiguity, causing generated dialogues to lose alignment with intended severity scales (BDI-II).

The authors emphasize that this is not merely a translation issue but a fundamental limitation in how LLMs handle cultural and linguistic nuances in clinical contexts. They argue for a shift toward culturally responsive data generation, where multilingual personas are designed and validated as separate entities rather than parameterized derivatives of English models. The work underscores the urgent need for language-specific evaluation and human-in-the-loop validation to ensure equitable mental health systems globally, cautioning against the deployment of unvalidated synthetic data in real-world applications.