Technical Summary: SPLIT – Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responses

Problem Statement

The rapid deployment of Large Language Models (LLMs) in emotional support and crisis-related contexts has outpaced research into their cross-lingual capabilities, particularly for low-to-mid resource languages. While existing benchmarks emphasize multilingual performance, they frequently fail to examine crisis-related empathy and cultural grounding. A critical gap exists between linguistic fluency and the ability to provide culturally appropriate emotional support. The paper posits that producing text in a specific language (e.g., Ukrainian) does not equate to producing culturally grounded emotional support in that language. This study addresses the inconsistency in LLM responses when transitioning from high-resource languages (English) to low-to-mid resource languages (Ukrainian) in high-stakes, crisis-related scenarios.

Methodology

Dataset: The SPLIT Benchmark

The authors introduce SPLIT, a 500-prompt benchmark designed to evaluate LLM consistency in generating emotionally grounded responses. The dataset consists of 500 distinct emotional support queries distributed across five crisis-related categories:

1. Stress
2. Panic
3. Loneliness
4. Internal Displacement
5. Tension

Prompts were generated in both English and Ukrainian. While machine translation was the primary source for Ukrainian data, a native Ukrainian speaker with C2 English proficiency rigorously verified a randomized 15% sample (n=75) to ensure Natural Language Generation (NLG) accuracy.

Model Selection

Three technically diverse LLMs were evaluated as response generators:

1. DeepSeek-V3: A Mixture-of-Experts (MoE) architecture with an auxiliary-loss-free strategy and multi-token prediction training.
2. LLaMA-3.3-70B-Instruct: A standard dense transformer model.
3. Gemini-2.5-Flash: A closed-source, sparse MoE transformer model with hybrid reasoning capabilities.

Evaluation Framework

Responses were assessed across three dimensions using a 1–5 scale:

1. Empathetic Accuracy: Identification of emotional state and avoidance of clichés.
2. Linguistic Naturalness: Preservation of natural flow, idioms, and expressions specific to the crisis.
3. Contextual & Cultural Grounding: Adaptation to the user's language, cultural background, and specific crisis context.

Evaluation Paradigm

The study employed a dual-evaluation approach:

• LLM-as-a-Jury: Three high-reasoning models (GPT-4o, Mistral Large, and Claude 4.5 Sonnet) acted as an automated jury to score all 3,000 generated responses (3 models × 500 prompts × 2 languages).
• Human Validation: A single native Ukrainian speaker with C2 English proficiency manually evaluated 10% of the responses (n=300).
• Statistical Analysis: The study calculated Pearson correlation coefficients ($r$), Mean Absolute Error (MAE), and Mean Error (ME) to measure the alignment between human and automated evaluations.

Key Results

Performance Disparities (Human Evaluation)

A significant performance gap was observed when models transitioned from English to Ukrainian:

• Gemini-2.5-Flash and LLaMA-3.3-70B-Instruct: Both models exhibited severe degradation in Ukrainian. LLaMA's Linguistic Naturalness score dropped by 1.76 points (from 3.92 in English to 2.16 in Ukrainian). Gemini's Cultural Grounding score fell by 1.2 points.
• DeepSeek-V3: This model remained comparatively stable, with scores in Ukrainian either matching or slightly exceeding its English baseline. For instance, its Cultural Grounding score increased from 3.54 (EN) to 3.74 (UA).

Automated vs. Human Agreement

The correlation between the LLM-as-a-jury and human evaluators was weak and dimension-dependent:

• Empathetic Accuracy: Weak but significant positive correlation ($r = 0.198$). The AI jury showed a leniency bias (ME = +0.281), often over-scoring basic empathetic vocabulary.
• Linguistic Naturalness: Weak positive correlation ($r = 0.149$). The AI jury heavily favored grammatical correctness and structure over authentic idioms, resulting in a large systematic leniency bias (ME = +0.506).
• Contextual & Cultural Grounding: The correlation was negative and non-significant ($r = -0.095, p > 0.05$). The automated jury failed to recognize authentic cultural adaptation, sometimes penalizing responses that humans found culturally appropriate but structurally divergent from standard patterns.

Specific Observations

• LLaMA exhibited "language drifting" and reliance on clichés in Ukrainian, likely due to its dense transformer architecture and English-heavy training data.
• Gemini displayed an "over-alignment bias," producing overly defensive, safety-oriented scripts that lacked emotional depth in Ukrainian.
• DeepSeek's stability was attributed to its MoE routing strategy and multi-token prediction objective, which allowed for better phrase-level planning in morphologically rich languages.

Key Contributions

1. SPLIT Benchmark: The introduction of a specialized 500-prompt dataset focused on crisis-related communication in Ukrainian, filling a gap in low-resource language benchmarks for emotional support.
2. Empirical Evidence of Divergence: The study provides data-driven evidence that multilingual competence does not equate to multicultural competence. Models can generate fluent Ukrainian text while failing to provide culturally grounded emotional support.
3. Evaluation of LLM-as-a-Jury: The paper critically assesses the reliability of automated evaluation in emotional support contexts, finding that while AI can approximate human judgment on superficial metrics (grammar, basic empathy), it fails significantly in assessing cultural grounding.
4. Architectural Insights: The results suggest that MoE architectures with specialized routing (like DeepSeek-V3) may offer better cross-lingual stability in emotional contexts compared to dense transformers or models reliant on massive, averaged web-scraped data.

Significance and Claims

The paper concludes that "Producing Ukrainian text is not equivalent to producing Ukrainian emotional support."

The authors argue that current benchmarks and evaluation methods are insufficient for crisis-response applications because they prioritize linguistic fluency and general coherence over deep cultural immersion. The study claims that:

• LLMs deployed in crisis contexts require deeper cultural adaptation beyond simple language translation.
• Automated evaluation (LLM-as-a-jury) is unreliable for assessing cultural grounding, as it tends to reward structural correctness over authentic cultural nuance.
• Future development must shift from merely "multilingual" benchmarks to "multicultural" ones that account for the specific norms, values, and implicit references of mid-to-low resource languages.

The work serves as a cautionary note for developers integrating LLMs into crisis response systems, emphasizing that linguistic fluency alone is not a sufficient metric for safety and effectiveness in emotionally charged, cross-lingual scenarios.