Cross-Lingual Response Consistency in Large Language Models: An ILR-Informed Evaluation of Claude Across Six Languages

1. Problem Statement

The paper addresses a critical gap in Large Language Model (LLM) evaluation: while existing benchmarks (e.g., MEGA, XGLUE) effectively measure factual accuracy and reasoning across languages, they fail to capture pragmatic, stylistic, and culturally embedded dimensions of language use.

• The Core Issue: LLMs trained on multilingual corpora often produce outputs that vary in quality, length, and cultural framing even when the semantic input is identical.
• The Risk: Users in non-English languages may receive responses that are factually correct but pragmatically inadequate, culturally misaligned, or calibrated to English discourse norms rather than their own linguistic community's conventions.
• The Gap: Current automated metrics cannot distinguish between a "model failure" and a "culturally appropriate adaptation," nor can they measure the functional adequacy of a response against the complexity of the prompt.

2. Methodology

The study employs a novel two-layer evaluation framework grounded in the Interagency Language Roundtable (ILR) Skill Level Descriptions, a standard framework for human language proficiency assessment.

A. Experimental Design

• Model: Claude Sonnet 4.6.
• Languages: English, French, Romanian, Spanish, Italian, and German (all high-resource European languages).
• Data Collection: 216 responses generated from 12 semantically equivalent prompt clusters (3 runs per prompt × 6 languages).
• Prompt Battery: 12 clusters spanning ILR complexity levels 1 through 3+, covering diverse domains (creative, technical, emotional, ambiguous).

B. The ILR-Informed Framework

The author introduces two key constructs adapted from human assessment:

1. Prompt Complexity Level (PCL): The ILR level a human would need to comprehend and respond appropriately to a specific prompt.
2. Response Adequacy Level (RAL): The ILR level a human would need to produce the observed response, measuring the model's linguistic and pragmatic sophistication relative to the prompt's demands.

C. Two-Layer Analysis

1. Layer 1: Quantitative Consistency (Automated)
   • Metrics: Response length (word count), structural formatting (list vs. prose), hedging marker density, and surface similarity to an English baseline.
   • Tool: TF-IDF character n-gram cosine similarity (used as a surface-level proxy for lexical divergence).
2. Layer 2: Qualitative Pragmatic Assessment (Expert Judgment)
   • Performer: A six-language professional with 12 years of ILR/OPI assessment experience.
   • Focus: Register appropriateness, pragmatic completeness, cultural calibration, and identification of "adequacy gaps."
   • Goal: To interpret why variation exists (e.g., is it a cultural adaptation or a model deficiency?).

3. Key Contributions

1. Novel Evaluation Framework: Operationalizes the ILR framework for LLM output analysis via PCL and RAL constructs, moving beyond accuracy to measure pragmatic and functional adequacy.
2. Public Dataset: A citable prompt battery of 12 clusters across six languages, annotated with ILR levels and researcher hypotheses, made publicly available.
3. Methodological Argument: Demonstrates that expert human judgment is essential to interpret cross-lingual variation. Automated metrics can detect divergence but cannot distinguish between "cultural appropriateness" and "model error."
4. Empirical Findings: Identifies five systematic, interpretable patterns of cross-lingual variation in Claude.

4. Key Results

Quantitative Findings

• Length Disparity: French responses were approximately 30% longer than German responses on identical prompts, suggesting a systemic variation in elaboration depth.
• Surface Divergence: Creative and affective clusters (e.g., Creative Narrative, Emotional Support) showed the highest divergence from the English baseline, while metalinguistic clusters showed the highest similarity.
• Structural Differences: German responses favored prose elaboration over lists, whereas French responses utilized bullet points more frequently.

Qualitative Findings (Five Variation Patterns)

1. Pragmatic Disambiguation Strategies:
   • German: Refused to resolve ambiguity, explicitly requesting clarification (prioritizing precision).
   • French: Embraced ambiguity, offering reflective guidance without demanding immediate resolution.
   • Spanish/Italian/Romanian: Silently resolved ambiguity and proceeded to advice.
2. Aesthetic and Literary Divergence:
   • Romanian creative responses utilized distinct folk similes (e.g., umbrellas blooming like mushrooms) and organic personification absent in English outputs, reflecting genuine literary tradition rather than translation drift.
3. Technical Terminology & Analogies:
   • German and French used native compound terms for ML concepts.
   • Cultural Analogies: The model selected culturally specific pedagogical analogies for each language (e.g., Dart throwing for German, Wine tasting for Romanian, Apartment pricing for French).
4. Cultural Calibration Gaps (The "Neutralization" Effect):
   • In Implicit Cultural Knowledge (e.g., honoring the deceased), the model defaulted to culturally neutralized templates.
   • Example: Romanian responses omitted specific Orthodox rituals (parastas, coliva), reading as generic translations of English templates rather than natively grounded advice.
5. Institutional Anchoring:
   • In Emotional Support, only the German response embedded a specific, callable national crisis resource (Telefonseelsorge), while others offered generic advice. This indicates domain-dependent access to institutional knowledge.

5. Significance and Implications

For AI Deployment

• Equity Concerns: Cross-lingual variation is not random noise; it is systematic. Users of different languages receive substantively different amounts of information (e.g., French users get 30% more elaboration) and different types of cultural support.
• Cultural Competence is Domain-Dependent: The study reveals a paradox: the model can generate culturally specific folk similes (creative domain) and specific crisis referrals (procedural domain) but fails to invoke deep cultural schemas in ritual/ceremonial domains. This suggests cultural knowledge is activated unevenly based on prompt type.

For Evaluation Methodology

• Beyond Automated Metrics: The paper argues that purely computational benchmarks are insufficient for open-ended conversational AI. The "interpretive gap"—understanding what a variation means—requires human expertise.
• ILR as a Standard: The ILR framework provides a robust vocabulary for describing register, pragmatics, and cultural calibration, offering a complementary approach to existing rubric-based evaluations.

Limitations

• Scope: Limited to six high-resource European languages; results may not generalize to low-resource or non-Indo-European languages.
• Single Model: Findings are specific to Claude Sonnet 4.6.
• Single Rater: Qualitative analysis relied on one expert, lacking inter-rater reliability metrics (though the author's credentials are extensive).

Conclusion

The paper concludes that cross-lingual consistency in LLMs is interpretable, domain-dependent, and consequential. While Claude demonstrates impressive cultural calibration in specific domains (creative writing, crisis referral), it defaults to culturally neutralized English templates in others (rituals). The study advocates for a hybrid evaluation approach that combines automated metrics with expert ILR-informed judgment to ensure equitable and culturally competent multilingual AI deployment.