CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks

1. Problem Statement

While Large Language Models (LLMs) are deployed globally, existing benchmarks for multilingual and multicultural competence suffer from significant limitations:

• Superficiality: Current benchmarks (e.g., BLEND, INCLUDE, Global MMLU) primarily test encyclopedic trivia or generic language understanding rather than real-world application.
• Lack of Grounding: They fail to evaluate "grounded tasks"—scenarios where models must reason within specific, context-rich, real-world situations (e.g., planning a trip, making a purchase, navigating local regulations).
• Narrow Scope: Recent attempts at grounded evaluation (e.g., CultureBank, NORMAD) are often English-centric or limited to narrow domains like social interactions.

Core Question: How effectively can LLMs tackle diverse, grounded tasks that require the integration of multilingual comprehension, long-tail cultural knowledge, and multi-step contextual reasoning across different regions?

2. Methodology: The CulturALL Benchmark

The authors introduce CulturALL, a comprehensive benchmark designed to fill the gap in evaluating grounded multilingual and multicultural tasks.

A. Data Construction Framework

CulturALL is built using a novel Human-AI Collaborative Framework consisting of four stages:

1. Cultural Topic Sourcing: A list of 16 diverse topics (e.g., Education, Finance, Heritage, Travel) was generated via human-LLM collaboration to cover facets of daily life.
2. Sample Creation: Annotators created seed instances based on personal experience, cross-lingual inspiration, existing datasets, and online resources (e.g., Xiaohongshu).
3. Sample Enrichment (Difficulty Elevation): To ensure high difficulty, "easy" samples underwent an up-leveling process using three strategies:
   • Long-Tail Swap: Replacing common entities with rare ones (e.g., "Hong Kong" $\to$ "MacLehose Trail").
   • More/Less Context: Adding conditional details or removing hints to force multi-step reasoning.
   • Compositional Examples: Merging independent knowledge points into a single query.
4. Release-Ready: Automated metadata completion (region/topic labeling) and translation, followed by rigorous human quality control (peer review).

B. Dataset Statistics

• Scale: 2,610 samples.
• Coverage: 14 languages, 51 regions (countries/territories), and 16 topics.
• Difficulty Distribution:
  • Hard: 55.17% (solved by $\le$ 4 model settings).
  • Medium: 26.82% (solved by 5–9 settings).
  • Easy: 18.01% (solved by 10–14 settings).
• Format: Each item includes a scenario, a question, an objective answer (or multiple-choice), and an explanation.

C. Evaluation Setup

• Models: 15 configurations of 8 state-of-the-art LLMs (including Gemini, GPT-5, Claude, and Qwen series), varying by reasoning capabilities (e.g., "auto," "high," "low") and web search integration.
• Metric: Accuracy measured via an automated judge (gpt-4o) comparing model predictions against gold answers.
• Prompting: Zero-shot prompting with a requirement for concise answers.

3. Key Contributions

1. Unified Framework: A scalable human-LLM collaborative pipeline for creating high-coverage, high-difficulty benchmarks.
2. CulturALL Benchmark: The first benchmark explicitly designed to assess LLMs on grounded multilingual and multicultural tasks, moving beyond trivia to real-world reasoning.
3. Comprehensive Analysis: An in-depth evaluation of SOTA models, revealing specific failure modes and the critical role of reasoning and retrieval.

4. Key Results

Experiments on CulturALL reveal that even the best models struggle significantly with grounded cultural tasks:

• Overall Performance: The best-performing model (gemini-2.5-pro with auto-reasoning and web search) achieved only 44.48% accuracy.
• Proprietary vs. Open-Source: Proprietary models (Gemini, GPT-5, Claude) significantly outperformed open-source models. The best open-source model (Qwen3-235b) achieved only 23.68%, highlighting a massive gap in handling complex cultural reasoning.
• Impact of Reasoning & Search:
  • Reasoning: Enhanced reasoning capabilities ("auto" vs. "low") improved performance for some models (e.g., Gemini) but had diminishing returns for others (e.g., GPT-5, Claude), suggesting a ceiling in their multi-step reasoning abilities for cultural contexts.
  • Web Search: Web search integration provided a substantial boost (+6.59%) for Gemini, but offered little benefit to Qwen models, indicating a lack of training in effectively leveraging retrieved information for grounded tasks.
• Difficulty Levels: Performance drops precipitously as difficulty increases. On "Hard" tasks, the best model scored only 18.47%.
• Language Variance: Performance varies significantly by language. For instance, models performed better on Serbian but struggled with Japanese and Arabic.
• Native vs. English Prompts: Using native language prompts yielded 8.08% higher accuracy than English translations, suggesting that translation dilutes crucial cultural nuances and context.

5. Significance and Future Directions

• Benchmarking Standard: CulturALL sets a new standard for evaluating LLMs, shifting focus from "what they know" (trivia) to "how they apply knowledge" (grounded reasoning).
• Identified Gaps: The results underscore that current LLMs lack the necessary integration of information retrieval, multi-step reasoning, and deep cultural context to be truly reliable in global, real-world applications.
• Limitations & Future Work: The authors acknowledge limitations including coverage bias in certain regions, a focus on geographic rather than socio-demographic cultural groupings, reliance on objective answers (excluding free-text generation), and the exclusion of multimodal inputs. Future work should address these to create a more holistic evaluation of cultural competence.

In conclusion, CulturALL demonstrates that while LLMs are advancing, there is substantial room for improvement in their ability to navigate the complexities of real-world, culturally grounded scenarios across diverse languages.