Technical Summary: ID-MoCQA – Indonesian Multi-hop Question Answering for Complex Cultural Understanding

1. Problem Statement

Current cultural understanding benchmarks for Large Language Models (LLMs) predominantly rely on single-hop questions, where answers can be retrieved directly from isolated facts. While effective for measuring factual recall, these setups fail to probe whether models can perform genuine cultural reasoning across context, tradition, and implicit social knowledge. Models may exploit shallow cues rather than demonstrating the ability to connect interrelated cultural concepts. Furthermore, existing datasets often lack depth in low-resource languages and specific national contexts, particularly regarding the complex, unwritten social norms of Indonesia. There is a critical need for a benchmark that forces models to reason through multi-step cultural clues to arrive at an answer, simulating realistic cultural understanding.

2. Methodology: The ID-MoCQA Framework

The authors introduce ID-MoCQA, the first large-scale multi-hop QA dataset grounded in Indonesian traditions, available in both English and Indonesian. The dataset creation follows a rigorous, multi-stage pipeline designed to transform single-hop questions into complex reasoning chains.

2.1 Data Source and Expansion

• Base Data: The framework starts with 1,847 province-specific single-hop QA pairs from the IndoCulture dataset. These pairs are filtered to ensure the cultural element is unique to a specific province (True label).
• Transformation: The authors employ a framework to convert these single-hop instances into two-hop reasoning chains.
  • First Hop (Clue Identification): The model must identify a target province based on indirect cultural clues.
  • Second Hop (Cultural Application): Once the province is identified, the model answers the original cultural question specific to that region.
• Clue Types: Six distinct clue types are utilized to generate the first-hop link:
  1. Commonsense: Descriptive scenarios (e.g., "If A is female and inherits...").
  2. Comparison: Rankings or superlatives (e.g., "third highest number of UNESCO sites").
  3. Entity: Specific cultural artifacts or figures (e.g., "Tor-tor dance").
  4. Geographical: Location-specific features (e.g., "Derawan Islands").
  5. Intersection: Combining multiple attributes to narrow down to one province.
  6. Temporal: Historical events with specific dates.

2.2 Generation and Validation Pipeline

The process involves a combination of LLM generation and human/automated verification:

1. Automatic Expansion: Using Claude-3.7-Sonnet, the system generates bilingual (English/Indonesian) multi-hop questions. It converts context statements into questions, removes direct province mentions, and integrates the selected clue types.
2. Human Verification: A manual review of 3,000 randomly sampled instances classified questions into four categories: OK, Minor, Moderate, and Significant. This stage revealed that Intersection and Comparison types had the highest rates of significant errors (factual inaccuracies or ambiguity).
3. LLM-as-a-Judge: To scale validation across 22,164 generated instances, three frontier models (GPT-4o, Claude-3.7-Sonnet, DeepSeek-V3) were used to evaluate factual accuracy, structural coherence, and linguistic quality.
   • Agreement: Inter-annotator agreement between humans and LLMs showed moderate consistency (Cohen's $\kappa$ average 0.54).
   • Filtering: Instances marked "Significant" by any judge were rejected; those receiving majority "Acceptable" votes were retained.
4. Structure Verification: A final automated pass checks for direct copying of answer options and ensures no province names are explicitly mentioned in the clues.
5. Language Rebalancing: The dataset was balanced to ensure equal representation in English and Indonesian.

2.3 Final Dataset Statistics

• Total Instances: 15,590 questions (7,795 per language).
• Coverage: 11 Indonesian provinces across 6 islands, covering 12 cultural topics (e.g., Food, Wedding, Art, Religious Holidays).
• Difficulty: Human annotators rated ~45% of questions as "Hard," highlighting the dataset's complexity.

3. Experimental Setup and Results

The authors evaluated a diverse range of models, including frontier LLMs (GPT-5, DeepSeek-V3, Claude-3.7-Sonnet), multilingual open models (Llama 3.3, Qwen 2.5, Gemma 2), and region-specific models (Merak-7B, SeaLLM-7B).

3.1 Human Baseline

Three native Indonesian speakers achieved a 70.0% multi-hop accuracy.

• First-hop (Province ID): 95.1% accuracy.
• Second-hop (Final Answer): The drop to 70% indicates that identifying the location is significantly easier than applying cultural knowledge to that location.

3.2 Model Performance

• Frontier LLMs: Models like GPT-5 and Claude-3.7-Sonnet outperformed the human baseline, achieving ~81-82% accuracy in both languages.
  • Geographic Bias: Humans performed better on culturally prominent provinces (e.g., Bali, Java) but dropped to 65% on less central regions (e.g., Papua, Aceh). Frontier LLMs maintained high performance (~77%) across all provinces, suggesting their training data offers more balanced regional coverage than individual lived experience.
• Region-Specific Models: Surprisingly, models fine-tuned on Indonesian data (Merak-7B, SeaLLM-7B) underperformed compared to general frontier models. While they handled single-hop questions well (~53% on IndoCulture), their accuracy dropped to ~51% on ID-MoCQA's multi-hop tasks. This suggests that regional specialization does not automatically translate to improved multi-hop reasoning capabilities.
• Model Size: Larger models consistently outperformed smaller models, with the performance gap widening for complex reasoning tasks regardless of language specialization.

3.3 Reasoning Analysis

• Chain of Thought: The paper notes that adding "Let's think step by step" (Chain-of-Thought) was tested, though specific quantitative improvements are detailed in the full results section.
• Error Patterns: Frontier models achieved over 96% accuracy on the first hop (identifying the province) but saw a significant drop (18–23%) when considering the full multi-hop answer. This indicates that while models can retrieve geographical/cultural facts, they struggle with the nuanced inference required to select the correct cultural practice once the context is established.

4. Key Contributions

1. Framework: A new, comprehensive framework for systematically transforming single-hop cultural data into multi-hop reasoning chains using six distinct clue types.
2. Dataset (ID-MoCQA): The release of a human-verified, bilingual dataset of 15,590 multi-hop questions focused on Indonesian culture, addressing the lack of deep cultural reasoning benchmarks in low-resource contexts.
3. Evaluation: Extensive benchmarking of state-of-the-art models reveals that while frontier LLMs surpass humans in raw accuracy, significant gaps remain in nuanced cultural inference. The study highlights that regional specialization alone is insufficient for complex cultural reasoning.

5. Significance and Claims

The paper positions ID-MoCQA as a challenging and essential benchmark for advancing the cultural competency of LLMs.

• Beyond Facts: It demonstrates that true cultural understanding requires reasoning across context and implicit knowledge, not just fact retrieval.
• Limitations of Current Models: The results expose substantial gaps in cultural reasoning, particularly for models that rely on shallow cues or lack deep contextual application.
• Future Research: The dataset establishes a robust foundation for future research into improving LLMs' ability to handle unwritten social norms, diverse local traditions, and complex cultural inferences, which are critical for avoiding cultural insensitivities and stereotypes in AI systems.

The authors remain modest, acknowledging that while frontier models perform well, the persistent challenges in multi-hop reasoning indicate that current systems have not yet fully mastered the depth of cultural understanding required for real-world deployment in diverse cultural settings.