MINERVA-Cultural: A Benchmark for Cultural and Multilingual Long Video Reasoning

1. Problem Statement

Current Video Large Language Models (Video-LLMs) have made significant strides in long-form video understanding. However, existing evaluation benchmarks suffer from two critical biases:

• Western-Centric Bias: Most datasets feature content, contexts, and cultural norms predominantly from Western societies.
• English-Dominance: Evaluations are primarily conducted in English, often relying on automatic translations of annotations rather than native-language curation.

These biases create a "cultural blind spot," preventing models from effectively reasoning about diverse global contexts, local festivals, cuisine, rituals, and regional sports. Furthermore, existing benchmarks often focus on final answers rather than the reasoning process, making it difficult to diagnose why models fail in multicultural settings.

2. Methodology: MINERVA-Cultural Benchmark

The authors introduce MINERVA-Cultural, a large-scale, open-ended Video Question Answering (VideoQA) benchmark designed to test multicultural and multilingual reasoning.

Dataset Construction

• Scale & Scope: The dataset comprises 2,400 questions derived from 540 videos across 18 global locales (e.g., Tamil in India, Spanish in Mexico, Arabic in Egypt, English in the UK).
• Content: Videos cover six core cultural domains: Sports, Cuisine, Festivals, Tourism, Rituals, and Education. Video lengths range from 1 minute to over 1 hour.
• Human-Centric Curation: Unlike prior work that translates English annotations, MINERVA-Cultural is entirely human-generated.
  • Curators: Native experts create complex, multi-step questions, answers, and reasoning traces in their native languages.
  • Auditors: Independent local experts verify objectivity, cultural authenticity, and difficulty.
  • Calibration: A rigorous process ensures questions cannot be solved by single frames, audio alone, or general knowledge; they require deep visual-cultural understanding and temporal reasoning.
• Reasoning Traces: Each question includes a detailed, human-authored reasoning trace (ground truth) that breaks down the solution into atomic steps, often spanning hundreds of words.

Diagnostic Framework: Evidence Graphs & Iterative Error Isolation

To move beyond simple accuracy metrics, the authors propose a novel diagnostic methodology:

1. Evidence Graphs: Human reasoning traces are converted into Directed Acyclic Graphs (DAGs).
   • Nodes: Represent "atomic evidences" (e.g., a specific visual observation, a fact, or a logical inference).
   • Edges: Represent prerequisite dependencies (e.g., Node B cannot be derived without Node A).
2. Iterative Error Isolation: A three-stage loop to analyze model failures:
   • Traversal: The model attempts to traverse the graph. If it fails to find an evidence node, the process halts.
   • Error Tagging: Failures are classified into a granular taxonomy (e.g., Temporal Localization, Spatial Grounding, Attribute Misidentification, Hallucination, Reasoning).
   • Counterfactual Hinting: The system provides a corrective hint for the specific error and re-evaluates the model. This loop repeats until the reasoning chain is complete, allowing researchers to isolate "root cause" errors versus cascading failures.

3. Key Contributions

1. The Benchmark: The first large-scale, multicultural, multilingual video reasoning benchmark with fully human-curated questions, answers, and reasoning traces in 18 native languages.
2. Diagnostic Framework: A novel Evidence Graph-based approach combined with Iterative Error Isolation to pinpoint exactly where and how models fail (perception vs. reasoning).
3. Comprehensive Evaluation: Extensive benchmarking of state-of-the-art (SoTA) Video-LLMs (including Qwen, Claude, GPT-5, and Gemini variants) against a human baseline.

4. Results & Analysis

The evaluation reveals a significant gap between current AI capabilities and human performance in multicultural contexts.

• Performance Gap:
  • Human Baseline: Achieved 95.22% accuracy (weighted average).
  • Top Model (Gemini-2.5-Pro): Achieved only 45.07% accuracy.
  • Cultural Disparity: Models perform significantly better in high-resource locales (e.g., Korean, British English) but struggle profoundly in low-resource or non-Western locales (e.g., Telugu: 28%, Tamil: 31.6%).
• Modality Importance:
  • Audio is Critical: Adding audio to video inputs improved performance by an average of 4.32%, with gains up to 8% in some locales, highlighting the necessity of native-language dialogue and cultural audio cues.
  • Temporal Reasoning: Accuracy increases monotonically with the number of input frames, confirming that static images are insufficient. However, performance plateaus even with high frame counts, indicating the bottleneck is reasoning, not visual sampling.
• Error Analysis (The "Cultural Visual Perception" Problem):
  • Using the Evidence Graph analysis, the authors found that 75% of all model failures stem from cultural visual perception errors.
  • These include:
    • Temporal Localization: Missing the correct time segment.
    • Spatial Grounding: Failing to identify culturally specific objects in the correct location.
    • Attribute Misidentification: Incorrectly identifying properties of cultural artifacts (e.g., clothing, rituals).
    • Spurious Objects: Hallucinating objects or events not present.
  • Pure reasoning errors (logical flaws after correct perception) were less frequent.
• Scaling Limits: Increasing the "thinking budget" (test-time compute) yields diminishing returns. While performance improves slightly, it remains far below human levels, suggesting current architectures lack the fundamental cultural grounding required.

5. Significance

• Equity in AI: MINERVA-Cultural addresses the "Western bias" in AI evaluation, providing a tool to measure and improve the global competence of multimodal systems.
• Interpretability: The shift from "black-box" accuracy to evidence-based error analysis allows developers to understand that the primary failure mode is not a lack of reasoning capability, but a failure to perceive and ground cultural elements visually.
• Future Directions: The benchmark serves as a blueprint for building fairer, more globally representative AI systems. It highlights that future progress requires not just larger models, but better training data that includes diverse cultural contexts and native-language reasoning traces.

In conclusion, MINERVA-Cultural demonstrates that while Video-LLMs are becoming powerful, they remain culturally myopic. The benchmark provides the necessary infrastructure to diagnose these gaps and guide the development of truly globally competent AI.