From Native Memes to Global Moderation: Cross-Cultural Evaluation of Vision-Language Models for Hateful Meme Detection

1. Problem Statement

The detection of hateful content in internet memes is a critical challenge for global content moderation. However, current Vision-Language Models (VLMs) are predominantly trained on Western, English-centric data, leading to significant cross-cultural bias.

• Cultural Nuance: Memes rely heavily on cultural context, humor, sarcasm, and local idioms. A model trained on English norms often misinterprets content from non-English cultures (e.g., misidentifying political satire as hate speech or missing genuine hate due to cultural blindness).
• The "Translate-Then-Detect" Flaw: A common industry practice involves translating non-English memes into English before detection. The authors argue this approach strips away essential cultural context and semantic nuance, leading to systematic performance degradation.
• Gap in Research: Existing benchmarks often use translated datasets rather than native, organically sourced memes, failing to capture the true complexity of non-Western digital ecosystems.

2. Methodology

The authors propose a systematic, multi-dimensional evaluation framework to diagnose and quantify the cross-cultural robustness of state-of-the-art VLMs.

Datasets and Scope

The study utilizes six distinct datasets representing diverse linguistic and cultural ecosystems:

• Arabic: Prop2Hate (Political/Religious propaganda)
• Bengali: BHM (South Asian code-mixing, local politics)
• English: HateMeme (Baseline)
• German: GerMemeHate
• Italian: DANKMEMES
• Spanish: DIMEMEX (Mexican Spanish, machismo/classism)

Experimental Design

The evaluation varies four key dimensions:

1. Model Type: 7 General-Purpose VLMs (e.g., Gemini-2.5-Flash, GPT-4o-Mini, LLaMA-4, Qwen-2.5-VL, CogVLM2) and 2 Task-Specific models (Pro-Cap, PromptHate).
2. Learning Strategy: Zero-shot (task definition only) vs. One-shot (one culturally representative example provided).
3. Prompt Language: English prompts vs. Native-language prompts.
4. Content Representation:
   • Native: Original image and caption.
   • Translated: Original image with captions machine-translated (via Google Translate) to other target languages. Crucially, image-embedded text remains untranslated to isolate the effect of caption translation.

Metrics

Performance is measured using Accuracy and Macro-F1 scores, with a specific focus on the trade-off between False Positives (over-censorship) and False Negatives (missed detection).

3. Key Contributions

1. Multi-Dimensional Framework: A novel evaluation protocol that decouples learning strategies, prompting languages, and content representations to isolate sources of cultural bias.
2. Empirical Evidence on Translation: Demonstrates that the "translate-then-detect" pipeline is systematically detrimental, causing a universal drop in performance across all models.
3. Intervention Strategies: Identifies that native-language prompting and one-shot learning are highly effective interventions that significantly mitigate cultural bias and improve detection accuracy.
4. Bias Diagnosis: Provides evidence of inherent Western-centric bias in VLMs, showing how safety guardrails trained on English norms fail when applied to non-English cultural contexts.

4. Key Results & Findings

A. Model Scale and Performance Tiers

• General vs. Specialized: General-purpose VLMs (e.g., Gemini-2.5-Flash) establish a robust cross-lingual baseline. Specialized models fine-tuned on a single language perform well only on linguistically proximate tasks but collapse on distant transfers ("feast or famine" pattern).
• Stability: Even top-tier models show performance variance across cultures. Smaller models (e.g., InstructBLIP) exhibit "predictable inefficacy" rather than random variance.

B. Prompting Strategies (The Three Tiers)

The study categorizes models based on their sensitivity to prompt language:

• Tier 1 (Robust): Models like Gemini-2.5-Flash and Qwen-2.5-VL show minimal performance drop between English and native prompts. They possess generalized abstract understanding.
• Tier 2 (Moderately Sensitive): Models like GPT-4o-Mini show a counter-intuitive drop in performance when using native prompts in zero-shot mode. The authors attribute this to Safety-Prior Interference: native prompts trigger the model's English-centric safety guardrails incorrectly (e.g., flagging benign slang as hate speech), leading to over-censorship.
• Tier 3 (Brittle): Smaller models (e.g., InstructBLIP, CogVLM2-Chinese) are highly sensitive. For InstructBLIP, native prompts act as a "lifeline," drastically improving performance. For CogVLM2-Chinese, native prompts are detrimental outside its training domain.

C. The Power of One-Shot Learning

• Providing a single, culturally aligned example (one-shot) significantly boosts performance, particularly for smaller models.
• It compensates for knowledge gaps, though it cannot fully bridge the gap to the zero-shot baseline of larger models.
• It effectively mitigates the "Safety-Prior Interference" seen in Tier 2 models, reducing False Positive Rates (FPR).

D. The Translation Penalty

• Universal Degradation: Translating captions to English (or other pivot languages) before detection consistently lowers F1 scores across all models and languages.
• Mechanism: Translation strips away sarcasm, dialectal nuances, and cultural context. For example, an Arabic phrase meaning "I hate girls" in a relationship context (exaggerated frustration) is translated literally, triggering English safety filters that flag it as hate speech, resulting in a Semantic Flip (false positive).

5. Significance and Implications

• Rethinking Moderation Pipelines: The paper argues against the standard "translate-then-detect" approach for global moderation. It suggests that relying on translation introduces a bias that harms fairness and accuracy for non-Western users.
• Actionable Strategies: To build globally robust systems, platforms should:
  1. Prioritize native-language prompting over English translation.
  2. Utilize one-shot learning with culturally representative examples to align model behavior with local norms.
  3. Adopt a hybrid ensemble strategy: combining locally fine-tuned smaller models (for cultural nuance) with powerful general-purpose VLMs (for broad safety oversight).
• Ethical Impact: The findings highlight that current AI safety systems may inadvertently suppress benign cultural expression (over-moderation) or fail to detect genuine hate (under-moderation) in non-English contexts, necessitating a shift toward culturally aware AI development.

In conclusion, this work moves the field from merely diagnosing cultural bias to providing concrete, data-driven interventions (native prompting and one-shot learning) to achieve fairer and more effective global content moderation.