💻 computer science

MemeBench: What LVLMs Miss When Interpreting Culture-Dependent Memes

This paper introduces MemeBench, a diagnostic benchmark utilizing the VIKR schema to evaluate and reveal the specific cultural knowledge gaps in Large Vision-Language Models' interpretation of memes, demonstrating that while entity-guided retrieval improves knowledge integration, it currently trades off visual coverage.

Original authors: Weihang Wang, Kainan Tu, Jielei Zhang, Run Yang, Boheng Sheng, Yuchen He, Yu Xie, Pengyu Chen, Peiyi Li, Huyang Sun, Longwen Gao, Zhouhui Lian

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Weihang Wang, Kainan Tu, Jielei Zhang, Run Yang, Boheng Sheng, Yuchen He, Yu Xie, Pengyu Chen, Peiyi Li, Huyang Sun, Longwen Gao, Zhouhui Lian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a room full of people speaking a language you don't know, but you can see their faces, their clothes, and the objects they are holding. You are a super-smart robot designed to describe exactly what you see. You can say, "That person is wearing a red hat," or "They are holding a golden cup." You are perfect at describing the pixels. But then, someone points to a picture of a cat wearing a tiny tuxedo and says, "This is a meme about how cats think they own the internet." If you don't know the internet, if you don't know the culture of cat owners, or if you don't understand the joke, you might just say, "It is a cat in a suit." You described the picture perfectly, but you completely missed the point.

This is the gap that researchers are trying to bridge with Large Vision-Language Models (LVLMs). These are AI systems that can both "see" images and "speak" about them. For a long time, we thought that if an AI could describe an image accurately, it understood it. But this paper argues that description is not the same as interpretation. It's the difference between reading the words of a joke and actually getting why it's funny. To understand a meme, you need more than just your eyes; you need a library of cultural knowledge, inside jokes, and community history that isn't written on the image itself. If an AI can't access that hidden library, it's like a tourist who can read a map but doesn't know the local slang or the history of the landmarks.

Enter MemeBench, a new diagnostic tool created by researchers from Bilibili, Fudan University, and Peking University to test exactly where these AI models get stuck. Think of MemeBench as a "cultural pop quiz" for AI, specifically focused on the wild, fast-moving world of anime, comics, games, and internet subcultures (often called ACG). The researchers collected 1,253 memes in both Chinese and English. But instead of just asking the AI, "Is this funny?" or "What is the answer?", they asked the AI to write a full explanation. Then, they broke those explanations down into four specific ingredients using a system they call VIKR:

Visual: Did you see what's actually in the picture?
Identity: Did you know who or what the characters are? (e.g., Is that a generic ninja, or is it specifically Naruto?)
Knowledge: Did you know the backstory or the cultural rule that makes this funny?
Reasoning: Did you connect the dots to explain the joke?

The researchers tested 26 different AI models, from the biggest commercial ones to open-source experiments. Here is the big surprise they found: Every single model was better at describing the picture than at understanding the joke. Even the smartest AI in the test could describe the visual details with high accuracy, but when it came to the cultural knowledge needed to interpret the meme, it stumbled. The best model still had a "gap" of 22.6% between how well it saw the image and how well it knew the culture. It's like having a pair of eyes that work perfectly, but a brain that's missing the instruction manual for the world.

The paper also argues against the idea that simply giving the AI more "reasoning steps" (telling it to "think step-by-step") fixes this problem. It doesn't. The models still missed the cultural context. However, the researchers did find a way to help. They introduced a method called KAR (Knowledge-Aware Retrieval). Imagine if, before the AI tried to tell the joke, it was allowed to quickly look up the specific character names and history in a specialized encyclopedia (which they built called CultureBase) before searching the general web. This didn't just make the AI smarter; it made it more precise. It helped the AI find the missing cultural pieces (Identity and Knowledge) without messing up the visual description.

In short, this paper suggests that while AI is getting great at being a camera, it is still struggling to be a cultural critic. The models aren't failing because they can't see; they are failing because they don't know the story behind the image. By using MemeBench, researchers can now pinpoint exactly which part of the explanation is missing—whether it's the name of the character, the history of the event, or the logic of the joke—rather than just giving a single score. This helps developers build AI that doesn't just describe the world, but actually understands the culture within it.

Technical Summary: MemeBench

Problem Statement

Large Vision-Language Models (LVLMs) have demonstrated significant progress in describing visual content; however, accurate description does not guarantee correct interpretation when meaning relies on knowledge external to the pixels. Memes serve as a critical stress test for this limitation because their humor and meaning depend heavily on cultural entities, background knowledge, and community-specific conventions. Existing benchmarks often reduce meme interpretation to classification labels, multiple-choice answers, or holistic correctness scores. These approaches obscure where an explanation fails—whether it misdescribes the image, misses a referenced entity, lacks necessary background knowledge, or fails to connect these elements into the intended meaning. Consequently, models with similar aggregate scores may omit fundamentally different information, and standard evaluations cannot determine if targeted evidence can repair specific gaps.

Methodology

MemeBench Dataset

The authors introduce MemeBench, a bilingual diagnostic benchmark comprising 1,253 Chinese and English memes. The dataset is centered on ACG (Anime, Comics, Games) and adjacent online subcultures, sourced from Bilibili, Reddit, and ImgFlip.

Composition: The set includes 768 Chinese and 485 English memes, covering domains such as ACG (50.1%), cross-domain references, daily life, movies/TV, and public life.
Knowledge Density: The benchmark references 2,072 distinct entities, 86.9% of which appear in only one meme, ensuring a long-tail distribution that prevents models from relying on frequent entity memorization.
Annotation: Each meme includes human-written reference explanations covering visual content, entity identity, cultural background, and humor mechanisms.

The VIKR Schema

At the core of MemeBench is VIKR, a schema that decomposes open-ended explanations into four observable requirements:

Visual (V): Entity-agnostic description of visual elements and OCR text.
Identity (I): Exact naming of entities and their source works, linked to their visual appearance.
Knowledge (K): Background facts and cultural codes required to understand the meme.
Reasoning (R): The mechanism connecting visual elements and cultural context to the intended humor or message.

Evaluation utilizes binary checklists for each dimension. A response achieves Success only if it satisfies all items across all four dimensions ( $Success = V \land I \land K \land R$ ). This allows for granular diagnosis of missing components rather than a single holistic score.

Entity-Guided Retrieval (KAR)

To test whether diagnosed gaps can be filled, the authors propose KAR (Knowledge-Aware Retrieval), an entity-guided retrieval baseline.

Pipeline: KAR extracts visual cues and entity hypotheses from the LVLM, then grounds these entities using CultureBase (a curated multilingual inventory of ~5,000 cultural entities from Moegirl and KnowYourMeme).
Mechanism: Unlike generic web search, KAR uses the grounded entity identity to construct specific search queries before retrieving web evidence. This ensures the retrieved context is anchored to the correct cultural entities.
Comparison: The authors compare KAR against a "Vanilla" setting, a "CoT" (Chain-of-Thought) setting, and a "Search-CoT" setting (generic web retrieval without entity grounding).

Key Contributions

MemeBench Benchmark: A bilingual dataset of 1,253 memes with VIKR annotations and dimension-specific checklists designed to diagnose open-ended interpretations rather than simply score them.
Diagnostic Evaluation of 26 LVLMs: A comprehensive evaluation revealing a consistent "Visual–Knowledge gap." Every model covers visible content more reliably than the background knowledge required for interpretation. Even the strongest model (Gemini-3.1-Pro) retains a 22.6% gap between Visual and Knowledge coverage.
KAR Baseline and Analysis: The introduction of KAR as an entity-guided retrieval method. The study demonstrates that while retrieval improves Identity and Knowledge coverage, it can inadvertently reduce Visual coverage. KAR outperforms generic retrieval by achieving a better "repair–damage" balance, fixing more errors without breaking correct visual descriptions.

Results

Baseline Performance

Across 26 commercial and open-source models:

Performance Gap: The gap between Visual coverage and Knowledge coverage ranges from 14.6% to 29.0% across models. The top-performing model, Gemini-3.1-Pro, achieves 60.3% Success, with Visual coverage at 92.3% but Knowledge coverage at 69.7%.
Granular Differentiation: Models with similar aggregate Success scores often fail in different dimensions. For example, GPT-5.6-Luna and Claude-Sonnet-5 differ by only 0.2% in Success, yet their failure modes are distinct: Sonnet covers 7.1% more visual content but grounds 1.6% fewer entities.
Language Disparity: Performance drops significantly in Chinese compared to English for top models (e.g., Gemini-3.1-Pro: 76.7% English vs. 49.9% Chinese Success), with the largest gaps appearing in Identity and Knowledge dimensions.

Retrieval Augmentation

Selective Improvement: Retrieval conditions (Search-CoT and KAR) consistently improve Identity and Knowledge scores but cause a decline in Visual coverage across all tested models.
KAR Superiority: KAR outperforms generic Search-CoT. For Gemini-3.1-Pro, KAR repairs 26.5% of failures while breaking only 11.5% of passes (ratio 2.31), whereas Search-CoT repairs 21.7% and breaks 13.8% (ratio 1.57).
Entity Grounding Value: The comparison between Search-CoT and KAR isolates the value of entity grounding. KAR's ability to anchor queries to specific cultural entities before searching leads to more effective retrieval that fills knowledge gaps without displacing visual accuracy as severely as generic search.

Significance and Claims

The paper claims that MemeBench shifts the evaluation of open-world meme interpretation from a single aggregate score to an actionable, component-level diagnosis. By revealing what an explanation gets right, what it misses, and how retrieval changes these components, the benchmark provides a framework for targeted model improvement.

The authors assert that the separation between visual description and culturally grounded interpretation is a fundamental bottleneck for current LVLMs. They demonstrate that while retrieval can bridge the knowledge gap, it introduces a trade-off where visual coverage may degrade. MemeBench and the VIKR schema allow researchers to measure this trade-off precisely, ensuring that improvements in cultural knowledge do not come at the cost of basic visual fidelity. The work highlights that progress in description has outpaced progress in culturally grounded interpretation, particularly in long-tail ACG and subculture domains.