💻 computer science

Hidden Meanings in Plain Sight: RebusBench for Evaluating Cognitive Visual Reasoning

The paper introduces RebusBench, a benchmark of 1,164 rebus puzzles designed to evaluate the cognitive visual reasoning of Large Vision-Language Models, revealing that despite their strong perceptual capabilities, current state-of-the-art models severely struggle with the multi-step neurosymbolic reasoning required to solve these puzzles, achieving less than 10% exact match accuracy regardless of model scale or in-context learning.

Original authors: Seyed Amir Kasaei, Arash Marioriyad, Mahbod Khaleti, MohammadAmin Fazli, Mahdieh Soleymani Baghshah, Mohammad Hossein Rohban

Published 2026-04-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Seyed Amir Kasaei, Arash Marioriyad, Mahbod Khaleti, MohammadAmin Fazli, Mahdieh Soleymani Baghshah, Mohammad Hossein Rohban

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot that can look at a picture and describe exactly what it sees. If you show it a photo of a cat sitting on a mat, it will happily tell you, "There is a cat on a mat." This robot is great at seeing.

But what happens if you show the robot a picture that isn't just a photo, but a visual riddle?

The Problem: The Robot Can See, But Can't "Get" the Joke

The paper introduces a new test called RebusBench. A "rebus" is a puzzle where pictures and letters are mixed together to hide a common phrase.

Think of it like this:

The Picture: You see a red letter "E" and two words saying "GO."
The Robot's View: "I see a red 'E' and two 'GO's." (This is what the robot actually says).
The Human View: "Wait, 'Red E' sounds like 'Ready', and 'Two Gos' sounds like 'To Go'. The answer is 'Ready to go'!"

To solve this, you can't just look at the pixels. You have to:

Notice the color and the repetition.
Remember that "Red E" sounds like "Ready."
Connect those sounds to a common idiom.

This is called System 2 thinking—slow, deliberate, creative reasoning. It's the difference between recognizing a face (System 1) and solving a mystery (System 2).

The Experiment: The "Giant Brain" Fails the Riddle Test

The researchers took the smartest, biggest AI models available (like Qwen, InternVL, and LLaVA) and gave them 1,164 of these riddles. They tried everything to help the models:

Making the models bigger: They used models with billions of parameters (like upgrading from a bicycle to a rocket ship).
Giving them examples: They showed the models a few solved riddles first (like giving a student a practice test).

The Result? The models failed miserably.

Even the biggest models only got about 5% to 8% of the answers right.
Making the models bigger didn't help much.
Giving them examples didn't help much either.

The Analogy: The Library vs. The Librarian

Imagine these AI models are like massive libraries containing every book ever written. They have all the words, all the pictures, and all the facts.

What they are good at: If you ask, "What is in this picture?" they can pull out the book that says "Cat" and show you the picture of a cat.
What they are bad at: If you ask a riddle, they get stuck. They have all the ingredients (the word "Red," the word "E," the word "Go"), but they lack the chef who knows how to mix them together to make a new dish.

They have the ingredients (vision and language), but they are missing the cooking recipe (the cognitive "glue" that connects the dots).

Why Does This Matter?

The paper argues that we are hitting a wall. We keep making AI bigger and giving it more data, but it still can't "think" in the way humans do when solving puzzles.

Current AI: "I see a red E and two Gs." (Literal)
Human Intelligence: "Red E + Two Gs = Ready to Go!" (Abstract)

The researchers call this a "cognitive gap." It suggests that simply adding more computing power isn't enough to teach AI how to be creative or solve riddles. We need a new kind of "glue" to help these models connect what they see with what they know in a clever way.

The Bottom Line

The paper is a wake-up call. It says: "Our AI is amazing at describing the world, but it's terrible at understanding the hidden jokes and riddles of the world."

Until we figure out how to give AI that "aha!" moment, it will remain a very smart parrot that can repeat facts but can't quite solve the puzzle.

1. Problem Statement

While Large Vision-Language Models (LVLMs) have achieved high proficiency in explicit visual recognition (describing what is directly visible), they exhibit a critical cognitive gap when visual inputs serve as clues rather than direct answers.

The Gap: Current models struggle with problems requiring neurosymbolic reasoning, where visual elements must be combined with linguistic prior knowledge (idioms, phonology, abstract concepts) to synthesize a meaning that does not exist in the pixel space.
Limitation of Existing Benchmarks: Standard datasets (e.g., VQA v2, GQA, CLEVR) primarily test "System 1" rapid perceptual processing (referential grounding). They fail to evaluate "System 2" deep, deliberate reasoning, modal entanglement, and the ability to perform multi-step symbolic manipulation.
The Specific Challenge: Solving a Rebus puzzle requires a complex cognitive workflow: extracting visual attributes, retrieving external linguistic knowledge, and performing abstract mapping to decode hidden idioms, all while suppressing the literal interpretation of the image.

2. Methodology: RebusBench

The authors introduce RebusBench, a novel benchmark designed to rigorously test this specific integration of perception and knowledge.

Dataset Composition:
- Size: 1,164 human-authored rebus puzzles.
- Sources: Aggregated from diverse sources specializing in lateral thinking (e.g., ESL Vault, Just Family Fun).
- Curation: Strict manual verification was applied to ensure unique, clear solutions free from cultural ambiguity or subjective interpretation.
Task Definition:
- Input: An image ( $I$ ) containing a visual riddle.
- Output: A target idiom or phrase ( $T$ ).
- Mechanism: The task requires the model to approximate a function $f(I) \to T$ that entangles visual attributes ( $V$ : position, color, size) with linguistic priors ( $L$ : phonology, idioms).
- Cognitive Spectrum: The dataset covers a continuous spectrum from visually dominant puzzles (e.g., geometric deformation of text, like a "slice" cut out of the word "CAKE") to symbolically dominant puzzles (e.g., the absence of a number in a sequence implying "No one").
Evaluation Metrics:
1. Exact Match (EM): A strict metric where predictions are normalized (lowercased, stripped of whitespace) and compared against the ground truth.
2. Semantic Score (GPT-4o Judge): A lenient metric using an LLM judge to rate semantic similarity on a scale of 0.0 to 1.0, accounting for synonyms and slight phrasing variations.

3. Experimental Setup

Models Evaluated: A representative suite of state-of-the-art open-weight LVLMs, including:
- LLaVA-1.5 (7B)
- InternVL 3.5 (4B, 8B, 30B)
- Qwen 2.5 (3B, 7B, 32B)
- Qwen 3 (4B, 8B)
Prompting Strategies:
- One-Shot: Providing a single solved example.
- Three-Shot: Providing three solved examples (In-Context Learning).
Goal: To determine if scaling model parameters or increasing context (few-shot prompting) improves performance on abstract reasoning tasks.

4. Key Results

The evaluation reveals a severe deficiency in current LVLMs regarding abstract visual reasoning.

Performance Ceiling:
- Exact Match (EM): All models, regardless of size, failed to surpass 10% accuracy.
- Semantic Accuracy: Even with a lenient semantic judge, performance plateaued below 20%.
Ineffectiveness of Scaling:
- Increasing model parameters yielded negligible returns. For example, scaling InternVL 3.5 from 4B to 30B only improved 1-shot EM from 4.73% to 5.15%.
- The massive Qwen 2.5 32B model achieved only 7.39% (1-shot), barely outperforming its smaller counterparts.
Ineffectiveness of In-Context Learning (ICL):
- Providing more examples (3-shot) did not significantly help and, in some cases (e.g., InternVL 3.5 4B), performance actually degraded (dropping from 4.73% to 4.04%).
- This suggests models cannot induce the necessary "System 2" logic from demonstrations alone.

5. Key Contributions

RebusBench Dataset: A high-quality, human-verified benchmark of 1,164 puzzles specifically designed to test the "cognitive glue" between vision and language, moving beyond simple pattern matching.
Empirical Evidence of a Cognitive Gap: The study provides concrete data showing that current scaling laws (more parameters, more data, more context) are insufficient to bridge the gap between pixel-level processing and abstract symbolic reasoning.
Identification of the "Suppression" Failure: The authors highlight that the core failure is not just a lack of knowledge, but an inability to suppress the high-probability literal caption (e.g., "A red letter E") in favor of the abstract idiomatic meaning (e.g., "Ready to go").

6. Significance and Future Work

Significance: The paper challenges the prevailing assumption that simply scaling up LVLMs will lead to human-level general intelligence. It demonstrates that current architectures lack the robust vision-text entanglement required for lateral thinking and open-ended symbolic manipulation.
Future Directions:
- Expanding the dataset to include fine-grained metadata to classify puzzles along the visual-textual spectrum.
- Releasing a standardized evaluation pipeline.
- Extending benchmarks to include proprietary closed-source models (e.g., GPT-5, Gemini) to establish a comprehensive baseline.

Conclusion: RebusBench serves as a rigorous stress test, revealing that while modern LVLMs possess the necessary visual and linguistic components, they currently lack the cognitive reasoning mechanisms to synthesize them into abstract, non-literal meanings.