💬 NLP

CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledge

The paper introduces CresOWLve, a novel benchmark designed to evaluate large language models on creative problem-solving tasks grounded in real-world knowledge, revealing that while models excel at factual retrieval, they struggle significantly with the non-obvious integration of information required for creative solutions.

Original authors: Mete Ismayilzada, Renqing Cuomao, Daniil Yurshevich, Anna Sotnikova, Lonneke van der Plas, Antoine Bosselut

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mete Ismayilzada, Renqing Cuomao, Daniil Yurshevich, Anna Sotnikova, Lonneke van der Plas, Antoine Bosselut

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a game of "Connect the Dots," but instead of dots, you are connecting entire worlds of knowledge. That is essentially what the paper CRESOWLVE is about.

Here is the story of the paper, told simply:

1. The Problem: The "Robot Who Knows Everything But Can't Think"

For a long time, we've been testing AI (Large Language Models) like we test a student who has memorized every textbook in the library. We ask them: "Who was the first president?" or "What is 2+2?" The AI answers perfectly.

But real-life creativity isn't just about memorizing facts. It's about lateral thinking. It's like being in a room with a locked door, and instead of looking for a key, you realize the door is actually a painting, and you need to walk through the frame.

Most current AI tests are like riddles with obvious answers or fake puzzles that don't exist in the real world. They don't tell us if the AI can actually think creatively or just recall information.

2. The Solution: A New "Obstacle Course" Called CRESOWLVE

The researchers built a new test called CRESOWLVE. They didn't invent fake riddles; they went to a famous Russian TV game show called "What? Where? When?" (think of it as a high-stakes, intellectual version of Jeopardy! but with riddles that require deep cultural knowledge and "aha!" moments).

They took thousands of these real-world puzzles, cleaned them up, and translated them into English.

The Analogy:
Imagine the AI is a chef who has read every recipe book ever written.

Old Tests: Ask the chef, "How do you boil an egg?" The chef knows this perfectly.
CRESOWLVE: The chef is given a basket of random ingredients: a banana, a wrench, and a map of Italy. The challenge is to create a dish that represents "a journey." The chef has to realize that the banana is a boat, the wrench is a steering wheel, and the map is the ocean. That is creative problem-solving.

3. What They Found: The "Knowledge vs. Wisdom" Gap

The researchers tested the smartest AI models available today (like GPT-4, Gemini, and others) on this new course. Here is what happened:

The Fact-Checkers: When the questions were just about facts (e.g., "Who wrote this book?"), the AIs did okay. They could pull the answer from their memory.
The Creative Gap: When the questions required connecting two unrelated things (e.g., "How is a poet like a clock?"), the AIs stumbled badly. Their scores dropped by up to 17%.

The Metaphor:
The AI is like a person with a perfect library in their head but no map to navigate it. They have all the books (facts), but they don't know how to open a book on "History" and a book on "Music" and realize they are talking about the same thing. They get stuck trying to find a direct link, missing the creative "leap" that humans make naturally.

4. The "Thinking" Models: Slow Down to Speed Up

The paper also tested a new type of AI called "Thinking Models." These are models that are forced to pause and "think" (write out a chain of reasoning) before giving an answer.

The Result: These models did significantly better.
The Analogy: It's the difference between a student who blurts out the first answer they remember (Non-thinking) and a student who takes a deep breath, draws a diagram, and connects the dots before speaking (Thinking). The "Thinking" models were much better at solving the creative puzzles.

5. Why This Matters

The paper concludes that while AI is getting smarter at remembering facts, it is still struggling to be truly creative. It can retrieve the pieces of the puzzle, but it often fails to see the picture they form together.

The Takeaway:
We have built AI that is a walking encyclopedia, but we haven't quite built one that is a creative genius yet. CRESOWLVE is a new ruler to measure exactly how far we have to go to bridge that gap. It shows us that the next big leap in AI won't just be about knowing more facts; it will be about learning how to connect them in surprising, human-like ways.

1. Problem Statement

Current Large Language Model (LLM) benchmarks often fail to adequately evaluate creative problem-solving. Existing evaluations typically suffer from two main limitations:

Component Isolation: They test specific cognitive skills in isolation (e.g., pure factual retrieval, mathematical logic, or lateral thinking via contrived brainteasers) rather than the integrated combination of skills required for real-world creativity.
Artificiality: Many creativity benchmarks rely on "brainteasers" or situational puzzles that are artificially constructed and do not reflect how creative insights emerge in reality (i.e., by retrieving diverse real-world knowledge and forming non-obvious connections).

There is a lack of a benchmark that evaluates an LLM's ability to combine vertical thinking (logical progression), lateral thinking (restructuring problems), analogy-making, and commonsense reasoning over real-world knowledge to solve single-answer puzzles.

2. Methodology

2.1 Benchmark Construction (CRESOWLVE)

The authors constructed CRESOWLVE (Creative Reasoning Over Real-World Knowledge) using questions from the renowned Russian intellectual game "What? Where? When?" (a game where experts solve complex puzzles requiring broad knowledge and creative insight).

Data Collection: Started with 3,789 questions from the game's public database (spanning 50+ years).
Filtering Pipeline:
1. Physical Material Filter: Removed 295 questions requiring external physical materials (images, handouts) to be solvable.
2. Cultural/Linguistic Filter: Removed 799 questions heavily dependent on Russian-specific linguistic nuances, idioms, or culture that made them untranslatable or unsolvable in English.
3. Translation: Translated remaining questions into English using GPT-4o, preserving logical clues and ambiguity.
4. Human Validation: Three authors manually validated the dataset, removing 282 samples that were still Russian-specific, unanswerable, or required outdated information.
Final Dataset: 2,413 total samples, split into:
- Creative Subset: 2,061 questions requiring non-obvious creative leaps.
- Factual Subset: 352 questions solvable via direct retrieval.
Annotation: Each question was annotated with:
- Knowledge Domains: Consolidated into 34 coarse-grained categories (e.g., Literature, History, Film, Languages).
- Creative Strategies: Labeled with thinking types (e.g., lateral thinking, analogy, metaphor, abstraction).
- Cultural Context: Annotated with the cultures/demographics involved.

2.2 Experimental Setup

Models Evaluated: A diverse set of frontier LLMs, including:
- Non-thinking models: GPT-4.1, Llama-3.3-70B, Mistral-Large-3, Qwen3-235B, etc.
- Thinking models: GPT-5.4, Gemini-3.1-Pro, DeepSeek-V3.2, Qwen3.5, etc. (models capable of explicit reasoning chains).
Evaluation Metrics:
- Exact Match (EM): Normalized string matching.
- LLM-as-a-Judge: GPT-4o evaluated correctness based on reference answers and context, ignoring minor formatting/typos.
Prompting: Non-thinking models used Chain-of-Thought (CoT) prompting; thinking models used standard prompts with varying "thinking effort" levels (minimal, medium, high).

3. Key Contributions

CRESOWLVE Benchmark: The first bilingual (English/Russian) benchmark specifically designed to evaluate creative problem-solving grounded in real-world knowledge, distinguishing between factual retrieval and creative reasoning.
Comprehensive Evaluation: Extensive testing of over a dozen frontier models (both open-weight and proprietary) across different reasoning modes and languages.
Deep Analysis:
- Identification of a consistent performance gap between factual and creative reasoning.
- Error taxonomy revealing that failures are rarely due to missing knowledge but rather the inability to form creative connections between facts.
- Investigation into data contamination and cross-lingual performance anomalies.

4. Key Results

4.1 Overall Performance

High Difficulty: The benchmark remains highly challenging. Even the strongest models struggle, with top-performing "thinking" models (Gemini-3.1-Pro high effort) achieving ~76% accuracy on English and ~85% on Russian, while non-thinking models generally stay below 30%.
Thinking vs. Non-Thinking: "Thinking" models significantly outperform non-thinking counterparts. For example, GPT-5.4's performance nearly doubled when moving from "no thinking" to "medium effort" (9.4% $\to$ 17.8% EM on English).
Cross-Lingual Anomaly: While most models performed worse on the Russian version, closed-source thinking models (specifically Gemini-3.1-Pro) performed better on Russian than English.
- Investigation: A data contamination analysis suggested this advantage stems from the model having seen these specific questions during training (memorization), as perturbing the questions caused the performance drop to vanish.

4.2 Creative vs. Factual Reasoning

Performance Gap: There is a consistent and substantial drop in performance when moving from factual to creative questions.
- The drop ranges from -6% (Gemini-3.1-Pro) to -17% (Mistral-Large-3) on the English benchmark.
- This confirms that models can retrieve facts but struggle to integrate them creatively.

4.3 Error Analysis

Manual analysis of 150 error cases revealed seven primary error categories. The two most frequent were:

Missing Creative Connection (47.4%): The model retrieved relevant knowledge but failed to recognize the metaphorical or associative link required to solve the puzzle.
Clue Misinterpretation (27.5%): The model identified the general context but selected the wrong specific element.

Key Insight: Failures are not primarily due to hallucination or lack of knowledge, but an inability to perform the "creative leap" required to connect disparate facts.

4.4 Complexity vs. Difficulty

The authors investigated whether question difficulty correlated with the number of domains, semantic distance between domains, or the number of required facts.

Finding: No significant correlation was found. This suggests that difficulty in CRESOWLVE arises from the nature of the creative connection required, not the sheer volume or complexity of the information.

5. Significance

New Evaluation Paradigm: CRESOWLVE shifts the focus from "can the model know this fact?" to "can the model creatively synthesize facts?" This is crucial for advancing AI toward human-like reasoning.
Bottleneck Identification: The paper identifies that the current bottleneck for LLMs is not knowledge retrieval (which is often sufficient) but creative reasoning—the ability to form non-obvious associations.
Future Directions: The benchmark serves as a rigorous testbed to drive progress in models that can perform extended reasoning and lateral thinking, moving beyond simple pattern matching to genuine problem-solving.
Ethical Considerations: The authors highlight the risks of cultural bias in the source material and potential data contamination, urging caution in interpreting cross-lingual performance gains.

In summary, CRESOWLVE demonstrates that while LLMs are becoming proficient at factual recall and logical deduction, they still lack the robust creative reasoning capabilities necessary to solve real-world puzzles that require synthesizing diverse knowledge into novel insights.