Riddle Quest : The Enigma of Words
This paper introduces a pipeline for generating analogy-based riddles and uses it to demonstrate that while large language models can often identify the primary answer, they frequently fail to recover the full set of valid interpretations, highlighting riddles as a valuable tool for evaluating reasoning coverage and ambiguity handling in AI.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are at a party where everyone is playing a game of "Guess Who?" but instead of asking, "Does your person have a nose?", they have to describe the person using a riddle like, "I have a face but no eyes, hands but no fingers. What am I?"
This paper is about building a robot chef that can create these riddles for us, and then testing if other robot brains (AI models) are smart enough to solve them all.
Here is the story of how they built the chef, how it cooks, and what they learned about the robots' brains.
1. The Problem: Robots Need to Learn to "Think" in Riddles
Riddles are tricky. They aren't just facts; they are puzzles that use metaphors (saying one thing is like another) and wordplay. Humans are great at this, but computers usually just memorize facts. The authors wanted to see if they could teach a computer to write riddles that require creative thinking and analogy (seeing connections between different things), not just rote memory.
2. The Solution: The "Riddle Factory" Pipeline
The authors built a four-step assembly line (a pipeline) to make riddles. Think of it like a factory that turns raw ingredients into a fancy dessert:
- Step 1: The Fact Collector (Triples Creator)
Imagine a librarian who grabs a book about a specific object (like a "Clock"). Instead of just reading the whole book, the librarian pulls out specific facts and writes them on index cards: Clock → has → hands, Clock → tells → time, Clock → is → round. - Step 2: The Translator (Semantic Mapper)
This step takes those dry facts and groups them into "flavors." It sorts the facts into categories like "how it looks," "what it does," or "how it behaves." It turns the raw data into a "flavor profile" for the riddle. - Step 3: The Poet (Stylized Generator)
This is the creative artist. It takes the "flavor profile" and writes the riddle. It can change the style on command!- Want a funny riddle? It adds jokes.
- Want a poetic riddle? It uses flowery language.
- Want a metaphorical riddle? It says, "I am a river that flows backward," to describe a clock.
- Step 4: The Safety Inspector (Validator)
Before the riddle is released, this inspector checks: "If I read this riddle, what are all the possible answers?" It doesn't just guess one answer; it lists every single thing that could fit the clues. This creates a "master answer key."
3. The Experiment: Can Robots Solve Their Own Riddles?
The authors made 120 riddles about everything from simple spoons to abstract ideas like "curiosity" or "gravity." They tested these riddles on a Large Language Model (a very smart AI chatbot).
The Big Surprise:
The AI chatbot was good at guessing the main answer. If the riddle was about a clock, the AI usually said "Clock."
However, the AI was terrible at finding all the answers.
- The Human/Validator approach: "This riddle could be a clock, a watch, a timer, or a sundial."
- The AI approach: "This is a clock." (It stops there).
The AI tends to get "overconfident" and lock onto one answer, missing the other valid possibilities. It struggles to see the full picture of ambiguity.
4. What This Means (The Takeaway)
The authors conclude that riddles are a perfect test drive for Artificial General Intelligence (AGI)—the idea of a robot that thinks like a human.
- Why? Because making and solving riddles requires analogy (connecting two different things), abstraction (thinking about ideas, not just objects), and creativity.
- The Result: Current AI is getting better at writing the riddles, but it still struggles to understand the full depth of the puzzle. It often misses the "hidden" answers that a human might spot.
Summary in a Nutshell
The paper built a machine that writes riddles in many different styles (funny, poetic, serious). They used this machine to test if smart AI chatbots can think deeply enough to find every possible answer to a riddle. They found that while the AI is clever, it often gets stuck on just one answer and misses the others. This shows that riddles are a great, simple tool to measure how close AI is to truly "thinking" like a human.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.