💬 NLP

Augmenting Lateral Thinking in Language Models with Humor and Riddle Data for the BRAINTEASER Task

This paper presents a system that enhances DeBERTaV3's performance on the SemEval 2024 BRAINTEASER task by fine-tuning it with humor and riddle data to improve lateral thinking, demonstrating that such augmentation significantly boosts accuracy on sentence puzzles and that framing the task as multiple-choice yields substantial gains over sequence classification.

Original authors: Mina Ghashami, Soumya Smruti Mishra

Published 2026-02-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mina Ghashami, Soumya Smruti Mishra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart robot how to solve riddles. But not just any riddles—these are the tricky kind where the answer isn't what you expect. If I say, "A man shaves every day but still has a long beard," a normal robot might get confused. It thinks, "If he shaves, he shouldn't have a beard!" But the answer is actually, "He is a barber." He shaves other people's faces.

This is called Lateral Thinking. It's like looking at a problem from the side instead of straight on. Most robots (AI models) are great at "Vertical Thinking"—solving math problems or following strict logic step-by-step. But they struggle with lateral thinking.

This paper is about a team of researchers who built a robot that got really good at these "Brain Teaser" riddles for a big competition called SemEval 2024. Here is how they did it, explained simply:

1. The Two Types of Puzzles

The competition had two levels of difficulty, like a video game:

Sentence Puzzles: These are like short stories or jokes. You have to rethink the meaning of a whole sentence. (Example: "The man who lives in the house with the red door is actually the mailman.")
Word Puzzles: These are trickier. They play with the actual letters or sounds of words. (Example: "What part of London is in France?" The answer isn't a city; it's the letter N, because the word "L-o-n-d-o-N" is inside the word "F-r-a-n-c-e".)

2. The Secret Sauce: Teaching the Robot to "Get" Humor

The researchers realized that to teach a robot lateral thinking, they couldn't just give it more logic puzzles. They needed to feed it things that also require thinking sideways.

They used two special "training snacks":

Riddles: They used a dataset of classic riddles. Riddles are like mental gymnastics; they force you to stop thinking literally.
Jokes: They used a computer (GPT-4) to write thousands of joke-style questions. Jokes work because the punchline surprises you by changing the meaning of the setup.

The Analogy: Imagine you are training a dog to fetch a ball. If you only throw balls, it learns to fetch balls. But if you want it to learn how to play any game, you might throw a frisbee or a stick too. The dog learns the concept of "chasing fun objects" rather than just "chasing a specific ball." By feeding the robot riddles and jokes, the researchers taught it the concept of lateral thinking, not just the specific answers.

3. Changing the Game Rules (The Architecture)

The team tried two different ways to ask the robot the questions:

Method A (Sequence Classification): They asked the robot, "Is this sentence true or false?" or "Pick the right answer from a list."
Method B (Multiple Choice): They asked, "Here are four possible answers. Which one is the cleverest?"

The Result: Method B worked much better. It was like the difference between asking a student, "Write an essay on this topic" (hard to grade, easy to get wrong) versus "Here are four options, circle the right one." The "Multiple Choice" format helped the robot focus on comparing ideas, which is exactly what lateral thinking needs. This simple change boosted their score by 10 points!

4. The Results: A Tale of Two Puzzles

The robot became a champion, but with a twist:

Sentence Puzzles: The robot got 92.5% right. It was amazing! The jokes and riddles helped it understand how to twist the meaning of sentences.
Word Puzzles: The robot got 80.2% right. Still good, but not as perfect.

Why the difference?
The researchers realized that the "joke training" helped with sentences because jokes rely on meaning. But Word Puzzles rely on spelling and sounds (like the "N" in London). The robot was still a bit clumsy at looking at individual letters. It's like teaching someone to drive a car by giving them a motorcycle; they learn the rules of the road, but they still need to learn how to balance on two wheels.

5. The Big Takeaway

The main lesson from this paper is that context matters.

If you want an AI to be creative with language, feed it humor and riddles.
If you want it to solve word tricks, you might need a different kind of training.
And sometimes, the way you ask the question (Multiple Choice vs. Open Ended) is just as important as the AI itself.

The team finished 6th out of 31 teams for sentence puzzles and 10th out of 23 for word puzzles. They proved that if you teach a robot to laugh at a joke and solve a riddle, it gets much better at thinking outside the box.

1. Problem Definition

The paper addresses the SemEval 2024 BRAINTEASER task, which challenges Language Models (LMs) to perform lateral thinking—a form of creative, non-linear reasoning that deviates from standard logical (vertical) processes. The task is divided into two subtasks:

Sentence Puzzle: Requires reinterpreting sentence-level meaning to defy commonsense associations (e.g., "A man shaves everyday, yet keeps his beard long" $\rightarrow$ "He is a barber").
Word Puzzle: Requires reasoning about orthographic or phonetic properties of words (e.g., "What part of London is in France?" $\rightarrow$ "The letter N").

The core difficulty lies in the fact that existing NLP benchmarks primarily test vertical (step-by-step) reasoning. Lateral thinking requires models to abandon default interpretations and recognize non-obvious, often humorous or metaphorical, solutions. A significant challenge is ensuring models are testing reasoning capabilities rather than memorizing puzzle solutions, which the dataset addresses through semantic and context reconstruction (adversarial variants).

2. Methodology

The authors propose a system based on fine-tuning pre-trained transformer models with specific architectural choices and data augmentation strategies.

A. Model Architecture

The authors compared two primary formulations using HuggingFace's AutoModel classes:

Sequence Classification (AutoModelForSequenceClassification): Treating the task as a standard classification problem where the model selects the best answer from a set.
Multiple-Choice QA (AutoModelForMultipleChoice): A specialized architecture that processes the question and all candidate answers together to compute a probability distribution over the choices.

Key Finding: The Multiple-Choice formulation significantly outperformed Sequence Classification, yielding a 10-point accuracy improvement even with the same base model (BERT).

B. Data Augmentation Strategy

To overcome the scarcity of training data (only ~400 examples per subtask) and to enhance lateral reasoning capabilities, the authors augmented the training set with two external sources:

GPT-4 Generated Humor Data: A synthetic dataset of joke-style multiple-choice questions generated via prompting. The authors hypothesized that jokes, like brain teasers, rely on subverting expectations and lateral thinking.
RiddleSense Dataset: A public dataset of riddles requiring creative, non-literal reasoning. The authors adapted this dataset from 5-choice to 4-choice to match the task format.

C. Model Selection

The system utilized DeBERTaV3-base as the primary backbone, initialized with pre-trained weights. The authors also experimented with BERT-base. They combined the original Sentence and Word Puzzle datasets to broaden exposure to different lateral thinking patterns.

3. Key Contributions

Architectural Superiority: Demonstrated that framing lateral thinking tasks as Multiple-Choice QA rather than Sequence Classification is critical, providing a substantial performance boost (+9.2% to +10% accuracy).
Effective Data Augmentation: Proved that augmenting training data with humor and riddle datasets significantly improves performance on sentence-level lateral reasoning tasks.
Differential Analysis of Reasoning Types: Provided a nuanced analysis showing that data augmentation strategies are not universal. While humor/riddle data helped Sentence Puzzles, it did not transfer to Word Puzzles, suggesting that word-level reasoning (orthographic/phonetic) requires different data characteristics than semantic reasoning.
State-of-the-Art Performance: Achieved top-tier rankings (6th/31 and 10th/23) on the respective subtasks, significantly outperforming zero-shot ChatGPT baselines.

4. Experimental Results

The system was evaluated on the test set using Instance-based Accuracy (individual question correctness) and Group-based Accuracy (correctly solving the original question and all its adversarial variants).

Sentence Puzzle:
- Best Performance: 92.5% overall accuracy.
- Ranking: 6th out of 31 teams.
- Improvement: Outperformed the zero-shot ChatGPT baseline by nearly 50 percentage points.
- Key Drivers: Switching to DeBERTaV3 (+28.3%), using Multiple-Choice architecture (+9.2%), and adding Humor/RiddleSense data (+4.2%).
Word Puzzle:
- Best Performance: 80.2% overall accuracy.
- Ranking: 10th out of 23 teams.
- Observation: Adding RiddleSense or Sentence Puzzle data decreased performance. The best model used only the original Word Puzzle data + Humor data.
- Improvement: Outperformed zero-shot ChatGPT by ~40 percentage points.
Ablation Study Insights:
- The shift from BERT to DeBERTaV3 provided the largest single gain.
- The Multiple-Choice architecture was a decisive factor.
- Failure Mode: There is a notable gap between instance-based and group-based accuracy (e.g., 92.5% vs 87.5% for Sentence Puzzles), indicating that while models solve the original question, they often fail on adversarial semantic reconstructions, suggesting a lack of robust reasoning.

5. Significance and Future Directions

This work highlights that lateral thinking is a distinct cognitive capability that requires specific architectural and data strategies, separate from standard logical reasoning.

Significance: It establishes that humor and riddles are valid and effective proxies for training lateral thinking in LMs, particularly for semantic tasks.
Limitations: The study notes that word-level puzzles remain a harder challenge, likely because subword tokenization fails to capture the character-level manipulations required.
Future Work: The authors suggest exploring character-level representations, larger-scale synthetic data generation with better quality control, and ensemble methods that combine semantic and orthographic reasoning to bridge the gap in word puzzles.

In conclusion, the paper successfully demonstrates that by aligning task formulation (Multiple-Choice) with domain-specific data augmentation (Humor/Riddles), language models can be significantly enhanced in their ability to perform creative, non-linear reasoning.