Technical Summary: The Riddle Riddle Paradigm

Problem Statement

While Large Language Models (LLMs) demonstrate high accuracy on various cognitive benchmarks, it remains unclear whether this performance stems from flexible reasoning or the retrieval of surface-level patterns from training data. As reasoning itself becomes part of training corpora, models may learn to mimic the appearance of reasoning (e.g., recognizing a riddle structure and outputting a "trick" answer) without evaluating the actual content of the problem. Existing benchmarks often fail to distinguish between a model that has memorized a solution and one that has dynamically selected an appropriate reasoning strategy. The core challenge is to design a paradigm that separates the surface structure of a problem from its content requirements.

Methodology

The authors introduce the Riddle Riddle Paradigm, a diagnostic framework utilizing matched pairs of word problems to test strategy selection.

Stimuli Design

The paradigm consists of 30 matched pairs, each containing two conditions:

1. Condition A (Genuine Riddle): A standard riddle requiring inventive, non-literal reasoning to resolve an apparent contradiction (e.g., "A cowboy rides into town on Friday... and rides out on Friday," where "Friday" is the horse's name).
2. Condition B (Riddle Riddle): A modified version of the same riddle that preserves the syntactic structure, phrasing, and sentence length but removes the "trick." These problems require only literal interpretation (e.g., "A cowboy rides into town on Friday... and rides out on Monday," where the answer is simply a calculation of days).

This design ensures that any performance differences are driven solely by problem content rather than topic, format, or phrasing.

Experimental Setup

The study comprises two experiments:

• Experiment 1 (LLMs): Tested nine state-of-the-art models (including GPT-5.4, Claude-opus-4-6, Gemini-3.1-Pro, and open-source variants like Llama-3.3 and Qwen3.5). Each model received 30 riddles and 30 riddle riddles with 10 repetitions per item. Responses were coded for accuracy and reasoning type (inventive vs. literal). A "memorization check" was also performed using partial prompt completion to assess if high performance on genuine riddles was due to verbatim recall.
• Experiment 2 (Humans): 100 human participants solved 6 items (3 from each condition) in a controlled setting. Responses were coded similarly to the LLM data.

Analysis

The authors employed mixed-effects logistic regression to analyze accuracy and reasoning strategy selection. They compared performance across conditions and between solvers (humans vs. LLMs), specifically looking for "diagnostic errors":

• Overgeneralization: Applying inventive reasoning to a problem that only requires literal interpretation.
• Undergeneralization: Applying literal reasoning to a problem that requires inventive interpretation.

Key Results

1. Opposite Performance Patterns

The study found a clear dissociation between human and LLM performance:

• LLMs: Performed significantly better on Genuine Riddles (84.9% accuracy) than on Riddle Riddles (50.7% accuracy).
• Humans: Performed significantly better on Riddle Riddles (80.5% accuracy) than on Genuine Riddles (50.5% accuracy).

2. Strategy Selection and Error Analysis

• LLM Behavior: LLMs systematically overextended inventive reasoning. On Riddle Riddles (where literal reasoning was sufficient), 90.8% of LLM errors were caused by inappropriately applying inventive reasoning. Even when models switched strategies, they did so less flexibly than humans.
• Human Behavior: Humans showed a preference for literal reasoning but demonstrated greater flexibility. Their primary error mode was undergeneralization (57.6% of errors on Genuine Riddles), where they failed to switch to inventive reasoning when necessary.
• Memorization: A strong correlation ($\rho = .88$) was found between a model's ability to recall genuine riddles verbatim (via partial prompt completion) and its accuracy on those riddles. Crucially, higher memorization predicted lower accuracy on the matched Riddle Riddles, suggesting models were retrieving answers rather than reasoning.

3. Reasoning Correctness

When measuring whether the correct strategy was used (regardless of the final answer), LLMs showed a significant drop in correctness on Riddle Riddles compared to Genuine Riddles, confirming that the formatting alone triggered the wrong reasoning mode.

Key Contributions

1. The Riddle Riddle Paradigm: A novel experimental design that isolates surface features from content requirements, allowing researchers to test if a system selects strategies based on "what a problem looks like" versus "what a problem requires."
2. Evidence of Surface-Driven Reasoning: The study provides empirical evidence that current SOTA LLMs rely heavily on surface features (riddle-like structure) to trigger specific reasoning heuristics, even when those heuristics are inappropriate for the specific content.
3. Dissociation of Human and Model Cognition: The paper demonstrates that while both humans and LLMs make reasoning errors, the nature of these errors is inverted. Humans struggle to override literal defaults for creative tasks, whereas LLMs struggle to override creative defaults for literal tasks.

Significance and Claims

The authors claim that the Riddle Riddle paradigm reveals a fundamental limitation in current LLMs: they have not yet achieved flexible reasoning in the sense of dynamically selecting strategies based on problem demands. Instead, their high performance on standard riddles likely reflects the retrieval of memorized patterns or "reasoning templates" seen during training.

The paper argues that aggregate accuracy metrics can be misleading; a model may appear to reason well on a benchmark simply because the benchmark's structure consistently triggers a high-frequency pattern in the model's training distribution. The Riddle Riddle paradigm exposes this by creating a "trap" where the structure suggests one strategy (inventive) but the content requires another (literal).

The authors conclude that distinguishing between genuine reasoning and pattern matching is critical for interpretability and trust, especially as LLMs are deployed in domains requiring novel problem-solving. They modestly suggest that while current models are sensitive to the frequency of task patterns in training data, they lack the capacity to evaluate whether a specific instance actually requires that pattern.