Technical Summary: Absurd World

Problem Statement

Despite the versatility of Large Language Models (LLMs) in various tasks, their reasoning capabilities remain a point of scrutiny, particularly when models fail to solve problems that humans can systematically address. Current literature often focuses on breaking LLM reasoning by increasing task complexity, memory horizons, or knowledge requirements (e.g., the "Tower of Hanoi"). However, the robustness of LLMs in simple logical reasoning within simple worlds remains underexplored.

Existing benchmarks often conflate reasoning difficulty with task complexity or memory demands. Consequently, it is unclear whether LLM failures stem from an inability to execute explicit rules or from an over-reliance on pre-trained world priors (memorized heuristics) that contradict the specific instructions provided in the prompt. There is a need for a benchmark that isolates basic rule execution in small, fully specified tasks where memory is not a limiting factor, allowing for the systematic identification of when and why models fail to follow stated rules.

Methodology: The Absurd World Framework

The paper proposes Absurd World, a controllable benchmarking framework designed to test whether LLMs execute stated rules or lean on pretrained world priors. The core methodology involves deconstructing a real-world scenario into symbolic primitives and systematically altering them to create "absurd" but logically coherent worlds.

1. De-structuring and Reconstruction

The framework breaks a real-world model into four components:

• Symbols: Actors and objects (e.g., players, balls, nets).
• Actions: Interactions producing events (e.g., hitting, missing).
• Rules: Constraints and consequences (e.g., scoring mechanisms, winning conditions).
• Sequences: The temporal ordering of events.

To generate an "Absurd World," the system modifies these components while keeping the underlying algorithmic structure (counting, bookkeeping) computationally trivial. Modifications include:

• Swapping symbol roles (e.g., a team shoots a net into a ball).
• Modifying actions (e.g., missing the net earns points).
• Altering rules (e.g., the team with the lowest score wins).
• Reordering event sequences.

2. Experimental Setup: Absurd Soccer

The primary testbed is a modified penalty-style soccer shootout called "Absurd Soccer."

• Baseline (REAL): Standard rules where hitting the net yields 1 point, and the highest score wins.
• Variations: The study introduces specific rulesets to test different failure modes:
  • MISSING: Scoring increases when a shot misses the net (Action modification).
  • LEAST: The team with the lowest score wins (Rule modification).
  • SWITCH: Teams shoot a net into a ball (Symbol role reversal).
  • ICE CREAM / CAR: Points are replaced by ice cream or cars (Symbol modification).
  • MISS & SWITCH: A combination of action and symbol changes.

3. Evaluation Protocol

The task, "Determine Outcome" (DO), requires models to read a match commentary and a specific ruleset to identify the winner.

• Prompting Techniques:
  • Zero-Shot (DO-0): Models answer directly based on the prompt.
  • Few-Shot (DO-FS): Models are provided with three correct question-answer pairs before the test question.
• Model Categories: The study evaluates a diverse set of models from OpenRouter, categorized as:
  • Cheap Non-Reasoning: Lower cost, standard inference.
  • Expensive Non-Reasoning: Higher cost, standard inference.
  • Reasoning Models: Specialized for complex reasoning (e.g., Chain-of-Thought).
• Metrics: Accuracy is measured as the proportion of correct responses over 100 trials per ruleset and model.

Key Contributions

1. A Novel Benchmarking Framework: Absurd World isolates rule execution failures by decoupling reasoning complexity from world familiarity. It forces models to override prior knowledge in favor of explicit, albeit absurd, instructions.
2. Systematic Perturbation Method: The paper introduces a method to automatically generate absurd variants of real-world tasks by manipulating symbols, actions, and rules, enabling controlled attribution of failures to specific types of rule changes.
3. Empirical Analysis of Prompting and Model Types: The study provides a comparative analysis of how different model classes (cheap vs. expensive, reasoning vs. non-reasoning) and prompting strategies (zero-shot vs. few-shot) perform under these specific perturbations.

Key Results

The experiments yielded three primary findings:

1. Sensitivity to Rule Changes: Non-reasoning models (both cheap and expensive) are significantly more sensitive to rule changes than reasoning models. While reasoning models achieved near-perfect accuracy (100%) across all rulesets, non-reasoning models showed a marked drop in performance, particularly in the MISSING and MISS & SWITCH scenarios. Expensive non-reasoning models performed significantly worse than cheap non-reasoning models.
2. Degradation via Few-Shot Prompting: Contrary to the expectation that in-context learning improves performance, Few-Shot prompting (DO-FS) consistently degraded performance compared to Zero-Shot prompting (DO-0) across all model categories. The authors hypothesize this indicates a lack of familiarity with the specific few-shot technique used or interference from the examples with rule execution.
3. Performance Disparity: Expensive non-reasoning models underperformed both cheap non-reasoning models and reasoning models across all rulesets. Entropy analysis suggested that incorrect answers from expensive models had higher entropy, implying lower confidence, though this did not provide a conclusive explanation for the failure.

Significance and Claims

The paper claims that Absurd World serves as an effective tool to determine an LLM's ability to think logically by ignoring patterns learned from the real world. The significance of the work lies in its ability to:

• Identify Robustness: Verify whether an LLM's reasoning capability is robust against variations of a task that violate real-world expectations.
• Guide Model Selection: Provide a practical tool for assessing reliability and guiding model selection and deployment, particularly for agentic AI where simple, reliable task execution is critical.
• Isolate Failure Modes: Systematically identify that failures often arise not from memory limits or task complexity, but from the model's inability to execute explicit rules when they contradict strong pre-trained priors.

The authors conclude that while LLMs are powerful, their reasoning is fragile when the task's logic deviates from real-world expectations, highlighting the need for benchmarks like Absurd World that move beyond standard realism-based evaluations.