💬 NLP

Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing

This paper evaluates the formal reasoning capabilities of four large language models within General Game Playing environments, revealing that while current models show significant progress, their performance degrades with increased task complexity and step depth, often due to specific reasoning errors like hallucinated rules and syntactic mistakes.

Original authors: Maciej Świechowski, Adam Żychowski, Jacek Mańdziuk

Published 2026-02-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Maciej Świechowski, Adam Żychowski, Jacek Mańdziuk

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Great Logic Game: Can AI Actually Think, or Just Guess?

Imagine you are teaching a super-smart robot to play a brand-new board game. You don't show it the board or the pieces. Instead, you hand it a thick, dry rulebook written in a strange, code-like language (called GDL). Your robot has never seen this game before. It has to read the rules, figure out what moves are legal, and predict exactly what the board will look like after you make a move.

This is exactly what the researchers in this paper did. They wanted to see if modern AI (Large Language Models, or LLMs) can truly reason like a human, or if they are just guessing based on patterns they've seen before.

Here is the breakdown of their experiment, the results, and what it all means, explained with some fun analogies.

1. The Setup: The "Blindfolded Chess" Test

Usually, when we ask an AI a question, it answers based on its training data (like a student who memorized the textbook). But what if the question is about something the AI has never seen?

The researchers used a framework called General Game Playing (GGP). Think of this as a "blindfolded chess" test.

The Rules: They gave the AI a game description written in a strict logic language (GDL). It's like giving someone a recipe written in a secret code.
The Task: The AI had to act as a referee. It had to look at the current state of the game, read the rules, and say: "If Player A does X, the board will change to Y."
The Twist: To make sure the AI wasn't just recognizing words like "King" or "Pawn" from its training data, they scrambled the names.
- Original: "The Knight moves in an L-shape."
- Scrambled: "The Flurg moves in an L-shape."
- Random: "The X9z moves in a Q2p shape."

If the AI could still play the game correctly with scrambled names, it proves it was actually understanding the logic, not just remembering facts about chess.

2. The Contestants

They tested four different "brains":

Gemini 2.5 Pro: The heavyweight champion (very smart, very expensive).
Gemini 2.5 Flash: The speedy lightweight (fast, slightly less deep).
GPT-OSS 120B: A powerful open-source model.
Llama 3.3 70B: Another popular open-source model.

3. The Four Challenges

The AI had to pass four different tests, ranging from easy to impossible:

Level 1: The One-Step Hop. "Here is the board. Here is the move. What does the board look like next?"
- Result: The top models were amazing. They got this right almost 95-99% of the time. It was like a human reading a simple instruction and following it perfectly.
Level 2: The Rulebook Check. "Here is the board. What moves are legal right now?"
- Result: This got harder. The AI started missing some moves or inventing illegal ones. It's like a referee who knows the rules but forgets to check the fine print.
Level 3: The Time Traveler. "Here is the starting board. Here are 5 moves in a row. What does the board look like at the end?"
- Result: This is where things got messy. The AI had to remember the state after move 1, use that for move 2, and so on. Small mistakes in step 1 would ruin step 5. The top models were okay, but they started making errors.
Level 4: The Grandmaster. "Play the game for 5 steps yourself, making legal moves, and tell me the final state."
- Result: This was the hardest. The AI had to choose the moves and calculate the result. The performance dropped significantly. It's like asking someone to play a game of chess while blindfolded and reciting the moves out loud; eventually, they lose track of where the pieces are.

4. The Big Discoveries

🧠 The "Scrambled Name" Surprise

When the researchers replaced words like "King" and "Pawn" with nonsense words like "X9z" and "Q2p," the smartest models (Gemini 2.5 Pro) didn't crash. They still played the game well.

What this means: The AI isn't just memorizing that "Knights move in L-shapes." It is actually learning the structure of the rules. It's like a child who learns that "if you push a block, it falls," regardless of whether you call the block a "rock," a "brick," or a "thingy."

📉 The "Long Chain" Problem

The longer the chain of reasoning (more moves in a row), the worse the AI got.

The Analogy: Imagine a game of "Telephone." You whisper a message to a friend, who whispers it to the next, and so on. By the time it reaches the 10th person, the message is garbled.
The AI: Every time the AI calculates a new state, it has a tiny chance of making a mistake. If it makes a mistake in step 1, step 2 is built on a lie, and by step 10, the whole game is nonsense. The smarter models made fewer mistakes, but they still couldn't go forever without slipping up.

🚫 The "Hallucination" Trap

The researchers found that when the AI got stuck, it didn't just say "I don't know." It made things up.

Fake Rules: It would invent a rule that didn't exist in the code.
Ghost Pieces: It would keep pieces on the board that should have been removed.
Syntax Errors: It would write the answer in the wrong format (like writing a sentence when asked for a list).

5. The Verdict: Are They Smart or Lucky?

The Good News:
Modern AI has made a huge leap in logical reasoning. They can look at a complex, formal set of rules they've never seen before and follow them with high accuracy. They are becoming much better at "thinking" rather than just "predicting words."

The Bad News:
They are not yet perfect "logic engines."

They struggle with long chains of reasoning (like planning 10 steps ahead).
They get confused when the rules are very deep or complex.
They are prone to hallucinations (making up facts) when the pressure is high.

The Takeaway

Think of these AI models as brilliant interns.

If you give them a simple, one-step task, they are perfect.
If you give them a complex project with many steps, they will do a great job at first, but they might miss a detail halfway through, and then they might try to "fix" it by making up a solution that sounds good but is actually wrong.

Conclusion: We can trust them to help us reason, but we shouldn't trust them to do the final check. We need a human (or a strict computer program) to verify their work, especially for long, complex tasks.

1. Problem Statement

The paper addresses the critical question of whether Large Language Models (LLMs) can reliably perform formal, symbolic reasoning in structured, rule-governed environments. While LLMs excel at natural language tasks, their ability to act as faithful symbolic interpreters—following strict logical rules without hallucination or semantic bias—remains uncertain.

Current benchmarks often rely on natural language descriptions or single-step inference, which conflate logical reasoning with language understanding. The authors argue that to truly evaluate LLM reasoning, one must test them in environments where:

Rules are formally defined (not natural language).
Correctness can be objectively verified against a ground truth.
Reasoning requires multi-step state transitions and consistent variable binding.

2. Methodology

The authors utilize the General Game Playing (GGP) framework and the Game Description Language (GDL) as a rigorous testbed. GDL is a first-order logic programming language (based on Datalog) that defines game rules, initial states, and state transitions formally.

Experimental Setup

Models Evaluated: Four contemporary LLMs representing different architectures and sizes:
- Gemini 2.5 Pro (Proprietary, State-of-the-art)
- Gemini 2.5 Flash (Proprietary, Optimized for speed)
- Llama 3.3 70B (Open-weight)
- GPT-OSS 120B (Open-weight)
Dataset: A curated subset of 35 games selected from a pool of 86 GDL descriptions, chosen for structural diversity and complexity.
Tasks: Four distinct reasoning tasks were designed to test different aspects of logic:
1. Next State Generation: Given a state and a joint move, predict the resulting state.
2. Legal Actions Generation: Given a state, enumerate all valid moves for all players.
3. Multistep State Generation: Simulate a sequence of $n$ moves from the initial state to a final state.
4. Multistep Action-State Generation: Generate a valid sequence of $n$ legal moves and the resulting final state.
Semantic Obfuscation: To distinguish between symbolic reasoning and semantic memorization (e.g., knowing a "rook" moves horizontally because of chess training data), the authors created obfuscated versions of the games where symbols were replaced with:
- Placeholder terms (e.g., term1, term2).
- Dictionary words (semantically neutral nouns like "chair", "forest").
- Random strings (alphanumeric codes).
Metrics:
- Jaccard Index (JI): Measures the overlap between the set of facts predicted by the LLM and the ground truth.
- Strict Success Rate (%S): A binary metric indicating if the entire state was generated perfectly (JI = 1). This is crucial because a single error in a multi-step simulation leads to a completely incorrect final state.

3. Key Contributions

GGP as a Benchmark: The paper establishes GDL-based GGP as a robust, interpretable benchmark for evaluating formal reasoning, moving beyond natural language logic puzzles.
Comprehensive Model Evaluation: It provides a comparative analysis of four leading models across varying reasoning horizons (1 to 10 steps) and task complexities.
Semantic vs. Structural Analysis: By using obfuscation, the study isolates the models' ability to reason based on structural logic versus reliance on pre-trained semantic knowledge.
Error Characterization: The paper identifies specific failure modes, including hallucinated rules, redundant state facts, and violations of movement constraints.
Correlation with Game Properties: It analyzes how specific game structural features (e.g., rule depth, number of conditions) correlate with model performance degradation.

4. Key Results

Performance Hierarchy

Gemini 2.5 Pro consistently outperformed all other models across all tasks and horizons.
Gemini 2.5 Flash followed, showing strong performance but slightly lower strict success rates.
GPT-OSS 120B and Llama 3.3 70B showed significantly lower performance, particularly in multistep scenarios.
Statistical Significance: Wilcoxon signed-rank tests confirmed the performance hierarchy is statistically significant ( $p < 0.05$ ).

Task Difficulty & Reasoning Horizon

Single-Step (Task 1): All models (except Llama 3.3) achieved high accuracy (Avg JI > 0.8). Gemini 2.5 Pro achieved >95% strict success.
Multistep (Tasks 3 & 4): Performance degraded as the reasoning horizon ( $n$ $n$ ) increased.
- For $n=10$ , even the best model (Gemini 2.5 Pro) saw a drop in strict success rate, though it maintained a lead.
- Weaker models (Llama 3.3) exhibited non-monotonic fluctuations, suggesting reliance on spurious correlations rather than consistent derivation.
Action vs. State: Models were generally better at applying known actions (Next State) than discovering all possible legal actions (Legal Actions), likely because the latter requires exhaustive enumeration of complex rule dependencies.

Impact of Obfuscation

General Trend: Performance dropped under obfuscation, but the drop was mild for top models.
Semantic Leakage: The "Dictionary Words" variant was the most difficult, suggesting that misleading real-world semantics disrupt reasoning more than the absence of semantics.
Random Strings: Surprisingly, random strings often yielded better performance than placeholders, likely because they acted as unique identifiers, reducing token confusion.
Conclusion: Strong models (Gemini 2.5 Pro) can perform multistep symbolic reasoning even without meaningful identifiers, proving they rely on structural patterns rather than memorized game semantics.

Correlation with Game Properties

Gemini 2.5 Pro: Performance was most negatively correlated with nested rule depth (Max/Avg rule depth), indicating sensitivity to deep, recursive logic chains.
Gemini 2.5 Flash: Performance was most negatively correlated with the total number of rules, suggesting it struggles with large search spaces even if individual steps are shallow.
Llama 3.3 70B: Showed a strong positive correlation with the "Converging property" (where facts are only added, never removed), implying its reasoning is limited by the fundamental tractability of the game dynamics rather than specific logical features.

Common Failure Modes

Hallucinated Rules: Models implicitly "corrected" missing rules based on common sense (e.g., assuming items are lost when bought) rather than strictly following the provided GDL.
Redundant Facts: Models often retained facts in subsequent states that should have been discarded (violating the closed-world assumption).
Constraint Violations: In games with specific movement rules (e.g., only cardinal directions), models frequently proposed invalid diagonal moves.
Formatting Errors: Even strong models occasionally produced syntactically invalid GDL output (e.g., missing parentheses), leading to parsing failures.

5. Significance and Conclusion

The paper demonstrates that contemporary LLMs have made substantial progress in formal symbolic reasoning, capable of accurately simulating complex, rule-based environments without prior domain knowledge. However, they are not yet "faithful" symbolic simulators.

Reliability: While single-step reasoning is robust, error propagation in long-horizon tasks remains a critical limitation. Small local errors quickly compound, leading to incorrect final states.
Implications: For critical applications (e.g., decision support, autonomous agents), LLM outputs in formal reasoning tasks should not be trusted blindly. They require iterative verification, intermediate state checking, or hybrid approaches (LLM + symbolic solver).
Future Work: The authors suggest that future benchmarks should explore parameterizable game families to isolate specific complexity factors and test a broader range of models to understand scaling behaviors in symbolic reasoning.

In summary, the study confirms that LLMs are moving beyond pattern matching toward genuine logical inference, but their reliability in long, complex reasoning chains is still bounded by their tendency to hallucinate and their sensitivity to structural depth.