💬 NLP

Don't Judge a Book by its Cover: Testing LLMs' Robustness Under Logical Obfuscation

This paper introduces Logifus and the LogiQAte benchmark to demonstrate that state-of-the-art large language models suffer significant performance drops when logical reasoning tasks are presented in obfuscated yet equivalent formats, revealing their reliance on surface patterns rather than deep semantic understanding.

Original authors: Abhilekh Borah, Shubhra Ghosh, Kedar Joshi, Aditya Kumar Guru, Kripabandhu Ghosh

Published 2026-02-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Abhilekh Borah, Shubhra Ghosh, Kedar Joshi, Aditya Kumar Guru, Kripabandhu Ghosh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Trick Question" Test

Imagine you have a very smart student who can ace a math test if the questions are written clearly. But the moment you rewrite the same math problem using fancy, confusing words or a different font, the student gets stuck and fails.

This paper argues that today's most advanced AI models (Large Language Models, or LLMs) are like that student. They are excellent at recognizing patterns they've seen before, but they struggle when a problem is logically the same but looks different.

The researchers wanted to see if these AIs truly "understand" logic or if they are just memorizing the way questions usually look. To test this, they created a new game called LogiQAte.

The Tool: "Logifus" (The Magic Shapeshifter)

To run their test, the authors built a tool called Logifus. Think of Logifus as a magical translator that takes a simple question and rewrites it in a way that is:

Logically identical: The answer hasn't changed at all.
Visually confusing: The words, structure, or symbols are completely different.

The Analogy:
Imagine a recipe that says, "Mix 2 cups of flour with 1 cup of sugar."

The Original: "Mix 2 cups of flour with 1 cup of sugar."
The Obfuscated Version: "Combine the white powder from the first bag (which holds two standard measures) with the sweet crystals from the second bag (which holds one standard measure)."

Both instructions result in the exact same mixture. A human chef understands they are the same. The researchers wanted to see if the AI would realize they are the same, or if it would get confused by the new wording.

The Four Challenges (The "Obfuscated" Tasks)

The researchers tested the AI on four types of puzzles, turning each into a "trick" version:

Logic Puzzles (Obfus FOL):
- Normal: "If it rains, the ground gets wet."
- Trick: "It is false that the ground stays dry whenever it rains."
- The Test: Can the AI see that these two sentences mean the exact same thing?
Family Trees (Obfus Blood Relation):
- Normal: "D is the wife of C, and C is the father of F. How is D related to F?" (Answer: Mother).
- Trick: "D is the wife of C, and C is the only son of F's grandfather. How is D related to F?"
- The Test: The AI has to untangle a long, winding family chain instead of a direct link.
Number Patterns (Obfus Number Series):
- Normal: "2, 4, 6, 8, ?" (Answer: 10).
- Trick: They replaced the numbers with planet names (Venus, Mars, Jupiter, Saturn) or even random-looking codes (like "a87ff...").
- The Test: Can the AI spot the pattern (adding 2) even when the numbers are hidden behind symbols or code?
Direction Sense (Obfus Direction Sense):
- Normal: "Walk 5 miles North, then 3 miles East."
- Trick: "Walk 6 miles North, 4 miles East, 6 miles South, 3 miles East, 4 miles West, and finally 5 miles North."
- The Test: The extra steps cancel each other out (North then South = 0), leaving the same final destination. The AI has to ignore the "noise" to find the true path.

What Happened? (The Results)

The results were a wake-up call. When the questions were "obfuscated" (tricked up), the AI's performance crashed.

The Drop: Even the smartest models, like GPT-4o and GPT-5, saw their accuracy drop significantly. On average, their scores fell by 27% to 47%.
The "Reasoning Failure Zone": The researchers looked inside the AI's "brain" (its neural layers) while it was answering. They found that when the question was tricky, the AI's confidence in its own answers dropped sharply in the deeper layers of its network. It was like a student who starts to panic halfway through a test because the wording doesn't match what they memorized.
Memorization vs. Understanding: The study showed that when faced with these trick questions, the AI relied much more heavily on memorized patterns from its training data rather than actually thinking through the logic. It was trying to guess the answer based on how the question looked, not what it meant.

The Conclusion

The paper concludes that current AI models are great at pattern matching but not yet great at deep reasoning.

The Final Metaphor:
Think of these AI models as actors who have memorized a script perfectly. If you give them the script exactly as written, they perform brilliantly. But if you give them the same script with the words rearranged or the stage directions changed (even if the story is the same), they forget their lines. They haven't learned the story; they've only learned the script.

The researchers say we need to build models that understand the meaning behind the words, not just the words themselves, so they can handle these "trick questions" without failing.

Technical Summary: Don't Judge a Book by its Cover: Testing LLMs' Robustness Under Logical Obfuscation

Problem Statement

Despite the impressive capabilities of Large Language Models (LLMs) in standard reasoning tasks—such as solving arithmetic equations, evaluating truth tables, and completing syllogisms—these models exhibit significant fragility when faced with logically equivalent problems presented in obfuscated formats. While current benchmarks (e.g., MMLU, BIG-Bench Hard) often score only the final answer, allowing models to succeed via surface-level pattern matching or memorization, they fail to distinguish between genuine logical reasoning and the recall of training data patterns.

The core problem addressed is the inability of state-of-the-art LLMs to maintain reasoning consistency when the surface form of a problem is altered while its underlying logical structure remains invariant. As noted by critics and researchers (e.g., Smolinski, 2024; Vafa et al., 2024), models may produce impressive outputs without recovering the underlying world model, crashing when simple detours or rephrasings are introduced. This paper investigates whether LLMs truly reason or merely exploit memorized patterns by systematically testing their robustness against logical obfuscation.

Methodology

The Logifus Framework

To study this vulnerability, the authors introduce Logifus, a structure-preserving logical obfuscation framework. Logifus functions as a mapping $f: Q \to Q'$ , where $Q$ is a set of original questions and $Q'$ is a set of logically equivalent but syntactically diverse (obfuscated) counterparts. The core principle is that while the surface-level linguistic or structural representation is altered, the correct solution remains invariant. This compels models to engage in deeper reasoning rather than relying on surface recall.

The framework generates obfuscated variants through specific transformation strategies tailored to different reasoning domains:

Logical Equivalences: Applying rules like De Morgan's laws or double-negation elimination to First-Order Logic (FOL) premises.
Relational Chains: Rewriting direct kinship statements into indirect, multi-hop descriptions (e.g., transforming "A is the brother of B" to "A is the sister-in-law's husband of B").
Symbolic Substitutions: Replacing numerical values in sequences with deterministic but opaque tokens, such as planet names, ASCII sums of those names, or MD5 hashes.
Spatial Detours: Inserting self-canceling vector pairs (e.g., walking North then South) into navigation paths to increase complexity without changing the net displacement.

The LogiQAte Benchmark

Leveraging Logifus, the authors present LogiQAte, a diagnostic benchmark comprising 1,108 questions across four distinct reasoning tasks:

Obfus FOL: 272 questions evaluating first-order logic entailment under equivalence-preserving rewrites.
Obfus Blood Relation: 300 questions testing family-graph reasoning under indirect relational chains (Level 1 and Level 2 complexity).
Obfus Number Series: 284 questions requiring inductive pattern detection under symbolic substitutions (Planet Name, ASCII Sum, and MD5 Hash types).
Obfus Direction Sense: 252 questions assessing navigation reasoning under altered directions and reference frames.

Validation: To ensure integrity, every base question and its obfuscated version underwent a rigorous validation process involving two human annotators and, for FOL tasks, the automated theorem prover Prover9. This ensured near-perfect agreement (Cohen's $\kappa > 0.88$ ) on logical equivalence and conclusion preservation.

Evaluation Setup

The benchmark was evaluated on nine state-of-the-art models (both open- and closed-source), including general-purpose models (GPT-4o, Claude 3.7 Sonnet) and reasoning-focused models (GPT-5, o4-mini, Qwen QwQ-32B, Gemini 2.5 Pro). Evaluations were conducted under zero-shot, few-shot, and Chain-of-Thought (CoT) prompting settings with a temperature of 0.0. Performance was measured using Exact Match (EM) Accuracy.

Key Results

Performance Degradation

Across all tasks and models, logical obfuscation severely degrades zero-shot performance.

Average Drop: The average performance drop across all tasks and settings was 28.8%.
Model Specifics:
- GPT-4o: Dropped by 47% on average.
- GPT-5: Dropped by 27% on average.
- o4-mini: Dropped by 22% on average.
- Reasoning Models: Even models specifically optimized for reasoning (e.g., GPT-5) saw significant declines, such as falling from 98% to 56% on Obfus FOL (zero-shot).
- General-Purpose Models: Experienced more drastic drops, with some tasks showing declines of up to 87% (e.g., GPT-4o on Obfus Number Series).

Task-Level Vulnerabilities

Obfus Blood Relation: Showed a steady decline as complexity increased from Level 1 to Level 2. For instance, Claude 3.7 (CoT) dropped from 59% on base questions to 24% on Level 2 obfuscation.
Obfus Number Series: Proved particularly punishing for general-purpose models, which struggled to identify patterns when numbers were replaced by planet names or MD5 hashes.
Obfus Direction Sense: Highlighted weaknesses in state tracking, with models failing to filter out self-canceling detours.

Prompting Strategies

Advanced prompting strategies like Few-shot and Chain-of-Thought (CoT) provided only partial relief, not robustness. While CoT reduced GPT-5's degradation from 27% to 18.7%, the effect remained inconsistent across models. Notably, smaller models (e.g., GPT-4o-mini, Llama-4-Maverick) exhibited even greater brittleness, with accuracy collapsing significantly under obfuscation.

Mechanistic Analysis

To understand the failure modes, the authors conducted intrinsic analyses on LLaMA 3.1 8B:

Memorization Sensitivity: Using Min-K%++ membership inference, the study found that obfuscated questions triggered higher reliance on memorized patterns. Prediction AUROC scores increased by 12–15% for Obfus FOL and 20–25% for Obfus Blood Relation, indicating that models lean more heavily on training data patterns when surface forms are obscured.
Layer-wise Confidence: Analysis of next-token log probabilities revealed a 50–80% collapse in confidence in the late layers (28–31) of the transformer when processing obfuscated questions compared to base questions. This suggests that obfuscation disrupts the deep-layer semantic consolidation required for genuine reasoning.

Significance and Contributions

The paper makes three primary contributions:

Logifus: A novel, structure-preserving algorithm for generating logically obfuscated questions that maintain logical equivalence while altering surface syntax.
LogiQAte: The first-of-its-kind diagnostic benchmark comprising 1,108 questions across four reasoning tasks, designed to rigorously evaluate LLM robustness against logical obfuscation.
Empirical Evidence: A comprehensive analysis demonstrating that current LLMs, including state-of-the-art reasoning models, parse questions without deep understanding. The findings highlight a critical gap between pattern matching and robust reasoning, showing that models fail when the "cover" of the problem is changed, even if the "content" (logic) remains the same.

The authors conclude that the observed degradation underscores the urgency of building models that genuinely comprehend and preserve meaning beyond surface form, rather than simply memorizing and matching patterns learned from training data.

Limitations

The authors note that LogiQAte is currently limited to English. Future work aims to expand the dataset to include multilingual data (particularly low-resource languages) and extend the framework to additional reasoning categories such as seating arrangements and commonsense inference.