💬 NLP

From 0-Order Selection to 2-Order Judgment: Combinatorial Hardening Exposes Compositional Failures in Frontier LLMs

This paper introduces LogiHard, a formal framework that transforms simple selection tasks into complex logical judgments via combinatorial hardening and adaptive testing, revealing that frontier LLMs suffer from significant accuracy degradation due to a training-induced combinatorial reasoning gap rather than knowledge deficits.

Original authors: Hanmeng Liu, Shichao Weng, Xiulai Liu, Zhicai Zhang, Anli Yan, Xiaozhang Liu

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hanmeng Liu, Shichao Weng, Xiulai Liu, Zhicai Zhang, Anli Yan, Xiaozhang Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are testing how smart a group of students is by giving them a multiple-choice quiz. For a long time, these quizzes have been easy to "game." The students (in this case, AI models) have memorized the answers or learned to spot tiny tricks, like "the answer is usually the longest sentence" or "the third option is rarely right." They get 90%+ scores, but are they actually thinking, or just pattern-matching?

This paper introduces a new way to test them called LogiHard. Think of it as upgrading the test from a simple "pick the right picture" game to a complex "solve the puzzle" challenge that forces the brain to do real math.

Here is the breakdown of what they did and what they found, using simple analogies:

1. The Problem: The "Trick-or-Treat" Test

Current AI tests are like a game of "Simon Says" where the rules are too obvious.

The Old Way: Researchers tried to make tests harder by changing the words (synonyms) or shuffling the order of answers.
The Flaw: This is like putting a new coat of paint on a broken car. The AI just ignores the paint and picks the answer based on the old, memorized pattern. It's still a "Level 1" task: Look at the options, pick the one that looks right.

2. The Solution: LogiHard (The "Logic Gym")

The authors built a new framework called LogiHard. Instead of just changing the paint, they changed the engine of the test.

The Analogy: Imagine a standard test asks, "Is the light on? A) Yes, B) No."
- LogiHard takes that same question and turns it into: "If the light is on, AND the switch is broken, OR the bulb is burnt out, which of these statements are true? Select all that apply."
The Shift: They moved the test from 0-Order Selection (just picking one answer) to 2-Order Judgment (evaluating a complex web of "If/Then/And/Or" rules).
The "Validity" Guarantee: They didn't just make up random hard questions. They used a strict mathematical recipe (like a chef following a perfect formula) to ensure every new question is logically sound. If the AI gets it wrong, it's because it failed the logic, not because the question was broken.

3. The "Smart Coach" (Adaptive Testing)

Usually, tests give everyone the same 50 questions. If the AI is a genius, it gets bored on the easy ones. If it's struggling, it gets frustrated on the hard ones.

LogiHard's Coach: They used a system called IRT-CAT (think of it as a personal trainer).
- If the AI gets a question right, the coach immediately picks a harder one.
- If it gets one wrong, the coach picks an easier one to find the exact limit of its ability.
- Result: They can measure the AI's true intelligence with only 15–20 questions instead of 50, saving time and energy.

4. The Big Surprise: The "Combinatorial Collapse"

The researchers tested 12 of the smartest AI models in the world (including models from OpenAI, Google, and others) on this new test.

The Result: The AI models crashed.
- On normal tests, they scored around 80–90%.
- On the LogiHard tests, their scores dropped by 31% to 56%.
- Some models that were "super smart" on normal tests fell to near-zero accuracy on the hardest logic puzzles.

Why did they fail?
The paper found two main reasons, using a "Two-Stage" analogy:

Stage 1 (The Brain): The AI could understand the individual facts perfectly. (e.g., "The light is on.")
Stage 2 (The Assembly): The AI failed to put those facts together into a complete list of all correct answers.
- The "Early Exit" Bug: The AI models are trained to find one good answer and stop. They are like a student who sees one correct option on a "Select All" test, circles it, and walks away, even though they knew two other options were also correct. They lack the "metacognitive trigger" to check, "Wait, did I miss anything?"

5. The Human Comparison

The researchers also had 30 human volunteers take the test.

Humans: Scored around 79.5%. They handled the complex logic well.
AI: Even the best AI models scored significantly lower than humans on these specific logic puzzles.
The Takeaway: The AI isn't "dumb"; it just has a specific blind spot. It is great at memorizing patterns and finding single answers, but it struggles when asked to juggle multiple logical rules at once and list every possible outcome.

Summary

The paper argues that we can no longer trust standard AI benchmarks because the models have "cheated" by memorizing patterns. LogiHard is a new, mathematically rigorous way to force AI to do real, multi-step reasoning. The results show that even the most advanced AI today has a fundamental gap: it can understand logic, but it cannot reliably compose complex logic to find all possible answers. It's a "combinatorial reasoning gap" that current training methods haven't fixed yet.

Technical Summary: From 0-Order Selection to 2-Order Judgment

Problem Statement

Multiple-choice question (MCQ) benchmarks for Large Language Models (LLMs) face two critical challenges: rapid saturation due to advancing model capabilities and data contamination that undermines static evaluations. While Large Reasoning Models (LRMs) have achieved superhuman accuracy on standard logical reasoning benchmarks (e.g., LogiQA, MMLU), this performance often stems from training set memorization and exploitation of surface patterns (position bias, lexical overlap) rather than genuine logical entailment.

Existing "hardening" methods, such as paraphrasing, adversarial perturbations, or "None of the Above" (NOTA) distractors, attempt to increase difficulty but fail to address the core issue. These approaches remain 0-order selection tasks where the model identifies a single correct option among candidates. They do not elevate the logical order of reasoning, allowing advanced models to bypass deep logical constraints through extended chain-of-thought (CoT) verification or pattern matching. Consequently, current benchmarks fail to challenge state-of-the-art reasoning models or expose fundamental gaps in compositional logic.

Methodology: The LogiHard Framework

The authors propose LogiHard, a formal framework that deterministically transforms 0-order MCQs into 2-order logical judgment tasks. This transformation increases thinking overhead and reasoning steps while ensuring validity-by-construction. The framework consists of three synergistic components:

1. Deterministic Combinatorial Protocol

Instead of stochastic perturbation, LogiHard maps atomic options to propositional variables and synthesizes compound formulas using logical connectives ( $\land, \lor, \neg$ ).

Transformation: An atomic question $Q = (C, O, a)$ is converted into a combinatorial question $Q' = (C', \Phi, A)$ . The context $C'$ includes natural language statements corresponding to atomized options. The formula set $\Phi$ contains compound propositions (e.g., exactness, disjunction, negation, compound negation) that the model must evaluate under the ground-truth assignment derived from the original answer.
Validity-by-Construction: The correct answer set $A$ is logically consistent with the atomic ground truth. The atomic option never appears directly in the output; only its propositional counterpart within compound formulas does. This renders memorization of surface patterns ineffective.
Difficulty Tiers: Complexity is calibrated via operator hierarchy:
- Easy: Exactness only ( $\land$ ).
- Medium: Adds Disjunction ( $\lor$ ).
- Hard: Adds Negation ( $\neg$ ).
- Expert: Adds Compound Negations.

2. Cognitive Difficulty Scoring

To ensure the source questions are intrinsically challenging before transformation, the framework employs a cognitive scoring module.

Metrics: It extracts nine dimensions from model thinking traces (e.g., oscillation points, logic density, abductive depth, inference chain length).
Gold Score: These metrics are aggregated into a weighted "Gold Score" to stratify questions into tiers (Easy, Medium, Hard, Expert). Only Hard and Expert tiers undergo combinatorial transformation to prevent floor effects.

3. Dynamic Evaluation via IRT-CAT

LogiHard integrates Item Response Theory (IRT) and Computerized Adaptive Testing (CAT) to address the inefficiency of static benchmarks.

Adaptive Selection: The system dynamically selects the most informative items based on the model's current ability estimate ( $\hat{\theta}$ ), maximizing Fisher information.
Dual-Subset Estimation: It separately estimates ability on atomic (0-order) and combinatorial (2-order) reasoning, allowing for a precise measurement of the "combinatorial gap" ( $\Delta\theta$ ).
Efficiency: This approach achieves measurement precision equivalent to static 60-item tests using only 15–25 adaptively selected questions.

Key Contributions

Validity-Guaranteed Hardness Synthesis: A deterministic protocol that elevates reasoning from 0-order selection to 2-order judgment. It resists memorization by ensuring the correct answer determines the truth assignment without the atomic option appearing in the output, eliminating the semantic drift found in adversarial methods.
Dynamic Difficulty Control: A unified system combining tiered operator configurations for generation-time calibration and IRT-based CAT for precise, efficient ability targeting.
Identification of Reasoning Failure: The instantiation of LogiHard-2k, a dataset of 2,000 high-stakes examination questions, reveals a fundamental multi-select failure and combinatorial reasoning gap in frontier models.

Experimental Results

The framework was evaluated across twelve state-of-the-art models (including GPT-5.4, o3, Claude-Opus-4.6, DeepSeek-R1, and Qwen3.6-plus) and compared against human evaluators.

Accuracy Degradation: Models exhibited severe performance drops on combinatorially hardened questions, with accuracy degradation ranging from 31% to 56%. For instance, Qwen3.6-plus dropped from 51.67% (H-Base) to 5.00% (H-Comb).
Resilience Tiers: Models were categorized into three tiers based on the latent ability drop ( $\Delta\theta$ ). The most resilient models (e.g., GLM-5, GPT-5.4) showed a drop of $\approx 1.05–1.11$ , while others (e.g., GLM-4.7) suffered a collapse of $\Delta\theta = 3.14$ .
Human vs. Machine Asymmetry: Human evaluators achieved 79.5% accuracy on Hard-Combinatorial questions ( $\Delta\theta \approx 0.28$ ), significantly outperforming even the most resilient models.
Multi-Select Bottleneck: Disaggregation revealed a specific failure mode: accuracy dropped from 31.2% for single-correct-option questions to 12.4% for multi-select questions ( $n_{correct} \ge 2$ ).
Zero-Shot Transfer: When applied to MMLU without domain adaptation, the framework showed a 47% accuracy degradation (89.84% $\to$ 42.86%), confirming the gap is domain-agnostic and stems from reasoning architecture rather than knowledge deficits.

Mechanistic Analysis and Significance

The paper identifies the root cause of failure as a combinatorial mapping gap rather than a lack of logical knowledge.

Two-Stage Failure: Models successfully perform Stage 1 (evaluating atomic truth assignments) but fail at Stage 2 (translating these truths into complete multi-select option sets).
Training-Induced Bias: The failure is attributed to an early-exit bias in training pipelines. Models are optimized for single-answer selection ("first correct answer wins") and lack metacognitive triggers for set-valued completeness verification. This leads to under-selection, where models identify valid options but fail to enumerate the complete set.
Significance: LogiHard demonstrates that current frontier models lack robust propositional constraint satisfaction. The framework provides a domain-agnostic, contamination-resistant evaluation protocol that isolates reasoning capability from domain knowledge. It suggests that future training regimes must incorporate deterministic combinatorial elevation to instill set-valued constraint satisfaction, moving beyond the current reliance on surface-level benchmark saturation.

The authors conclude that the distinction between "knowing logic" and "composing logic" remains a persistent ceiling for machine reasoning, and that static benchmarks are approaching obsolescence for evaluating true reasoning capabilities.