💬 NLP

LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening

This paper introduces LLMEval-Logic, a rigorously verified Chinese benchmark for logical reasoning that employs expert auditing, Z3 formal verification, and adversarial hardening to reveal significant performance gaps in current frontier large language models.

Original authors: Ming Zhang, Qiyuan Peng, Yinxi Wei, Yujiong Shen, Kexin Tan, Yuhui Wang, Zhenghao Xiang, Junjie Ye, Zhangyue Yin, Zhiheng Xi, Shihan Dou, Tao Gui, Maxm Pan, Ruizhi Yang, Qi Zhang, Xuanjing Huang

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ming Zhang, Qiyuan Peng, Yinxi Wei, Yujiong Shen, Kexin Tan, Yuhui Wang, Zhenghao Xiang, Junjie Ye, Zhangyue Yin, Zhiheng Xi, Shihan Dou, Tao Gui, Maxm Pan, Ruizhi Yang, Qi Zhang, Xuanjing Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to think logically. You give it a riddle, and it has to figure out the answer based strictly on the rules you provided.

For a long time, the tests we used to check if these robots (Large Language Models, or LLMs) were getting smarter were like fill-in-the-blank worksheets. They were often generated by computers taking a math formula and turning it into a sentence. The problem? The robots learned to spot the "pattern" of the sentence rather than actually doing the math. They were cheating by guessing based on how the question looked, not by following the logic.

The paper introduces a new, much tougher test called LLMEval-Logic. Think of it as swapping those fill-in-the-blank worksheets for a real-life escape room.

Here is how it works, broken down into simple parts:

1. The "Real-Life" Stories (Forward Authoring)

Instead of computers generating fake questions, real humans who are experts in logic wrote these problems from scratch.

The Analogy: Imagine a music composer writing a song based on a specific rule (e.g., "If you play the trumpet, you must also play the tuba"). The test asks: "Which instruments can we use together?"
Why it matters: These stories feel like real situations (like scheduling a meeting or deciding who gets a job), so the robots can't just guess based on a pattern. They have to actually read and understand the story.

2. The "Truth Machine" (Z3 Verification)

Every single question on this test was checked by a super-strict "Truth Machine" (a computer program called Z3).

The Analogy: Imagine a referee in a game who has a calculator. Before the test is even released, the referee runs the rules through the calculator to make 100% sure the answer is correct. If the calculator says the answer is "A," but the human wrote "B," the question is thrown out and rewritten.
Why it matters: This guarantees that the test itself is perfect. There are no "trick questions" where the answer key is wrong.

3. The "Hard Mode" (Adversarial Hardening)

The researchers didn't stop at making the test hard; they made it harder on purpose. They used a team of AI agents to play "Devil's Advocate" against the questions.

The Analogy: Imagine you write a riddle. Then, a team of "tricksters" tries to break it. They say, "What if we add a confusing detail here?" or "What if we ask a follow-up question that changes the whole situation?" They keep rewriting the riddle until it becomes so complex that even a smart human might struggle, but it still has a logical answer.
The Result: They created a "Hard" version of the test with multi-step puzzles. You can't just solve one step; you have to keep the whole picture in your head while answering a chain of questions.

4. The "Grading Rubric" (Not Just Right or Wrong)

When the robots answered, the researchers didn't just check if the final answer was right. They checked how the robot translated the story into logic.

The Analogy: Imagine a student solving a math problem. If they get the right answer but used the wrong formula, a normal teacher might give them full credit. But this test is like a strict professor who says, "You got the number right, but you missed the rule about 'if and only if.' That's a failure."
The Score: They gave the robots two scores: one for the final answer, and one for how well they translated the story into a logical language.

What Did They Find?

The results were a bit of a shock to the industry:

The Ceiling is Low: Even the smartest, most advanced robots currently available only got about 37.5% of the "Hard" questions right.
The Translation Gap: Even when the robots got the final answer right, they often failed to translate the story into the correct logical rules. It's like a student who guesses the right answer on a multiple-choice test but can't explain why it's right.
The "Thinking" Difference: Some robots that were allowed to "think" (slow down and reason step-by-step) did much better than those that just spit out answers immediately. However, even the "thinkers" struggled with the hardest, multi-step puzzles.

The Bottom Line

This paper says: "We built a new, uncheatable, real-world logic test. When we ran our best robots through it, they failed more often than we expected. They are good at guessing patterns, but they are still terrible at strictly following complex rules in a changing environment."

The test is now open for anyone to use to see if their own robots can pass the escape room.

Technical Summary: LLMEval-Logic

Problem Statement

Evaluating large language models (LLMs) on natural-language logical reasoning is critical for high-stakes applications such as contract review, regulatory compliance, and clinical guideline checking, where conclusions must strictly follow stated premises. However, existing logical reasoning benchmarks suffer from three primary deficiencies:

Lack of Semantic Grounding: Many benchmarks are generated via reverse construction (sampling formal formulas and rendering them into text), resulting in template-like phrasing that allows models to predict answers via surface cues rather than genuine inference.
Insufficient Auditing of Formalization: Existing benchmarks often provide only coarse or unaudited formal annotations. They fail to evaluate whether a model's translation from natural language to formal logic (NL-to-FL) faithfully encodes local semantic units, logical relations, and constraints.
Rapid Saturation: Current benchmarks saturate quickly under frontier reasoning models, offering little headroom to discriminate between models or localize specific failure modes in complex reasoning tasks.

Methodology

The authors present LLMEval-Logic, a Chinese logical reasoning benchmark constructed through a three-stage, audited pipeline designed to address the above challenges.

1. Dataset Construction (Base Subset)

The Base subset (246 items) is built via forward authoring, where contributors with logic coursework write items based on realistic situational scenarios (e.g., eligibility rules, scheduling) rather than reverse-engineering formulas.

Expert Review & Normalization: Items undergo expert review for concept stability and reasoning validity, followed by a four-layer normalization pipeline (lexical, syntactic, semantic alignment, and type/parameter standardization) to ensure consistency.
Z3 Verification: Every item is verified using the Z3 SMT solver. The solver checks if the formalized premises and queries logically support the annotated answer under three task types: possible (satisfiability), necessary (entailment), and enumerate_models (model enumeration).
Expert Rubrics: Each item is paired with expert-developed rubrics that decompose NL-to-FL faithfulness into three atomic groups: logical_relation (e.g., implication, quantification), stated_constraint (facts, boundaries), and query_alignment. These rubrics exist in both natural language and Z3-checkable forms.

2. Adversarial Hardening (Hard Subset)

To prevent saturation, items that perform too well on initial model probing enter a closed-loop adversarial hardening workflow involving five agent roles: Decider, Proposal, Review, Answering, and Verification.

Hardening Strategies: The workflow rewrites items using six strategies: branching, valid distractors, explicit uncertainty, set-valued outputs, counterfactual variants, and alias/coreference variation. These strategies alter the closed candidate space or reasoning path rather than merely rewording the surface.
Hard Subset: The resulting Hard subset contains 190 items comprising 938 multi-step sub-questions. An item is only considered correct if every sub-question is correct, forcing models to maintain the full candidate space across related queries.

3. Evaluation Protocol

The benchmark evaluates 14 frontier LLMs across two axes:

Answer Accuracy: Measured via Item Accuracy (all sub-questions correct) and Sub-Question Accuracy.
Formalization Accuracy: Evaluated under Free (model chooses symbols) and Fixed (reference symbols provided) settings. Performance is scored using two complementary signals:
- Z3 Execution: Checks if the solver-derived answer matches the reference.
- Rubric Matching: Checks if the model's formalization passes the expert-developed atomic checklist.
- Joint Score: Requires agreement between Z3 and Rubric signals.

Key Results

Evaluations of 14 frontier LLMs (including GPT-5.4 Pro, Claude Opus 4.6, Gemini 3.1 Pro, and various Chinese models) reveal significant gaps in current capabilities:

Performance Drop: While the best model achieves 75.5% accuracy on the Base subset, performance drops sharply to 37.5% on the Hard subset.
Formalization Gaps: Even with reference symbols provided, the highest joint Z3+Rubric formalization score is only 60.16%, indicating that models struggle to encode premises with the required semantic strength even when the symbol inventory is fixed.
Ranking Divergence: There is a strong negative correlation ( $\rho = -0.61$ ) between Base and Hard rankings. Models that excel at single-question reasoning (e.g., Seed 2.0 Pro) often collapse on multi-question adversarial tasks, while others (e.g., Claude Opus 4.6) show relative stability.
Failure Modes: Analysis identifies specific failure patterns, including:
- Modal Quantifier Tracking: Confusing existential and universal queries.
- Incomplete Enumeration: Failing to traverse the full constraint space.
- Counterfactual Recomputation: Patching local edits without globally recomputing dependent branches.
- Candidate Space Maintenance: Losing track of the full set of feasible assignments when handling branching or distractors.

Significance and Claims

The paper claims that LLMEval-Logic provides a more rigorous and discriminative evaluation of logical reasoning by:

Eliminating Template Artifacts: Through forward authoring from realistic scenarios, the benchmark resists surface-pattern shortcuts.
Auditing Faithfulness: By combining Z3 execution with expert rubrics, it evaluates the process of formalization (faithfulness to source semantics) rather than just the final answer.
Sustaining Difficulty: The adversarial hardening workflow ensures the benchmark remains challenging for frontier models, exposing specific limitations in maintaining closed candidate spaces and performing global recomputation.

The authors conclude that current frontier models, while capable of single-shot reasoning, still lack the robustness required for multi-step, adversarially hardened logical tasks, and that faithful natural-to-formal translation remains a critical open challenge. The benchmark, workflow, and audit traces are released publicly to facilitate further research.