🤖 AI

A Syllogistic Probe: Tracing the Evolution of Logic Reasoning in Large Language Models

This paper investigates the evolution of logical reasoning in large language models by using existential import as a probe to demonstrate that model scaling, thinking mechanisms, and base model selection collectively drive a shift from traditional to modern syllogistic logic.

Original authors: Zhengqing Zang, Yuqi Ding, Yanmei Gu, Changkai Song, Zhengkai Yang, Guoping Du, Junbo Zhao, Haobo Wang

Published 2026-01-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhengqing Zang, Yuqi Ding, Yanmei Gu, Changkai Song, Zhengkai Yang, Guoping Du, Junbo Zhao, Haobo Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a group of students (the AI models) how to solve logic puzzles. For centuries, humans have debated how to solve these puzzles. There are two main schools of thought:

The "Old School" (Traditional Logic): This approach assumes that if you talk about something, it must exist. If you say, "All unicorns have horns," the Old School assumes unicorns are real and therefore concludes, "Some unicorns have horns."
The "Strict School" (Modern Logic): This approach is more rigorous. It says, "Just because you talk about unicorns doesn't mean they exist." If unicorns don't exist, the statement "All unicorns have horns" is technically true (because there are no counter-examples), but you cannot conclude that "Some unicorns have horns" because there are no unicorns to have horns.

This paper asks a simple question: As AI models get bigger and smarter, do they naturally switch from the "Old School" to the "Strict School"?

Here is the story of what the researchers found, using some everyday analogies.

1. The "Size" Effect: Bigger Isn't Always Better (Unless It's the Right Kind of Big)

The researchers tested many different AI models, from tiny ones to massive ones.

The Analogy: Imagine a library. A small library might only have storybooks that assume everything in them is real (Old School). As the library gets huge, it might just get more storybooks with the same old assumptions.
The Finding: For most AI families (like Llama and Gemma), making the model bigger just made it better at following the "Old School" rules. They got stronger at the old way of thinking.
The Exception: However, for one specific family (Qwen), making the model bigger caused a paradigm shift. As they grew larger, they started dropping the "Old School" assumptions and adopted the "Strict School" rules. They realized, "Wait, just because I'm talking about it doesn't mean it exists."

2. The "Thinking" Engine: Quality Over Quantity

The researchers discovered that you don't necessarily need a giant brain to think strictly; you just need to think hard.

The Analogy: Imagine a student taking a test.
- Scaling: This is like giving the student a bigger brain (more neurons).
- Thinking (RL): This is like giving the student a "scratchpad" where they are forced to write down every step of their reasoning before answering.
The Finding: A medium-sized model that was trained to "think step-by-step" (using a technique called Reinforcement Learning) performed just as well as a massive model that didn't think. The "Thinking" process acted like a turbocharger. It forced the model to check its logic rules rather than just guessing based on patterns it saw in its training data.
The Catch: Just telling a model "Let's think step by step" in a prompt wasn't enough. The model had to be trained to do this deeply. It's the difference between a student who is told to "try harder" versus a student who has been drilled on the method of solving problems.

3. The "Foundation" Matters: You Can't Build a Skyscraper on a Swamp

The researchers looked at the "Base" models (the raw, untrained versions of the AI) to see where they started.

The Analogy: Think of the Base model as the foundation of a house.
- If the foundation is already slightly tilted toward the "Strict School," it's easy to build the "Strict" house on top of it.
- If the foundation is solidly "Old School," trying to build a "Strict" house on top is shaky and unstable.
The Finding: The Qwen models that successfully switched to Modern Logic had a "foundation" that already showed hints of understanding the strict rules. The Llama and Gemma models had foundations that were deeply rooted in the Old School, so even after training, they struggled to make the switch.

4. The "Empty Room" Problem

The researchers found that AI still struggles when the subject of a sentence is "empty" (like unicorns, or a box with no apples in it).

The Analogy: It's easy for a human to say, "If there are no apples in the box, then 'Some apples are red' is false." But for an AI, if it can't see the apples, it gets confused. It tends to fall back on its training data, which is full of real-world examples where things do exist.
The Finding: When the logic puzzle involved "empty" things, the models were more likely to make mistakes and revert to the Old School thinking. This shows that their "logic" is still a bit tied to real-world knowledge rather than pure abstract rules.

5. The "Language" Bias

The models didn't think the same way in every language.

The Analogy: Imagine a student who learned math in English but speaks Spanish at home. They might solve a problem differently depending on which language they hear the question in.
The Finding: The AI's logic ability wasn't "language-neutral." A model might be a "Strict Logic" expert in English but a "Old School" thinker in Chinese (or vice versa, depending on the model). This suggests their reasoning is still heavily influenced by the specific patterns of the language they were trained on.

Summary

The paper concludes that AI logic isn't just a result of getting "bigger." It's a mix of:

The Foundation: Did the raw model start with a hint of modern logic?
The Training: Was it taught to "think" deeply and check its rules?
The Architecture: Some designs (like "Mixture of Experts") helped the shift happen faster.

The "Strict School" of logic is emerging in the smartest models, but it's a fragile transition. If the foundation is weak or the training isn't deep enough, the AI will happily go back to assuming that unicorns exist just because we talked about them.

Technical Summary: A Syllogistic Probe: Tracing the Evolution of Logic Reasoning in Large Language Models

1. Problem Statement

Human logic has historically evolved from intuition-driven inference (Aristotelian) to rigorous formal systems (Modern/Boolean logic). A critical distinction between these frameworks is Existential Import (EI): Traditional logic assumes that universal propositions (e.g., "All S are P") imply the existence of the subject class, whereas Modern logic treats them as conditional statements that remain true even if the subject class is empty.

Current Large Language Models (LLMs) demonstrate remarkable reasoning capabilities, yet existing benchmarks often suffer from a systematic bias. Most syllogism datasets implicitly default to Traditional Logic. Consequently, a model may score highly by adhering to Traditional shortcuts or score poorly by correctly applying Modern logic (refusing to infer existence from universal premises) and being marked "wrong." This paper addresses the need to disentangle a model's true logical reasoning ability from its alignment with specific evaluation conventions. The core question is: Do LLMs exhibit an analogous evolution from intuition-driven (Traditional) to rigorous formal (Modern) logic as they scale, and what mechanisms drive this shift?

2. Methodology

Dataset Construction

The authors constructed a new syllogism dataset using a multi-stage agent pipeline to ensure factual consistency and logical constraints.

Scale: 9,600 syllogisms generated from 100 concept triplets with empty minor-term extensions and 100 with non-empty extensions.
Variations: Includes Chinese and English versions, covering 15+9 syllogistic forms.
Logic Split: The dataset distinguishes between the 15 forms valid under Modern Logic and the 9 forms that are valid only under Traditional Logic (due to existential import).

Evaluation Metrics

The study evaluates models under two distinct criteria:

$Acc_t$ (Traditional Accuracy): Measures performance assuming all 24 moods are valid (Traditional Logic).
$Acc_m$ (Modern Accuracy): Measures performance assuming only the 15 Modern-valid moods are correct; the model must reject the 9 forms that commit the existential fallacy.
Consistency (Cons): The proportion of moods where the model provides consistent answers across all instances of that mood.

Experimental Design

The authors conducted extensive experiments on:

Model Families: Qwen (0.6B to 235B, including Dense and MoE), Llama 3, and Gemma 3 series.
Training Variants: Base models, Instruct (SFT), and Thinking (RL-trained) variants.
Closed-Source Models: Including GPT-5, Gemini-3, and Claude variants.
Ablation Studies:
- Prior-Check Prompt: Explicitly asking the model to check if terms are empty sets before answering.
- Chain-of-Thought (CoT): Prompting standard Instruct models to "think step by step."
- Distillation: Evaluating models distilled from RL-trained teachers (e.g., DeepSeek-R1).
- Architecture: Comparing Dense vs. Mixture-of-Experts (MoE) and Auto-regressive vs. Diffusion LLMs.

3. Key Findings and Results

3.1 Scaling Effects: A Family-Specific Shift

General Trend: As model parameters increase, Modern Logic accuracy ( $Acc_m$ ) generally rises across all families.
Qwen Series: Exhibits a distinct logic paradigm shift. Small-to-mid Qwen models (e.g., 0.6B–4B) strongly prefer Traditional Logic ( $Acc_t$ is high). However, larger Qwen models (especially 8B+ and Thinking variants) shift decisively toward Modern Logic ( $Acc_m$ becomes significantly higher than $Acc_t$ ).
Llama and Gemma Series: These families largely retain a Traditional Logic reasoning style even at larger scales (e.g., Llama-3-70B, Gemma-3-27B). Scaling improves their performance within the Traditional framework but does not trigger a shift to Modern logic.
Instability: During the transition (the "Turning Point"), consistency scores fluctuate, suggesting the model temporarily mixes surface pattern matching with rigorous reasoning.

3.2 Thinking as an Efficient Accelerator

RL-Driven Shift: "Thinking" variants (RL-trained) accelerate the shift to Modern Logic significantly more than parameter scaling alone. For instance, Qwen3-8B-Thinking achieves Modern Logic performance comparable to the much larger Qwen3-30B-Instruct.
Mechanism: Reinforcement Learning (RL) appears to enforce step-by-step, rule-based deduction, changing the decision criterion rather than just improving instruction following.
Limitations of Alternatives:
- CoT Prompting: Induces only a partial shift toward Modern Logic.
- Distillation: Distilling from RL-trained models (e.g., DeepSeek-R1) does not reliably produce strict Modern Logic behavior, suggesting that the transition requires task-aware design and specific post-training optimization, not just imitation.

3.3 Base Model Constraints

Starting Point: The Base model sets the initial logical stance. Models like Qwen3-8B-Base already show signals of Modern Logic (high recall for valid modern moods), making the subsequent shift easier and more stable.
Constraints: In contrast, Base models like Gemma and Llama often default to rejecting valid Modern moods (low recall), making the shift to Modern Logic harder and less stable even after post-training.

3.4 Additional Analysis

Prior-Check Prompt: Explicitly asking about empty sets increases $Acc_m$ without flipping the model's fundamental stance, confirming that the shift is not merely a prompt artifact.
Empty Minor Terms: Models struggle more with empty minor terms, often falling back to Traditional logic because constructing counterexamples for empty sets is less intuitive.
Cross-Lingual Gaps: Performance varies by language (e.g., Qwen/Llama perform better in Chinese; Gemma in English), indicating that logical reasoning is still tied to language-specific training patterns.
Architecture: MoE architectures in AR models correlate with a stronger tendency toward Modern Logic. Diffusion-based LLMs (dLLMs) largely follow Traditional Logic, with rare exceptions (e.g., LLaDA2.0-flash) that combine MoE and large scale.

4. Significance and Contributions

The paper claims to provide the first systematic trace of the evolution of logical frameworks in LLMs using existential import as a probe. Its primary contributions are:

Evidence of Evolution: Demonstrating that advanced LLMs (specifically the Qwen series and top-tier closed-source models) are evolving from intuition-driven Traditional logic to rigorous Modern logic, mirroring the historical evolution of human logic.
Decoupling Scale and Reasoning: Showing that while scaling helps, RL-based thinking is a more efficient driver for achieving Modern Logic reasoning, allowing smaller models to outperform larger non-thinking counterparts in logical rigor.
Base Model Dependency: Establishing that the Base model acts as a critical constraint; post-training can shift logic, but the stability and ease of this shift depend heavily on the Base model's initial alignment with Modern semantics.
Benchmarking Rigor: Highlighting the systematic bias in current evaluation methods and proposing a framework to distinguish between a model's reasoning capability and its adherence to specific logical conventions.

5. Limitations

The authors modestly note that their conclusions are specific to syllogistic reasoning and the existential import probe. It remains unclear if these evolutionary patterns hold for broader first-order logic. Furthermore, the study relies on final validity decisions (endpoint metrics) rather than diagnosing intermediate semantic representations, limiting the ability to pinpoint the exact mechanisms of the shift. The distillation findings are also noted as empirical observations within a specific setting rather than a general negative result for all distillation methods.