💻 computer science

Are you Talking Logic to Me? Assessing Language Models Syllogistic Reasoning Capabilities

This paper investigates how different formal Knowledge Representation notations and a novel syllogistic categorization method (SEF) enhance the syllogistic reasoning capabilities of Small Language Models, offering a competitive, faster alternative to natural language inputs through the open-source Common Logic Grammar Construction (CLGC) framework.

Original authors: Hanna Abi Akl, Fabien Gandon, Catherine Faron, Pierre Monnin

Published 2026-08-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hanna Abi Akl, Fabien Gandon, Catherine Faron, Pierre Monnin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to think. You might assume that if you just give the robot enough books to read, it will eventually figure out how to solve a logic puzzle, like a riddle or a math problem. But here's the catch: robots (or "Language Models," as scientists call them) are like super-fast parrots. They are amazing at mimicking human speech and writing stories, but when you ask them to follow strict logical rules, they often get confused. They might guess the answer based on how the sentence sounds rather than what it actually means. This is a big problem because we want these AI tools to be reliable helpers, not just chatty friends who make up facts. To fix this, researchers are trying to teach these robots to "speak" in different languages. Some languages are like our everyday words (Natural Language), while others are like strict math formulas or code (Formal Notations). The big question is: does speaking in a strict, logical code help the robot think better than speaking in normal English?

This paper, titled "Are you Talking Logic to Me?", dives right into that question. The authors, a team of researchers from France, decided to test if changing the "language" a robot uses to solve logic puzzles could make it smarter. They focused on a specific type of puzzle called a "syllogism." Think of a syllogism as a three-part logic chain: "All cats are furry. Fluffy is a cat. Therefore, Fluffy is furry." If the robot can't get this right, it can't be trusted with more complex tasks. The researchers took existing datasets of these puzzles and translated them into several different "formal" languages, ranging from standard computer logic (like FOL) to more abstract, symbol-heavy codes (like TFLPLUS) and even some new, simplified versions they invented called "MINIFOL." They then tested these puzzles on "Small Language Models" (SLMs)—which are like the lightweight, energy-efficient versions of the giant AI models we hear about in the news.

The team didn't just throw the puzzles at the robots; they tried two different teaching methods. First, they used "Supervised Fine-Tuning" (SFT), which is like giving the robot a crash course where they show it the puzzles and the correct answers over and over until it learns the pattern. Second, they used "Zero-Shot" (ZS) testing, where they just handed the robot a new puzzle and a set of instructions, asking it to figure it out on the spot without any prior practice. They also tried adding "reference sheets" to the instructions, giving the robot a definition of the puzzle type (like "This is a Disjunctive syllogism") to see if that extra context helped.

What did they find? It turns out that the "language" the robot speaks matters a lot, but it depends on how much the robot has studied. When the robots were given a crash course (SFT), the results were surprising. On smaller datasets, the most abstract, symbol-heavy languages (like TFLPLUS) actually worked better than normal English, even though the robots had seen way more English during their initial training. The researchers suggest this might be because the symbols in these abstract languages are simpler and less confusing for the robot to process. However, when the dataset got bigger, the robots started to prefer a mix of English and a compact code (CLIF). This combination seemed to make the robots more careful and less likely to guess wrong, especially when the answer was "Unknown."

In the "Zero-Shot" tests, where the robots had to rely on what they already knew, the results were a bit more mixed. The best language depended entirely on which specific robot model was being tested. Some models, which had been trained on lots of math and logic data, did better with the abstract codes. Others, trained mostly on text, did better with English-like formats. Interestingly, adding those "reference sheet" definitions to the instructions helped some models but hurt others, suggesting that there is no single "magic bullet" language for all robots.

One of the coolest discoveries was about speed. The robots solved the puzzles much faster when using the compact, symbol-based languages compared to normal English. It's like the difference between reading a long, winding story to find a specific fact versus looking up a number in a short, neat table. The abstract languages were not only faster to process but also sometimes led to more accurate reasoning. The authors conclude that while we can't just pick one perfect language for all AI, using a mix of natural language and formal codes might be the key to building smarter, more reliable reasoning systems. They even released a free tool called "CLGC" that lets anyone automatically translate these logic puzzles into different formats, hoping to help other researchers explore this "language of logic" further. Ultimately, the paper suggests that teaching AI to think isn't just about feeding it more data; it's about teaching it to speak the right language for the job.

Technical Summary: Are you Talking Logic to Me? Assessing Language Models Syllogistic Reasoning Capabilities

Problem Statement

Language Models (LMs) currently struggle with logical tasks, particularly syllogistic reasoning, which requires abstract thinking and consistency. While Knowledge Representation (KR) has been shown to aid models by providing unambiguous, interpretable input formats (such as First-Order Logic or FOL), existing research largely limits evaluation to Natural Language (NL) and FOL. Current datasets often suffer from issues regarding human involvement, size, diversity, and generalization. Furthermore, the impact of varying levels of abstraction in KR notations—ranging from natural to highly symbolic—on the reasoning capabilities of Small Language Models (SLMs) remains under-explored. Additionally, the potential benefit of explicitly categorizing syllogisms to guide reasoning has not been systematically integrated into prompting strategies.

Methodology

The authors propose a comprehensive protocol to address these gaps, centered on the Common Logic Grammar Construction (CLGC) framework.

1. The CLGC Framework

CLGC is an open-source Python library designed to automatically generate syllogisms in various formal notations and classify them.

Generation Process: The system starts with syllogisms in FOL (defined by a Backus-Naur Form grammar). It generates an Abstract Syntax Tree (AST) and reconstructs statements in target notations using manually implemented BNF grammars for each target.
Notation Selection: The study evaluates notations based on four criteria: Verbosity (syntactic variations), Frequency (likelihood of exposure during LM pre-training), Abstractness (complexity from natural to symbolic), and Finiteness (definable grammar).
Supported Notations: The framework supports:
- CL Suite: Common Logic Interchange Format (CLIF) and Conceptual Graph Interchange Format (CGIF).
- Plus-Minus Algebra: Tensor Functor Logic Plus (TFLPLUS).
- Answer Set Programming: CLINGO.
- MINIFOLx: A family of lightweight FOL variants introduced as a baseline to test sensitivity to frequency. These replace standard FOL symbols with more common words or simpler operators (e.g., replacing $\forall$ with "all", $\neg$ with "not").

2. Syllogism Evaluation Framework (SEF)

The authors utilize the SEF categorization method to classify syllogisms into four structural categories: Hypothetical (implication), Disjunctive (disjunction), Categorical (standard 2-premise), and Complex. These categories are automatically detected and used to enrich Zero-Shot (ZS) prompts.

3. Datasets and Experimental Setup

Datasets: The study extends the human-curated FOLIO and P-FOLIO datasets. Using CLGC, the authors created FOLIO-KR and P-FOLIO-KR, which include the original syllogisms translated into the various formal notations and annotated with SEF categories.
Models: Experiments focus on SLMs due to their frugality and relevance to resource-constrained reasoning.
- Supervised Fine-Tuning (SFT): Flan-T5-small and Flan-T5-large.
- Zero-Shot (ZS): Gemma-2-2b-it, Llama-3.2-3b-instruct, and Phi-3.5-mini-instruct.
Scenarios:
- SFT: Models are trained on premises and conclusions in a specific notation to predict truth labels (True, False, Unknown).
- ZS Scenario 1: Models are prompted with the syllogism and the notation's BNF grammar.
- ZS Scenario 2: Models are prompted with the syllogism, grammar, the syllogism's SEF category definition, and an example.

Key Results

Supervised Fine-Tuning (SFT)

Notation Impact: Performance is highly dependent on the interaction between model size, dataset size, and notation abstraction.
- On P-FOLIO-KR (smaller dataset), Flan-T5-small performed best on abstract notations like TFLPLUS and combinations (CLIF + TFLPLUS), outperforming Natural Language significantly. The authors hypothesize that the simplicity of TFLPLUS tokens (e.g., +, -) aids learning in small models with limited training data.
- On FOLIO-KR (larger dataset), Natural Language (NL) and CLIF became the top performers for both model sizes. The increased data volume allowed models to learn complex notations better, but abstract notations like TFLPLUS struggled, potentially due to reliance on less frequent symbolic tokens.
Model Scaling: Generally, performance increased with model size regardless of abstraction. However, FOL and highly abstract notations showed different growth trends compared to NL and CLIF.
Conservative Reasoning: Combining NL with CLIF (NL + CLIF) shifted model behavior toward "conservative" reasoning. This combination improved the prediction of Unknown (inconclusive) cases and reduced hallucinations (incorrect "True" predictions), suggesting that compact notations help models better handle uncertainty and reasoning by refutation.

Zero-Shot (ZS)

Prompting with Categories: Adding SEF category definitions (Scenario 2) generally improved performance for models like Gemma and Phi, particularly on NL-like notations. However, for some models (e.g., Llama) or abstract notations, the added descriptions sometimes decreased performance or had negligible effects.
Model Sensitivity:
- Gemma (text-heavy pre-training) excelled with NL and CLIF.
- Phi (logic/math pre-training) showed better adaptability to abstract notations like CLINGO when provided with category descriptions.
- Llama (distilled web data) showed resilience to symbolic tokens in FOL-like notations.
Inference Speed: KR notations demonstrated faster inference times than NL. TFLPLUS was the fastest due to its compact syntax, while NL was the slowest. CLIF offered a favorable compromise between speed and performance.

Significance and Claims

The paper claims three primary contributions to the field of logical reasoning in AI:

Framework and Tooling: The introduction of CLGC, the first Python library for automatically generating syllogisms in diverse KR notations and defining their SEF categories, enabling reproducible research in this domain.
Data Expansion: The release of FOLIO-KR and P-FOLIO-KR, enriched datasets that allow for the evaluation of LMs across a spectrum of formal notations and logical structures.
Empirical Insights: The establishment of trends showing that:
- The choice of input notation significantly impacts SLM performance, with no single notation being universally superior; the optimal choice depends on model size, training paradigm (SFT vs. ZS), and dataset scale.
- Combinations of notations (e.g., NL + CLIF) can refine reasoning, specifically improving the model's ability to identify uncertainty and perform reasoning by refutation.
- Logical definitions (SEF categories) in prompts can boost reasoning, but their efficacy is contingent on the model's pre-training data and the abstraction level of the notation.

The authors conclude that while SLMs can perform well on reasoning tasks with the right notation, the dichotomy between natural and abstract notations suggests a future direction for neuro-symbolic architectures that combine both in multi-stage pipelines to achieve complex, robust reasoning.