💬 NLP

SEF-CLGC at SemEval-2026 Task 11: Logical Notation Impact on Language Model Performance

This paper presents the SEF-CLGC pipeline, which integrates formal logical notations with Small Language Models to achieve a 27.80% content score on SemEval-2026 Task 11 while significantly reducing content bias in reasoning.

Original authors: Hanna Abi Akl, Fabien Gandon, Catherine Faron, Pierre Monnin

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hanna Abi Akl, Fabien Gandon, Catherine Faron, Pierre Monnin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but tiny, robot how to solve logic puzzles. Usually, when people build these robots, they make them enormous and feed them the entire internet to learn. But this paper is about a different approach: using Small Language Models (SLMs). Think of these as "pocket-sized" robots that are frugal with energy but still capable of complex thinking if trained correctly.

The researchers entered a competition called SemEval-2026 Task 11. The challenge was simple on the surface: look at a logical argument (called a syllogism) and decide if it is True (valid) or False (invalid).

Here is the tricky part: Humans (and big AI models) often get tricked by how the argument sounds. If the conclusion sounds like a real-world fact (e.g., "All cats are mammals"), we might say "True" even if the logic is broken. This is called content bias. The goal was to teach the robot to ignore the "flavor" of the words and focus only on the "recipe" (the logic structure).

The Secret Sauce: Translating the Puzzle

The team used a pipeline they built called SEF-CLGC. Here is how it works, using a cooking analogy:

The Raw Ingredients (Natural Language): The puzzle starts as a sentence in English, like "All cars are vehicles. No animal is a car..."
The Translation (FOL): First, they use a powerful translator (an AI called ChatGPT) to turn that English sentence into First-Order Logic (FOL). This is like translating a recipe from "add a pinch of salt" to a precise chemical formula. It removes all the ambiguity.
The Flavor Variations (Logical Notations): Once they have the chemical formula, they translate it into different "dialects" of logic. Some look like standard math, some look like computer code (CLINGO), and others are custom-made shortcuts.
- Analogy: Imagine you have a song. You can play it on a piano (Natural Language), a synthesizer (FOL), or a drum machine (CLINGO). The song is the same, but the instrument changes how it sounds.
The Training: They fed these different versions of the logic puzzles to their tiny robot models. They wanted to see if teaching the robot to read the "chemical formulas" (logic) instead of just the "English sentences" would help it stop getting tricked by the content.

The Results: Small but Mighty

The researchers tested their models and found some surprising things:

The "Hybrid" Approach Won: The best model wasn't one that only spoke English or one that only spoke pure logic. The winner was a model that saw both the English sentence and the Logic formula together. It was like giving the robot a map and a compass.
Beating the Bias: By training on these logical "dialects," the models became much better at ignoring the real-world facts and focusing on the rules. They reduced the "content bias" significantly.
The Score: Their best model achieved a score of 27.80% on a specific metric designed to measure how well they balanced accuracy with fairness (ignoring bias). While that number might look low, in this specific, bias-heavy test, it was a strong performance for such a small model.
What Didn't Work: Some of the custom, highly abstract logical languages were too confusing for the tiny robots. It's like trying to teach a beginner to read using a language with no vowels; the robot just got lost.

The Takeaway

The paper claims that you don't need a massive, energy-hungry supercomputer to solve complex logic problems. By using Small Language Models and teaching them to speak "logic languages" alongside natural language, you can create a system that is:

Frugal: It uses very little computing power.
Fairer: It is less likely to be tricked by how an argument sounds.
Competitive: It can hold its own against much larger models in logic tasks.

The authors also noted a limitation: they relied on a commercial AI to do the initial translation from English to Logic. If that commercial AI changes its mind or gets updated, it could mess up the whole pipeline, like if the translator suddenly started speaking a different dialect.

In short, they proved that with the right "translation tools," a small, efficient robot can learn to think logically without getting distracted by the story it's reading.

Technical Summary: SEF-CLGC at SemEval-2026 Task 11

Problem Statement
This paper addresses SemEval-2026 Task 11 Subtask 1, which challenges Large Language Models (LMs) to determine the logical validity of syllogisms while disentangling content reasoning from formal reasoning. The core issue is that LMs often exhibit "content bias," where their validity judgments are swayed by the real-world plausibility of the arguments rather than their logical structure. While larger models generally perform better, they remain prone to the same reasoning biases as humans. The authors investigate whether Small Language Models (SLMs), when trained on a combination of natural language and formal logical notations, can achieve competitive reasoning performance while mitigating these biases.

Methodology
The authors utilize and enhance a pipeline called the Syllogistic Evaluation Framework-Common Logic Grammar Construction (SEF-CLGC). The methodology proceeds through the following stages:

Dataset Transformation:
- NL-to-FOL Translation: Natural Language (NL) syllogisms from the task dataset are first translated into First-Order Logic (FOL) using OpenAI's ChatGPT 5.2 model.
- Categorization: The SEF component categorizes syllogisms into four types: Hypothetical, Disjunctive, Categorical, and Complex.
- Notation Generation: Using Backus-Naur Form (BNF) grammars, the system generates Abstract Syntax Trees (ASTs) and transforms the FOL representations into various logical notations. The pipeline introduces two new notations alongside existing ones:
  - CLINGO: A language for Answer Set Programming (ASP).
  - MINIFOL2: A custom notation replacing FOL operators with Boolean equivalents and eliminating the existential quantifier ("∃").
- The resulting dataset includes the original NL syllogisms and their transformations into notations such as FOL, CLIF, CGIF, TFL+, CLINGO, and MINIFOL2.
Model Training and Architecture:
- The authors employ Flan-T5 models (specifically small and large variants), limiting the scope to models with fewer than 1 billion parameters to ensure frugality.
- Two model families are trained:
  - SEMEVAL Models: Vanilla Flan-T5 models fine-tuned directly on the task dataset splits.
  - FOLIO-SEMEVAL Models: Flan-T5 models pre-fine-tuned on the FOLIO dataset (a logical reasoning dataset) before being fine-tuned on the SemEval task data.
- Training involves Supervised Fine-Tuning (SFT) where the input consists of syllogisms in one or multiple notations, and the target is the validity label ("True" or "False").
Evaluation Metrics:
- Performance is measured using Accuracy (ACC) and the Content Effect (CE), which quantifies the performance gap between plausible and implausible syllogisms.
- The primary ranking metric is the Content Score (CS), defined as $CS = \frac{ACC}{1 + \log(1 + CE)}$ . This metric rewards high accuracy while penalizing high content bias.

Key Results

Performance of SLMs: The best-performing model, a FOLIO-SEMEVAL Flan-T5-large trained on a combination of Natural Language and FOL (NL-FOL), achieved a Content Score of 27.80% and an accuracy of 90.57% on the blind evaluation set.
Impact of Notations:
- In the SEMEVAL family (no pre-training on FOLIO), Natural Language (NL) alone performed best, followed by NL-FOL. Pure formal notations (e.g., FOL, CLIF) generally underperformed compared to NL.
- In the FOLIO-SEMEVAL family, pre-training on logical data significantly altered the hierarchy. The NL-CLIF combination achieved the highest training accuracy (0.95), and NL-FOL achieved the highest evaluation Content Score.
- Notations like MINIFOL2 and TFL+ performed poorly, likely due to the models' unfamiliarity with their specific syntactic structures.
Bias Mitigation: The use of formal notations, particularly in combination with NL, significantly reduced the Content Effect (CE). For instance, the NL-FOL model reduced CE to 8.55 compared to 9.57 for the NL-only model in the FOLIO-SEMEVAL family.
Error Analysis: The models demonstrated a strong ability to detect valid syllogisms but struggled with invalid ones, showing a high rate of False Positives (predicting invalid syllogisms as valid). This bias was more pronounced in models pre-trained on the FOLIO dataset.

Significance and Claims
The paper claims that its work demonstrates the viability of Small Language Models as competitive agents for complex logical reasoning tasks when augmented with formal logical notations. Specifically, the authors assert that:

Frugality and Competitiveness: It is possible to achieve high accuracy and competitive rankings (7th in Content Effect, 10th in Accuracy) using SLMs, challenging the trend of relying solely on massive models.
Bias Reduction: Integrating formal logical notations (neuro-symbolic approach) helps reduce content bias, allowing models to reason more about structure than real-world plausibility.
Value of Pre-training: Pre-fine-tuning on logical datasets (like FOLIO) enhances the model's ability to learn and utilize formal notations effectively, even on small-scale tasks.

The authors conclude that while their approach is effective, future work is needed to address the imbalance in syllogism categories within the dataset and to investigate the specific reasons for the models' difficulty with invalid syllogisms. They also acknowledge the limitation of relying on a commercial model for the initial NL-to-FOL translation step.

The Secret Sauce: Translating the Puzzle

The Results: Small but Mighty

The Takeaway

More like this