💬 NLP

FregeLogic at SemEval 2026 Task 11: A Hybrid Neuro-Symbolic Architecture for Content-Robust Syllogistic Validity Prediction

The paper presents FregeLogic, a hybrid neuro-symbolic system for SemEval-2026 Task 11 that combines an ensemble of LLMs with a Z3 SMT solver to resolve content-biased errors in syllogistic validity prediction, achieving a 94.3% accuracy and significantly reducing the content effect metric through targeted formal verification on disputed cases.

Original authors: Adewale Akinfaderin, Nafi Diallo

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Adewale Akinfaderin, Nafi Diallo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge in a logic court. Your job is to decide if a specific argument (a syllogism) is logically valid.

Here is the tricky part: Humans (and the AI models we build) are terrible at this when the argument sounds "believable" in real life.

The Trap: If an argument says, "All cats have whiskers. My pet has whiskers. Therefore, my pet is a cat," it sounds plausible, but it's logically invalid.
The Trap 2: If an argument says, "All unicorns are purple. My pet is purple. Therefore, my pet is a unicorn," it sounds silly, but the logic is actually the same structure.

Most AI models get confused by the "silly" or "believable" parts. They let their "real-world knowledge" interfere with their "logic skills." This is called the Content Effect.

This paper introduces FregeLogic, a clever system designed to beat this trap. Here is how it works, using a simple analogy.

The Team of Five Detectives (The LLM Ensemble)

First, the system hires five different AI detectives (Large Language Models).

They are all smart, but they have different personalities and ways of thinking.
They are asked to vote on whether an argument is "Valid" or "Invalid."
Usually, they all agree. If 4 or 5 of them say "Valid," the system accepts it.

The "Disagreement" Alarm

Here is the genius part: The system watches for close votes.

If the vote is 5-0 or 4-1, the detectives are confident. The system trusts them.
But if the vote is 3-2, the system hits the alarm bell.

Why? The authors realized that when the AI detectives disagree, it usually means the argument is tricky. The "believable" content has confused some detectives, while others stuck to the logic. This 3-2 split is a signal that the AI is struggling with the "Content Effect."

The Strict Math Professor (The Z3 Solver)

When the detectives disagree (the 3-2 split), the system calls in a Strict Math Professor (a formal logic solver called Z3).

The Detectives are great at understanding language, but they get distracted by the story.
The Professor doesn't care about the story at all. He strips away the words about "cats," "unicorns," or "glass." He only looks at the skeleton of the argument.
- Detective: "Wait, unicorns aren't real! This is invalid!"
- Professor: "I don't care if unicorns exist. Does the math hold up? Yes. This is Valid."

The Professor acts as a tiebreaker. If the detectives are fighting, the Professor makes the final call based purely on math.

The Results: Why It Worked

The team tested this on 960 different logic puzzles.

The Pure Detective Team: Got about 93.4% right, but they were still easily tricked by the "believable" stories.
The Hybrid Team (FregeLogic): Got 94.3% right.
- More importantly, they were much less biased. They stopped getting tricked by the "silly" stories almost 16% more often than the detectives alone.

The Secret Sauce: The "Structured" Translator

There was one big problem: The Professor (Z3) is a robot who only speaks "Math." The Detectives speak "English." To get the Professor to understand the argument, the Detectives had to translate the English sentence into a strict JSON code.

Old Way: The Detectives tried to write the code in free-form text. They made mistakes 22% of the time, and the Professor couldn't read it.
New Way: The system forced the Detectives to use a structured template (like filling out a strict form). This reduced translation errors to nearly zero.

The Bottom Line

FregeLogic is like a team that knows when to trust its gut (the AI Detectives) and when to call in a strict rulebook (the Math Professor).

By only using the strict rulebook when the team is confused, they managed to solve the hardest logic puzzles without getting distracted by the story. It's a perfect example of combining human-like intuition with rigid mathematical precision to get the best of both worlds.

1. Problem Statement

The paper addresses SemEval-2026 Task 11 (Subtask 1), which focuses on predicting the validity of syllogisms (deductive reasoning). The core challenge is content effects: the tendency of Large Language Models (LLMs) to let real-world believability (plausibility) interfere with logical judgment.

Goal: Predict whether a syllogism is logically valid or invalid, regardless of whether the content is believable or unbelievable.
Metric: The task uses a combined score that rewards high accuracy while penalizing content bias (Content Effect or CE). The formula is $Score = \frac{Accuracy}{1 + \ln(1 + CE)}$ .
Dataset: 960 syllogisms balanced across four subgroups (Valid/Invalid $\times$ Believable/Unbelievable).

2. Methodology: FregeLogic Architecture

The authors propose FregeLogic, a hybrid neuro-symbolic system that combines the pattern recognition of LLM ensembles with the rigorous, content-neutral verification of a formal logic solver (Z3). The system operates in three stages:

A. LLM Ensemble (The "Neuro" Component)

Composition: An ensemble of five classifiers formed by pairing three open-weight models (Llama 4 Maverick, Llama 4 Scout, and Qwen3-32B) with varied prompting strategies.
Prompting Strategies: Includes Zero-shot, Few-shot, Few-shot Chain-of-Thought (CoT), and Simple CoT.
Selection: For each cross-validation fold, the top 5 configurations (based on an inner validation set) are selected to maximize diversity and minimize correlated errors.
Output: Each classifier produces a binary vote (Valid/Invalid). The ensemble majority vote is the default prediction.

B. Z3 Formal Verification (The "Symbolic" Component)

Role: Serves as a "tiebreaker" and content-neutral verifier.
Pipeline:
1. Structure Extraction: An LLM extracts the logical structure (terms, premises, conclusion, proposition types A/E/I/O) into a JSON object using Structured Output APIs. This reduced extraction failure rates from ~22% to near zero.
2. FOL Encoding: The structure is encoded into First-Order Logic (FOL) using the Z3 SMT solver. The authors adopted an Aristotelian interpretation with existential import (adding $\exists x: S(x)$ axioms) based on dataset analysis.
3. Satisfiability Check: Validity is determined by checking if the premises are consistent and if the negation of the conclusion leads to a contradiction (UNSAT).
Constraint: Z3 is only invoked when the LLM ensemble is uncertain.

C. Tiebreaker Decision Module

Mechanism: The system calculates the vote margin ( $m$ $m$ ) of the 5-member ensemble.
- If $m > 1$ (e.g., 4-1 or 5-0 splits): The ensemble majority vote is used.
- If $m \le 1$ (specifically 3-2 splits): The system defers to the Z3 verdict (if available).
Hypothesis: Close votes (3-2 splits) indicate cases where content bias is causing disagreement among LLMs. These are the precise cases where a content-neutral formal solver can correct errors.

3. Key Contributions

Hybrid Neuro-Symbolic Integration: Demonstrates that selectively applying formal verification only during low-consensus ensemble states improves overall performance more than using either method alone or applying formal verification universally.
Structured Extraction for Reliability: The use of structured-output APIs for logical structure extraction drastically reduced failure rates, making the symbolic pipeline robust enough for real-world application.
Aristotelian Encoding Validation: The paper empirically validates the need for existential import axioms in the FOL encoding to match the dataset's ground truth (specifically for Felapton-type syllogisms).
Content Effect Reduction: The approach successfully decouples logical reasoning from semantic believability, addressing a known weakness in LLM reasoning circuits.

4. Results

Evaluated via nested 5-fold cross-validation on the 960-sample dataset:

Performance Metrics:
- Pure Ensemble: 93.4% Accuracy, CE 3.39, Combined Score 39.12.
- FregeLogic (Tiebreaker): 94.3% Accuracy, CE 2.85, Combined Score 41.88.
Improvements:
- +2.76 points in the combined score.
- 16% reduction in Content Effect (3.39 $\to$ 2.85).
- 0.9% increase in accuracy.
Subgroup Analysis: The tiebreaker strategy significantly narrowed the accuracy gap between "Valid" and "Invalid" subgroups, particularly improving performance on Invalid-Believable cases (from 90.2% to 94.5%), where content bias is strongest.
Failure Analysis:
- Z3 only intervenes in 7.9% of cases (3-2 splits).
- Of the 30 times Z3 overrode the ensemble, 19 were correct flips and 11 were wrong flips (net gain of 8).
- Error Source: The 11 wrong flips were caused by extraction errors (LLM misidentifying proposition types or term boundaries), not logical encoding errors. This confirms the bottleneck is the extraction step, not the solver.

5. Significance

Targeted Intervention: The paper proves that formal methods do not need to replace LLMs entirely. Instead, they are most effective when used as a "safety net" specifically for cases where the LLM ensemble is confused by content bias.
Scalability: The system requires no parameter updates (training) and relies on inference-time strategies (prompting and API calls), making it adaptable to new models.
Robustness: By reducing the content effect, FregeLogic moves closer to evaluating pure reasoning capabilities rather than a model's ability to recall world knowledge.
Limitations & Future Work: The system relies on accurate extraction; if the LLM fails to parse the logic correctly, the formal solver cannot help. Future work involves exploring adaptive thresholds for the vote margin and generalizing this architecture to other reasoning tasks.

In summary, FregeLogic achieves state-of-the-art results for SemEval-2026 Task 11 by intelligently fusing the flexibility of LLMs with the rigor of formal logic, specifically targeting the "weak spots" where human-like biases interfere with machine reasoning.