💬 NLP

HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimization

The HABIB_TAZ system achieved top rankings in SemEval-2026 Task 11 by employing mDeBERTa-v3 models trained on synthetic syllogistic data with multi-objective optimization to effectively disentangle formal logic from content bias across multiple languages.

Original authors: Abdullah Shaikh, Zain Naqi, Taha Zahid, Sandesh Kumar, Abdul Samad

Published 2026-07-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Abdullah Shaikh, Zain Naqi, Taha Zahid, Sandesh Kumar, Abdul Samad

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a super-smart robot how to solve logic puzzles. You give it a rule: "If all birds can fly, and a penguin is a bird, then a penguin can fly." A human might pause and say, "Wait, penguins can't fly! That rule is wrong in the real world." But a robot that relies too much on its "common sense" might get confused. It might think, "Oh, penguins are birds, and birds fly, so the conclusion must be true," even though the logic itself is flawed because the starting rule was false. This is the tricky problem scientists are trying to solve with Large Language Models (LLMs). These are the powerful AI brains behind chatbots and search engines. They are amazing at writing stories and answering questions, but they often struggle to separate what is logically true from what sounds believable. They get tricked by "content effects," where the real-world meaning of the words distracts them from the actual structure of the argument. The goal of this research is to build an AI that acts like a strict logician: one that checks if the math of the argument works, regardless of whether the story sounds silly or sensible.

The team behind this project, HABIB_TAZ, entered a competition called SemEval-2026 Task 11 to test exactly this ability. Their mission was to create a system that could look at syllogisms (a type of logical argument with two premises and a conclusion) in 12 different languages and decide if the conclusion followed the rules, even if the premises were nonsense or if there were extra, distracting sentences thrown in.

To solve this, the researchers didn't just feed their AI more real-world data. Instead, they built a "synthetic" training ground. Imagine they created a giant factory that generates millions of logic puzzles using strict, made-up rules and gibberish words, rather than real animals or objects. This was to stop the AI from learning shortcuts based on real-world knowledge. They then trained their AI model, which is based on a powerful language engine called mDeBERTa-v3, using a special three-part training recipe. First, they used a technique to make sure the model didn't ignore the harder, trickier examples. Second, they added a "bias penalty," which is like a teacher gently scolding the student every time they get distracted by how "plausible" a story sounds, forcing them to focus only on the logic. Third, they used a consistency check to ensure the model understood that a complex sentence meant the same logical thing as a simple one.

The results were a mix of perfect scores and interesting challenges. On the simpler tasks—English-only puzzles, English puzzles with distractions, and puzzles in 12 languages without distractions—their system achieved a perfect 100.0 score. It completely ignored the "real-world" tricks and got the logic right every single time, with zero bias. However, the hardest challenge was the "Noisy Multilingual" task, where the AI had to deal with 12 languages and distracting sentences. Here, the system ranked 6th. It got 89.06% of the answers right, but it still showed a small amount of bias (2.89%).

The researchers found that their special training recipe worked wonders for the easier tasks, proving that you can teach an AI to be a pure logician if you train it on synthetic, rule-based data rather than messy real-world text. For the hardest task, they noticed that the model struggled more with the original multilingual data than with English translations, suggesting that the way the model handles different languages is still a bit shaky. They also discovered that adding a specific "consistency" check (using KL-Divergence) helped the model stay stable across different languages, though it required a lot of computer power to run. Ultimately, the paper suggests that while we can build AIs that are incredibly good at pure logic, making them robust across every language and every type of noise is still a work in progress. The team has made their code and data generation tools public, hoping others will use them to build even better, more logical AI in the future.

Technical Summary: HABIB_TAZ at SemEval-2026 Task 11

Problem Statement
While Large Language Models (LLMs) demonstrate proficiency in general Natural Language Processing (NLP) tasks, their formal reasoning capabilities are frequently compromised by "content effects." Models tend to overestimate the logical validity of arguments that align with real-world plausibility, conflating world knowledge with formal logic. SemEval-2026 Task 11 addresses this by evaluating a model's ability to disentangle formal logic from content across 12 languages, specifically requiring the identification of relevant premises amidst irrelevant noise (distractors). The core challenge lies in achieving high accuracy in logical validity classification while minimizing the Total Content Effect (TCE), a metric quantifying bias toward plausible content.

Methodology
The HABIB_TAZ system employs a multi-faceted approach combining synthetic data generation, specialized architecture, and multi-objective optimization to decouple semantic patterns from logical structure.

Architecture: The system utilizes mDeBERTa-v3 backbones pre-trained on Natural Language Inference (NLI) tasks. For subtasks involving noise (ST2 and ST4), a larger backbone (DeBERTa-v3-large) is employed to better disentangle irrelevant premises. The architecture features two specialized heads: a classification head for logical validity (using the [CLS] token) and a premise selection head using span-based pooling and masked mean pooling to identify relevant premises.
Data Strategy: To overcome the scarcity of the provided training data (N=960), the authors developed a rule-based synthetic data engine. This engine generates data based on 256 Aristotelian syllogistic schemes, ensuring balanced representation across validity and plausibility quadrants. Key strategies include:
- Vocabulary Control: Using WordNet hypernym/hyponym chains to create plausible samples and cross-chain nouns for implausible ones.
- Symbolic Data: Introducing gibberish or symbolic variables to reduce reliance on real-world semantics.
- Complex Rephrasing: Mapping simple logical forms to complex linguistic variations to enforce structural invariance.
- Multilingual Expansion: Translating English structures into 12 target languages using language-specific templates to preserve quantifiers.
- Robustness: Integrating samples from the HANS (Heuristic Analysis for NLI Systems) dataset to prevent models from exploiting surface-level syntactic heuristics.
Optimization: The training pipeline employs a multi-objective loss function ( $L_{total} = L_{DRO} + \lambda_{bias}L_{bias} + \gamma L_{KL}$ $L_{t o t a l} = L_{D R O} + λ_{bia s} L_{bia s} + γ L_{K L}$ ):
- Adaptive Group DRO: Instead of standard Cross-Entropy, Group Distributionally Robust Optimization (DRO) focuses on low-performing subgroups (defined by validity/plausibility quadrants and languages) to ensure consistency.
- Differentiable Bias Penalty: A loss component calculated from sigmoid confidence scores that penalizes uneven confidence distributions across the validity-plausibility quadrants, forcing the model to maintain logical certainty regardless of content plausibility.
- KL-Divergence Consistency: This minimizes the divergence between the output distributions of simple syllogisms (teacher) and their complex rephrasings (student), anchoring the model to fundamental logical forms rather than lexical shortcuts.
- Dynamic Scheduling: The bias penalty weight ( $\lambda_{bias}$ ) and DRO step-size are initialized at zero and incrementally increased to prevent the model from collapsing into sub-optimal minima before mastering formal structures.

Key Results
The system achieved state-of-the-art performance across the first three subtasks and strong results on the most complex subtask:

Subtask 1 (English): Achieved a perfect Ranking Score of 100.0 with 0.00% bias and 100.0% accuracy.
Subtask 2 (Noisy English): Achieved a perfect Ranking Score of 100.0 with 0.00% bias and 100.0% accuracy, successfully disentangling reasoning from noisy premises.
Subtask 3 (Multilingual): Achieved a perfect Ranking Score of 100.0 with 0.00% bias and 100.0% accuracy. Ablation studies indicated that the combination of Bias loss and HANS data was sufficient for this task, while additional regularizers (KL/MSE) did not improve performance.
Subtask 4 (Noisy Multilingual): Ranked 6th with an Accuracy and F1-score of 89.06%, a bias of 2.89%, and a Ranking Score of 37.78.
- Analysis: The authors noted a significant performance gap when evaluating on the original multilingual test set versus an English translation of the test set. The KL-Divergence regularization was identified as the critical factor in closing this cross-lingual gap, making the model robust to linguistic variations. However, due to computational constraints during the competition, the full dataset (648,632 samples) was reduced to approximately 4% for training, suggesting potential for further improvement.

Significance and Contributions
The paper claims that its primary contribution is demonstrating that bias-aware optimization combined with synthetic, rule-based data can effectively isolate formal reasoning from real-world plausibility. By avoiding the semantic noise inherent in LLM-augmented data and employing a multi-objective loss function, the system successfully mitigates content effects that typically plague LLMs.

The authors highlight that:

Synthetic Data is Crucial: Training on abstract symbols and rule-based structures allows models to generalize logical reasoning better than training solely on natural language data.
Multi-Objective Optimization is Necessary: Isolated regularization techniques (like DRO or Bias alone) can degrade performance or fail to generalize; a balanced combination (specifically KL-Divergence + DRO + Bias) is required to maintain structural invariance while reducing bias.
Scalability: The capacity to abstract logical relationships from noisy premises scales directly with model parameters, as evidenced by the perfect scores achieved with the Large backbone on Subtask 2.

The authors make their dataset generation engine and codebase publicly available to facilitate future research into robust logical reasoning. They conclude that while the system achieved perfect scores on clean and single-language noisy tasks, the challenge of fully robust multilingual logical reasoning remains an active area for improvement, particularly regarding the computational costs of consistency regularization.

Technical Summary: HABIB_TAZ at SemEval-2026 Task 11

More like this