💬 NLP

ITLC at SemEval-2026 Task 11: Normalization and Deterministic Parsing for Formal Reasoning in LLMs

The ITLC team proposes a novel method for SemEval-2026 Task 11 that mitigates content biases in multilingual reasoning by transforming syllogisms into canonical logical forms and applying deterministic parsing, achieving top-5 rankings across all subtasks without relying on complex fine-tuning.

Original authors: Wicaksono Leksono Muhamad, Joanito Agili Lopo, Tack Hwa Wong, Muhammad Ravi Shulthan Habibi, Samuel Cahyawijaya

Published 2026-03-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wicaksono Leksono Muhamad, Joanito Agili Lopo, Tack Hwa Wong, Muhammad Ravi Shulthan Habibi, Samuel Cahyawijaya

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly distracted, student (a Large Language Model or LLM) how to solve logic puzzles. The student is brilliant at reading stories and knows a lot about the world, but when it comes to pure logic, it keeps getting tricked by the story itself rather than the rules.

This paper is about a team that built a "logic translator" to fix this problem. Here is the breakdown of their solution using simple analogies.

The Problem: The "Story Trap"

Imagine you give your student this puzzle:

Premise 1: All cats are mammals.
Premise 2: No dogs are cats.
Conclusion: Therefore, no dogs are mammals.

Logically, this is wrong. But because the student knows that dogs are mammals in real life, it gets confused. It thinks, "Wait, I know dogs are mammals, so the conclusion must be true!" It's letting its real-world knowledge (the "content") override the strict rules of the puzzle (the "logic").

This happens even more when the puzzle is in a different language, like Spanish or Swahili, where the student might get lost in translation nuances.

The Solution: The "Logic Translator"

The team (ITLC) didn't try to retrain the student's brain (which is hard and expensive). Instead, they built a translator that sits between the puzzle and the student.

Their method has three main steps:

1. The "Alphabet Soup" Normalization (Stripping the Flavor)

Think of the puzzle as a delicious, complex soup with carrots, potatoes, and spices. The student gets distracted by the carrots (the specific words).
The team's first step is to take that soup and turn it into Alphabet Soup.

Instead of "Cats," "Dogs," and "Mammals," they replace them with A, B, and C.
Instead of "All" or "No," they use standard symbols like A (All) or E (No).

Now the puzzle looks like this:

All A are B.
No C are A.
Therefore, No C are B.

Suddenly, the student can't get distracted by "dogs" or "cats." It has to look at the structure. It's like removing the color from a shape-sorting toy so the child has to focus on the shape, not the red or blue paint.

2. The "English Bridge" (For Foreign Languages)

When the puzzle is in a foreign language (like French or Russian), the student might struggle to understand the "flavor" of the words.
The team uses a clever trick: They translate only the grammar (the "All," "No," "Some") into English, but they keep the nouns in the original language.

Original (French): "Tous les chiens sont des mammifères."
Translated Logic: "All chiens are mammifères."

This acts like a bridge. It lets the student understand the rules of the sentence (which are universal) without getting confused by the specific vocabulary of that language. It's like giving a tourist a map where the street names are in their native language, but the "Turn Left" and "Go Straight" signs are in English.

3. The "Deterministic Check" (The Rigid Rulebook)

Once the puzzle is translated into "Alphabet Soup" with the English bridge, the team doesn't ask the student to "guess" if it's right. Instead, they use a rigid rulebook (a computer program).

The program checks: "Does this pattern of A, B, and C match one of the 24 valid logical forms?"
If yes -> Valid.
If no -> Invalid.

This is like a vending machine. You put in a specific coin (the logic pattern), and the machine either gives you a snack (Valid) or nothing (Invalid). There is no guessing, no "feeling," and no bias.

The Results: Why It Matters

The team tested this on a global competition (SemEval-2026) with puzzles in English and many other languages.

The Old Way (LLM-only): The student tried to solve it using its brain. It got the answer right most of the time, but it was often biased by the story content. It was like a chef who cooks great food but sometimes adds salt just because they like salt, even if the recipe says no.
The New Way (ITLC): The student used the translator and the rulebook.
- Accuracy: They got near-perfect scores.
- Bias: They almost completely eliminated the "story trap." The student stopped guessing based on real-world facts and started following the rules.
- Multilingual: It worked just as well in Spanish, Swahili, and Russian as it did in English.

The Big Takeaway

The paper proves that you don't need to build a super-complex, expensive AI to fix reasoning errors. Sometimes, you just need to simplify the input and use a strict rulebook.

By turning messy, colorful, real-world sentences into clean, abstract "Alphabet Soup," they forced the AI to stop daydreaming about the content and start doing the math. It's a reminder that sometimes, the best way to think clearly is to stop thinking about the words and start thinking about the structure.

1. Problem Statement

Large Language Models (LLMs) struggle with content-independent reasoning, particularly in multilingual contexts. While LLMs can perform logical tasks, they often exhibit content effects: they rely on real-world knowledge, semantic plausibility, or surface-level cues rather than strict logical form. This leads to:

Bias: Accepting conclusions that are semantically true but logically invalid (plausibility bias).
Multilingual Fragility: Performance degradation when reasoning tasks involve non-English languages due to variations in quantifier expression and term realization.
Lack of Robustness: Current mitigation strategies often require complex architectural changes, heavy fine-tuning, or latent-level interventions (e.g., activation steering), which are computationally expensive and less interpretable.

The goal of SemEval-2026 Task 11 is to disentangle content from formal reasoning in syllogistic arguments across English and multiple other languages, measuring both validity accuracy and the magnitude of content effects.

2. Methodology

The authors propose a novel, unbiased method based on Explicit Structural Abstraction and Deterministic Parsing. The pipeline consists of three main stages:

A. Normalization (Structural Abstraction)

The system transforms natural language syllogisms into a canonical categorical form to strip away semantic noise.

Term Identification: The model identifies exactly three distinct semantic categories: the Subject (S), Predicate (P), and Middle Term (M).
Symbolic Mapping: These terms are mapped to symbolic constants ( $A, B, C$ ) based on their order of appearance.
English Pivot Normalization (EPN): For non-English inputs, the system does not perform free-form translation. Instead, it uses an LLM to:
- Translate only quantifiers (All, No, Some) and copulas (are) into English.
- Preserve the original subject and predicate terms in the source language.
- This ensures structural standardization without introducing "lexical drift" that could alter term identity or merge distinct concepts (e.g., preventing "dog" and "canine" from collapsing into a single term if they are distinct in the source logic).

B. Deterministic Parsing

Once normalized, the argument is converted into a structured string (e.g., P1. P2. Conclusion).

Regex Matching: A deterministic parser uses regular expressions to match sentences against four categorical types: A (All), E (No), I (Some), and O (Some not).
Structure Extraction: The parser extracts the form ( $f$ ), subject ( $s$ ), and predicate ( $p$ ) for each sentence, producing a tuple: $\langle(f_1, s_1, p_1), (f_2, s_2, p_2), (f_3, s_3, p_3)\rangle$ .
Mood and Figure Calculation:
- Mood: The ordered triple of proposition types (e.g., AAA, EIO).
- Figure: Determined by the syntactic position of the Middle Term ( $M$ ) in the premises.
Trivial Validity Check: The system also detects degenerate cases (e.g., petitio principii where the conclusion restates a premise) to avoid misclassification.

C. Formal Validation

Rule-Based Lookup: Validity is determined by checking the derived (Mood, Figure) pair against a predefined lookup table of the 24 valid syllogistic forms.
Relevance Premise Identification: For valid arguments, the system identifies the minimal set of premises (usually the two containing the Major and Minor terms) required to derive the conclusion. Invalid arguments are assigned an empty set.

3. Key Contributions

Content-Effect Mitigation: The method effectively decouples reasoning from world knowledge by forcing the model to operate on abstract logical structures rather than semantic content.
Multilingual Robustness via EPN: The "English Pivot Normalization" strategy allows the system to handle diverse languages without losing logical fidelity, addressing the specific challenge of cross-lingual quantifier variation.
Deterministic vs. Probabilistic: Unlike standard LLM inference which is probabilistic and prone to hallucination, this approach uses deterministic parsing (temperature=0, fixed seed) to guarantee correctness by construction once the structure is correctly identified.
Interpretability: The system provides a transparent decision procedure (Mood-Figure lookup) rather than a "black box" neural prediction.

4. Results

The method was evaluated on the SemEval-2026 Task 11 benchmark across four subtasks (English/Multilingual Validity and Relevance Premises).

Ranking: The team (ITLC) achieved Top-5 rankings across all 4 subtasks.
English Validity:
- Achieved 100% accuracy with 0.0 content effect bias.
- Outperformed the LLM-only baseline (98.43% accuracy, 2.13 bias) by eliminating false positives caused by semantic plausibility.
Multilingual Validity:
- With EPN, achieved 100% accuracy and 0.0 bias.
- Without EPN (Norm + Parsing only), accuracy dropped to 96.88% with higher bias, highlighting the necessity of the translation step for handling quantifier variations.
Relevance Premises:
- The structured approach significantly reduced bias compared to LLM-only baselines.
- In Multilingual settings, EPN+Norm+Parsing achieved a 37.88 combined score, outperforming the LLM-only baseline (28.05).
Bias Reduction: The method consistently reduced content-effect bias by 50–100% compared to direct LLM inference across all tasks.

5. Significance

Alternative to Fine-Tuning: This work demonstrates that simple structural abstraction combined with deterministic rules is a highly competitive alternative to complex fine-tuning or activation-level interventions for reasoning tasks.
Scalability: The approach is language-agnostic regarding the logical engine; it only requires a normalization step to map any language to the canonical form, making it scalable for low-resource languages.
Reliability: By removing the probabilistic nature of reasoning steps, the system offers a reliable, interpretable framework for applications where logical consistency is critical (e.g., legal reasoning, scientific deduction).
Insight into LLM Limitations: The results confirm that LLMs' reasoning failures in multilingual settings are often due to quantifier interpretation and term identity issues rather than a lack of logical capability, and these can be solved via preprocessing.

In conclusion, the paper presents a robust, interpretable pipeline that successfully mitigates content bias in LLMs, proving that formalizing the input structure is often more effective than trying to "teach" the model to reason better through training.