UFAL-CUNI at SemEval-2026 Task 11: An Efficient Modular Neuro-symbolic Method for Syllogistic Reasoning
The UFAL-CUNI team presents an efficient modular neuro-symbolic system for SemEval-2026 Task 11 that combines a small 4B-parameter LLM parser with a symbolic theorem prover to achieve competitive accuracy in syllogistic reasoning while outperforming zero-shot baselines, though it highlights limitations in the multilingual capabilities of smaller models.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to teach a very smart, but slightly distracted, student (a Large Language Model) how to solve logic puzzles called syllogisms. These are puzzles like:
- Premise 1: All cats are animals.
- Premise 2: Some animals are fluffy.
- Conclusion: Therefore, some cats are fluffy.
The problem is that this student has a bad habit: they let their real-world knowledge get in the way. If the puzzle says, "All cats are fluffy," the student might say "True" just because they know cats are fluffy, even if the logic of the puzzle doesn't actually support that conclusion. This is called the "content effect." The student is biased by what they think is true, rather than what the rules of the puzzle say.
The authors of this paper built a special "tutoring system" to fix this. Here is how their system works, explained simply:
1. The Translator (The Bilingual Interpreter)
First, if the puzzle is in a foreign language (like Portuguese or Russian), the system uses a large language model to translate it into English. Think of this as a translator ensuring everyone is speaking the same language before the logic game begins.
2. The Notation Expert (The LaTeX Writer)
This is the system's clever trick. Instead of asking the student to solve the puzzle directly, the system asks them to rewrite the sentences into a specific "code" called First-Order Logic (FOL), written in a format called LaTeX (which looks like mathematical formulas).
- Why LaTeX? The authors realized that asking the student to write directly in the computer's "native language" (Prover9 syntax) was like asking a human to speak in binary code. It caused too many mistakes.
- The Analogy: It's like asking a student to write a math problem on a whiteboard using standard symbols (, , ) first, rather than trying to type it directly into a calculator that only understands a weird, specific code. The student is much better at writing the "whiteboard" version because they've seen it in their training data.
3. The Translator (The Code Converter)
Once the student writes the logic in the "whiteboard" (LaTeX) format, a simple computer script (a "transpiler") acts like a strict editor. It instantly converts that clean LaTeX code into the specific, rigid code the computer needs to run the proof (Prover9 syntax). This step is purely mechanical and doesn't involve the "distracted" student, so it rarely makes mistakes.
4. The Judge (The Automated Prover)
Finally, the system hands the rigid code to a Theorem Prover (a piece of software called Prover9). This is a robot judge that has zero emotions and zero real-world knowledge. It doesn't care if cats are fluffy or if the moon is made of cheese. It only checks: Does the conclusion mathematically follow from the premises? If the math works, it says "Valid." If not, "Invalid."
5. The Detective (Finding the Important Clues)
For harder puzzles where there are extra, useless sentences mixed in, the system uses a "greedy algorithm." It acts like a detective who tries removing one clue at a time. If the puzzle still makes sense without a specific clue, that clue is irrelevant. If the puzzle falls apart, that clue was necessary. This ensures the system only focuses on the facts that actually matter.
What Did They Find?
- Small is Beautiful: They used a relatively small AI model (4 billion parameters) for the "student" part. Even though small models usually struggle with complex logic, this system made them very good at it by offloading the actual reasoning to the robot judge.
- Beating the Bias: By forcing the AI to translate to logic first and then letting a robot judge the result, they successfully stopped the AI from being biased by real-world facts. The "content effect" dropped significantly.
- The Metric Problem: The paper also points out a flaw in how the competition was scored. The scoring system was so sensitive that even a tiny, random mistake could tank a team's score, making it hard to tell if a system was truly "good" or just "lucky."
The Bottom Line
The paper shows that you don't need a giant, super-intelligent AI to solve logic puzzles. Instead, you can use a small AI as a translator to turn human language into math, and then let a dumb but perfect robot do the actual thinking. This combination stops the AI from getting distracted by what it "knows" about the world and forces it to stick strictly to the rules of logic.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.