🤖 AI

Distilling Answer-Set Programming Rules from LLMs for Neurosymbolic Visual Question Answering

This paper proposes a neurosymbolic approach for Visual Question Answering that distills Answer-Set Programming rules from Large Language Models by leveraging few-shot examples and solver feedback to automatically extend reasoning theories, offering a scalable and interpretable alternative to traditional data-driven rule learning.

Original authors: Thomas Eiter, Nelson Higuera Ruiz, Johannes Oetsch

Published 2026-06-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Thomas Eiter, Nelson Higuera Ruiz, Johannes Oetsch

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Robot to Think with Logic

Imagine you have a very smart robot assistant that can look at a picture and answer questions about it (like "Is the cat wearing a hat?"). This is called Visual Question Answering (VQA).

Usually, these robots are trained like athletes: they practice millions of times until they get the answer right by guessing patterns. But this makes them a bit like a "black box"—you don't know why they got the answer right, and if you ask a slightly new type of question, they might get confused.

The authors of this paper want to build a robot that thinks more like a human logician. They use a special "rulebook" called Answer-Set Programming (ASP). Think of this rulebook as a set of strict, logical instructions (like a recipe) that tells the robot exactly how to deduce the answer.

The Problem: Writing these rulebooks by hand is hard. If you want the robot to learn a new trick (like counting objects in a specific way), a human developer has to sit down and write new rules from scratch. It's slow and tedious.

The Solution: The authors asked a super-smart AI (a Large Language Model, or LLM) to write these rules for them. They call this process "Distilling Rules."

How It Works: The "Tutor and Student" Analogy

Imagine the LLM is a brilliant but slightly scatterbrained student who knows a lot about logic but hasn't seen this specific puzzle before. The "Teacher" (the computer system) wants the student to write the correct rulebook.

Here is the step-by-step process they used:

The Setup: The teacher gives the student a "broken" rulebook. It works for most things, but it's missing a specific rule (e.g., how to find the shortest path on a map).
The Example: The teacher shows the student a single example: "Here is a picture of a map, here is the question, and here is the correct answer."
The Attempt: The student (the LLM) tries to write a new rule to fix the broken book.
- The Catch: The student sometimes makes mistakes. They might write a rule with bad grammar (syntax error) or a rule that looks right but gives the wrong answer (logic error).
The Correction Loop (The "Mending"):
- Syntax Check: The computer checks if the rule is written in the correct language. If the student used a symbol that doesn't exist in the rulebook language, the computer says, "Hey, fix your spelling," and the student tries again.
- Logic Check: The computer runs the new rule. If the answer is wrong, the computer says, "You got the wrong answer. Try again."
- The "Safety Net" (Regression Testing): Once the student writes a rule that works for the new example, the computer checks if that rule breaks any of the old examples. It's like making sure a new ingredient in a cake doesn't ruin the flavor of the previous cakes you baked.
The Result: Once the rule passes all checks, it is added to the master rulebook.

The "Magic" of Few Examples

One of the coolest findings in the paper is that the student didn't need to study thousands of examples. Just a handful (sometimes as few as 1 or 2) was enough for the smartest AI models to figure out the logic and write the perfect rule.

It's like showing a genius chef one picture of a perfect soufflé and asking them to write the recipe. They don't need to taste 1,000 soufflés; they just need to understand the logic of the ingredients.

The Contenders: Who Did the Best?

The researchers tested several different "students" (AI models) to see who could write the best rules:

The Top Performers (GPT-4o, DeepSeek, Gemini-3): These were like the valedictorians of the class. They wrote perfect rules almost every time, even for very tricky puzzles involving maps and graphs.
- Gemini-3 was particularly special. It seemed to have a "thinking mode" where it paused to double-check its own logic before writing the rule, resulting in very few mistakes and very short, clean rulebooks.
The Struggling Students (LLaMA3, Mistral): These models were okay at simple tasks but got confused by complex logic. Sometimes, giving them too many hints (complex prompts) actually made them perform worse, as if they got overwhelmed by the instructions.

The "Pruning" (Cleaning Up the Mess)

Sometimes, the AI would write a rulebook that worked but was messy. It might include five different ways to say the same thing, or rules that weren't actually needed.

The researchers created a "gardener" tool (a pruning heuristic) that went through the rulebook and cut out the dead branches (redundant rules).

Analogy: Imagine the AI wrote a sentence like: "If it rains, bring an umbrella. Also, if it rains, bring an umbrella. Also, if it rains, bring an umbrella." The pruner deletes the extra copies, leaving just one clear instruction.
Result: This made the rulebooks smaller and easier for humans to read, without changing the robot's ability to answer questions correctly.

What This Means (According to the Paper)

The paper concludes that we don't always need to hand-code complex logic for AI systems anymore. We can use a smart AI to "distill" the logic for us, using just a few examples as a guide.

It works best when the task is clear and the AI model is very smart.
It is different from traditional machine learning because it doesn't "memorize" data; it learns the logic of the problem.
It creates transparency: Because the output is a set of logical rules (like a recipe), humans can actually read and understand why the robot made a decision.

In short: The authors built a system where a human provides a broken logic puzzle and a few examples, and a smart AI writes the missing pieces of the puzzle, checks its own work, and hands back a clean, working rulebook.

Technical Summary: Distilling Answer-Set Programming Rules from LLMs for Neurosymbolic Visual Question Answering

Problem Statement
Visual Question Answering (VQA) requires integrating multimodal inputs (vision and text) with complex reasoning chains. While modular neurosymbolic approaches offer interpretability by separating perception (neural) from reasoning (symbolic), they face a significant maintenance burden. Specifically, when task requirements change—such as the introduction of new question types or reasoning operators—the symbolic reasoning component, often implemented in Answer-Set Programming (ASP), must be manually extended or adapted. Traditional data-driven rule learning methods (e.g., Inductive Logic Programming) typically require large datasets and hard-coded search spaces (mode declarations) to generalize rules. This paper addresses the challenge of automating the extension of ASP-based reasoning theories in VQA systems without extensive retraining or manual rule engineering.

Methodology
The authors propose a declarative knowledge distillation framework where Large Language Models (LLMs) are prompted to generate or extend ASP rules based on few-shot examples. The system operates within a modular neurosymbolic VQA pipeline consisting of:

Vision Module: Converts images into symbolic scene graphs (ASP facts).
Language Module: Converts natural language questions into functional programs (ASP facts).
Reasoning Module: An ASP solver derives answers from the combined facts and an ASP theory (a set of rules).

The core algorithm (Algorithm 1) iterates through a distillation suite of VQA examples $(s, q, a)$ . If the current ASP theory $T$ fails to derive the correct answer $a$ for a given scene $s$ and question $q$ , the system prompts an LLM to generate candidate rules to fix the gap. The process employs a feedback loop involving:

Prompting Strategies:
- Multi-prompting: Requests multiple solution variants in a single call.
- Chain-of-Thought (CoT): Instructs the LLM to break down the problem into intermediate logical steps before generating rules (simple and complex versions).
Validation and Mending:
- Syntax Mending: If the LLM generates syntactically invalid ASP (e.g., hallucinated mathematical symbols), the solver's error message is fed back to the LLM for correction.
- Semantic Mending: If the rules are syntactically correct but produce the wrong answer, the solver's incorrect output is compared against the ground truth and fed back to the LLM for revision.
- Regression Testing: New rules are tested against previously solved examples to ensure consistency and prevent regression.
Redundancy Elimination: A heuristic (Algorithm 2) prunes rules that do not affect the answer set, ensuring the distilled theory remains compact.

Key Contributions

Declarative Knowledge Distillation: A method to automatically extend ASP theories in VQA systems by prompting LLMs with examples, rather than using them as training data for statistical learning.
Prompting Strategies: The design and evaluation of specific prompting techniques (multi-prompting, CoT) and repair mechanisms (syntax/semantic mending) tailored for generating logic programs.
Compositional Rule Synthesis: Demonstration that LLMs can compose new, higher-arity predicates (e.g., in_between, connected) from existing primitive operations within the ASP theory.
Empirical Evaluation: A comprehensive evaluation across three diverse datasets:
- GQA: Real-world scenes with complex object relations.
- CLEVR: Synthetic geometric scenes with compositional questions.
- CLEGRV: Graph-based transit networks requiring recursive reasoning (e.g., shortest paths).

Results
The experiments evaluated several LLMs (GPT-4o, DeepSeek, Mistral-Large, LLaMA3-70B, Gemini-3 Pro) on restoring removed predicates from the ASP theories.

Performance: Large-scale models (GPT-4o, DeepSeek, Gemini-3) achieved near-perfect accuracy (often >98%) on GQA and CLEVR with very few examples (typically 1–4 per distillation attempt).
Dataset Complexity: Performance dropped on the more complex CLEGRV dataset (involving recursion), though Gemini-3 Pro maintained high accuracy (100% on several predicates), attributed to its "high reasoning" mode.
Model Sensitivity: Smaller or less capable models (LLaMA3-70B, Mistral-Large) showed high variance and struggled with complex tasks. Interestingly, for these models, disabling complex prompting strategies (like CoT) sometimes improved performance, suggesting they can be "overburdened" by verbose instructions.
Efficiency: Gemini-3 Pro was the most efficient, requiring the fewest examples and tokens, and producing the least redundant rule sets.
Ablation: The full prompting pipeline (mending + CoT + multi-prompting) significantly boosted performance for top-tier models but was not always necessary for weaker models.

Significance and Claims
The paper claims that rule distillation from LLMs is a promising alternative to traditional data-driven rule learning (ILP/SRL). Unlike ILP, which requires explicit search space definitions and large datasets, this approach leverages the implicit knowledge of LLMs, requiring only a few examples to elicit correct, generalizable rules.

Interpretability: The method preserves the transparency of neurosymbolic systems by producing human-readable logic rules rather than opaque neural weights.
Adaptability: It offers a viable mechanism for practitioners to extend reasoning modules dynamically as task requirements evolve, acting as an "interactive copilot" for logic programming.
Limitations: The authors note that the method assumes decomposable, compositional question semantics and relies on empirical validation (regression testing) rather than formal proofs of correctness. The approach is currently most effective for large, capable LLMs.

The work suggests that while LLMs may not replace formal reasoning engines, they can effectively serve as a bridge to automate the maintenance and extension of symbolic reasoning components in complex AI systems.