💬 NLP

NeSy-RAG: Neuro-Symbolic RAG for Explainable Question Answering

NeSy-RAG is a modular neuro-symbolic retrieval-augmented generation framework that synthesizes attributable Prolog modules from retrieved text to provide transparent, deterministic answers and automatically detect missing user context, achieving superior accuracy over standard RAG baselines on the ShARC benchmark.

Original authors: Jonas Gann, Michael Gertz

Published 2026-08-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jonas Gann, Michael Gertz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery, but instead of a detective, you have a super-smart, incredibly chatty robot assistant. This robot has read almost every book in the library and can tell you the answer to almost any question. This is what we call a "Large Language Model" (LLM). However, there's a catch: sometimes this robot makes things up, or "hallucinates," and when it explains its thinking, it sounds like a foggy dream rather than a clear map. You can't always tell why it gave you an answer or if it actually read the right page in the library. This is a big problem in serious fields like law or medicine, where you need to know exactly where the information came from and be sure it's true. To fix this, scientists have been trying to combine these chatty robots with "symbolic reasoning"—a type of computer logic that works like a strict, step-by-step math proof. Think of it as giving the robot a rigid checklist to follow so it can't just guess. The big question is: Can we make a robot that is both chatty enough to understand us and strict enough to be trusted?

Enter NeSy-RAG, a new framework proposed by Jonas Gann and Michael Gertz that tries to solve this puzzle. Imagine the robot's library is a massive warehouse full of loose papers (text chunks). Usually, when you ask a question, the robot grabs a few papers, reads them, and guesses the answer. NeSy-RAG does something different. When it grabs a paper, it doesn't just read it; it instantly translates that paper into a tiny, self-contained "logic puzzle" written in a language called Prolog. It's like turning a paragraph of text into a set of strict "If-Then" rules. For example, if the text says, "You can claim benefits if you are over 18," the system turns that into a rule: if age > 18, then eligible.

Here is where it gets clever. The system builds a whole collection of these logic puzzles from the papers it finds. But instead of trying to solve the whole messy library at once, it uses a special trick called "0-arity predicates." Think of this as turning every complex rule into a simple "Yes/No" question, like "Is the person over 18?" or "Is the person a citizen?" The robot then uses a smart search to find only the specific "Yes/No" questions that match your query. It stitches these together into a final logic query and runs it. Because it's running a logic program, the answer is deterministic—it's either true or false, no guessing involved. Even better, the system keeps a perfect "receipt" for every step, showing exactly which piece of text led to which rule, so you can trace the answer back to the source.

But what if the robot doesn't have enough information? Maybe you ask, "Am I eligible?" but the robot doesn't know your age. A normal robot might just guess "Yes" or "No" and hope for the best. NeSy-RAG, however, has a "knowledge-gap detector." It looks at its logic puzzle and realizes, "Hey, I'm missing a value for the 'age' variable!" Instead of guessing, it stops and asks you, "How old are you?" It keeps asking these targeted follow-up questions until it has all the pieces to solve the puzzle. This prevents the robot from making up facts.

The researchers tested this system on a dataset called ShARC, which contains legal questions about UK government benefits. They didn't train the system on this specific data; they just let it figure it out on the fly. The results were promising. NeSy-RAG got the right answer 61.1% of the time. This was significantly better than a standard "chatbot-only" system that tried to do the same thing without the logic rules, which only got 42.8% right. The new system was also faster, taking an average of 7.4 seconds per question compared to 11.4 seconds for the standard chatbot approach.

However, the paper is careful not to call this a perfect solution. The system is still a prototype. Sometimes, it gets a bit too cautious. If it's not 100% sure about a missing fact, it will ask for more information even when it might not be strictly necessary, leading it to say "I need more info" (which the researchers call "more") instead of giving a definitive "Yes" or "No." In fact, it misclassified about 31% of the "Yes" answers as "more," suggesting it generates too many "missing info" flags. While this is a bit annoying for speed, the authors note that in high-stakes situations like law, it's safer to be cautious than to be confidently wrong. The system also struggled a bit with the "irrelevant" questions, sometimes getting confused about whether a text snippet was even related to the question.

In short, NeSy-RAG shows that by turning text into strict logic puzzles and asking for missing pieces of information, we can build AI that is more transparent, more accurate, and less likely to lie. It's not a magic bullet that solves everything perfectly yet, but it's a strong step toward making AI assistants that you can actually trust with the truth.

Technical Summary: NeSy-RAG for Explainable Question Answering

1. Problem Statement

Retrieval-Augmented Generation (RAG) systems enhance Large Language Models (LLMs) by grounding them in external text corpora, yet they suffer from three critical limitations in high-stakes domains:

Opaque Reasoning: Intermediate reasoning steps are difficult to verify, and generated answers cannot be reliably attributed to specific source evidence. Standard Chain-of-Thought (CoT) explanations are often untrustworthy.
Scalability and Query Construction: Constructing symbolic queries over large, monolithic knowledge bases is brittle. Existing neuro-symbolic approaches often rely on manually curated rules or require the model to inspect entire knowledge bases, leading to scalability issues.
Missing User Context: Current RAG pipelines rarely detect missing user-specific information systematically. This often leads to incomplete or incorrect outputs, as LLMs struggle to proactively seek clarifying information when user context is incomplete.

2. Methodology: The NeSy-RAG Framework

NeSy-RAG is a modular neuro-symbolic framework designed as a plug-in extension to existing RAG systems. It synthesizes and executes attributable Prolog modules from retrieved text chunks to perform explainable question answering. The pipeline operates as follows:

2.1. Attributable Prolog Synthesis

For each retrieved text chunk, the system synthesizes a dedicated Prolog module containing rules and facts derived solely from that chunk.

Modularity: By processing chunks in isolation, the system ensures that every fact and rule can be uniquely mapped to its source text, enabling granular explainability and verifiability.
Dynamic vs. Universal Facts: The system distinguishes between universal facts (fully specified by domain knowledge) and dynamic facts (containing arguments whose values are unknown at synthesis time, e.g., user attributes). Dynamic facts are flagged for runtime instantiation.

2.2. 0-Arity Predicate Abstraction & Hierarchical Retrieval

To address the scalability bottleneck of query generation, NeSy-RAG introduces a two-stage retrieval strategy:

Module Synthesis: During the synthesis of a Prolog module, the LLM generates 0-arity rules (Boolean claims) that encapsulate the module's functionality without requiring variable bindings (e.g., can_not_get_winter_fuel_payment :- lives_in_cyprus(true)...).
Predicate Retrieval: Instead of loading the entire knowledge base, the system uses a joint Natural Language–Prolog (NL-PL) embedding model to retrieve the top- $k$ most relevant 0-arity rules based on the user's question.
Query Composition: An LLM composes the final Prolog query using only this small, relevant subset of rules. This constrains the generation space, reduces hallucinations, and keeps context windows small.

2.3. Symbolic Knowledge-Gap Detection

The framework addresses incomplete user context through a symbolic mechanism:

Runtime Interaction: During Prolog execution, if the system encounters a dynamic fact with an undefined value, it triggers a specific interaction method.
Follow-up Generation: The system automatically generates targeted follow-up questions to the user to resolve the missing truth value.
Outcome Handling: If a user cannot provide the necessary information, the system evaluates the query under both Boolean assignments. If the outcome varies, the system returns "more" (indicating the answer is undetermined) rather than guessing.

2.4. Answer Generation

The system executes the composed Prolog query against the retrieved modules. The output includes:

A deterministic answer (Yes/No/More/Irrelevant).
An execution trace that links every reasoning step back to its originating text chunk and document, providing transparent justification.

3. Key Contributions

The paper proposes four main contributions:

Modular Neuro-Symbolic Framework: A system that synthesizes and executes attributable Prolog modules from retrieved text chunks, ensuring every reasoning step is traceable to its source.
0-Arity Predicate Abstraction: A novel method to surface Prolog modules as Boolean claims, enabling constrained and scalable query construction via natural language-to-code retrieval (NL2C).
Symbolic Knowledge-Gap Detection: A mechanism that identifies missing user facts during symbolic inference and proactively triggers follow-up interactions, preventing premature or incorrect conclusions.
Empirical Validation: Demonstration that NeSy-RAG outperforms domain-agnostic LLM baselines on the ShARC benchmark without requiring domain-specific training.

4. Experimental Results

The framework was evaluated on the ShARC benchmark, a conversational question-answering dataset based on UK government legal texts. The evaluation simulated a setting where user context is withheld initially, requiring the system to recover missing information via follow-up questions.

Accuracy: NeSy-RAG achieved 61.1% accuracy (with $k=10$ retrieved 0-arity predicates), significantly outperforming the LLM RAG baseline (42.8%) and the LLM RAG baseline with a larger model (53.7%).
Efficiency: NeSy-RAG had a lower mean execution time (7.4s) compared to the RAG baseline (11.4s). This efficiency is attributed to module reusability, smaller context windows during query generation, and algorithmic follow-up triggering rather than LLM-based generation.
Knowledge Gap Handling: NeSy-RAG correctly classified 61% of instances requiring further information (labeled "more"), compared to only 19% for the LLM RAG baseline. This highlights the effectiveness of the symbolic gap detection mechanism.
Error Analysis: The primary error mode for NeSy-RAG was the over-generation of dynamic predicates, leading to conservative "more" classifications for answerable questions (31% of "Yes" instances misclassified as "more"). However, this conservative bias is noted as preferable to false certainty in high-stakes domains.

5. Significance and Claims

The paper posits that NeSy-RAG offers a viable path toward trustworthy, explainable AI in high-stakes domains like law and healthcare. By combining dense retrieval with symbolic reasoning, it directly addresses the opacity and lack of accountability in standard RAG pipelines.

The authors claim that the framework's significance lies in its ability to:

Provide transparent execution traces that link reasoning steps to source text, a capability pure LLM pipelines lack.
Enable scalable reasoning over large domain corpora without requiring monolithic knowledge bases or manual rule curation.
Systematically detect and resolve missing user context, preventing the system from hallucinating answers when information is insufficient.

While the approach currently relies on a single relevant snippet (single-hop retrieval) and focuses on Boolean dynamic facts, the authors view the successful integration of modular Prolog synthesis with RAG as a foundational step toward more robust, verifiable, and interactive question-answering systems.