💬 NLP

KGCaRe: Explainable Complex Conditional Question Answering using Automatic Knowledge Graph Construction and Context Retrieval with LLMs

The paper introduces KGCaRe, a hybrid framework that enhances complex conditional question answering by combining neural retrieval with LLM-driven automatic knowledge graph construction and iterative graph traversal, demonstrating superior performance over existing baselines across multiple models and datasets.

Original authors: Ghanshyam Verma, Simanta Sarkar, Devishree Pillai, Hotaka Shiokawa, Yourong Xu, Fiona Veazey, Peter Hubbert, Hui Su, Paul Buitelaar

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ghanshyam Verma, Simanta Sarkar, Devishree Pillai, Hotaka Shiokawa, Yourong Xu, Fiona Veazey, Peter Hubbert, Hui Su, Paul Buitelaar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a massive, multi-layered mystery, like figuring out exactly why a character in a video game can or cannot buy a specific item. You have a huge library of rulebooks (documents) and a giant, messy notebook of clues (a knowledge graph). If you just ask a super-smart robot (an AI) to read the whole library and guess the answer, it might get confused, miss a tiny rule buried on page 400, or make up a reason that sounds good but isn't true. This is the world of "Question Answering" for computers. Scientists are trying to teach these robots to be better detectives, especially when the questions are tricky, full of "if this, then that" conditions, and require connecting dots across different pages. The paper you are about to read tackles this by giving the robot two superpowers at once: the ability to read the messy text like a human, and the ability to follow a strict, logical map of facts like a computer.

The researchers behind this study, Ghanshyam Verma and his team, noticed that while AI is great at chatting, it often stumbles when asked complex questions that depend on specific conditions, like legal rules or policy details. To fix this, they built a new system called KGCaRe. Think of KGCaRe as a detective who doesn't just read the case file; they also build a physical map of the clues as they go. Instead of just asking the AI to "guess" the answer from a pile of text, KGCaRe first uses the AI to turn the documents into a structured "Knowledge Graph"—a web of connected facts (like "Applicant" → "must submit" → "Form"). Then, it uses a second method to find relevant text snippets that might not be in the map yet.

Here is how the magic happens: When a question is asked, KGCaRe doesn't just stop and answer. It starts walking through the web of facts, step-by-step. If the first step doesn't give it enough info, the AI acts like a curious explorer, asking, "Okay, who else is involved? What else should I look for?" It uses these new clues to jump to the next part of the map. If it still can't solve the mystery, it grabs relevant text from the original documents to fill in the gaps. Finally, it combines the logical path it walked on the map with the text it found to give an answer that is not only correct but also comes with a clear explanation of how it figured it out.

The team tested this system on two tough datasets: one with complex UK policy questions and another with Wikipedia-based riddles. They pitted KGCaRe against other methods, including AI that just guesses, AI that only reads text, and AI that only follows a graph. The results were clear: KGCaRe consistently won. Whether they used a smaller open-source AI or a powerful one like GPT-4o, the hybrid approach of combining the map and the text led to better answers, especially for those tricky "if-then" questions. For example, on the UK policy dataset, KGCaRe achieved an average score of 57.89, beating the next best method by a significant margin. The paper suggests that by giving AI both a structured map and the freedom to read raw text, we can make them much better at solving real-world puzzles where missing a single condition could change the whole answer. And the best part? Because the system keeps a record of every step it took on the map, it can show you exactly how it reached its conclusion, making the AI's thinking transparent and trustworthy.

Technical Summary: KGCaRe

Problem Statement

Answering complex conditional questions using Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) remains a significant challenge, particularly within domain-specific contexts. General-purpose LLMs often underperform when input involves conditional logic or requires reasoning across specialized content. This performance drop is attributed to a lack of fine-tuning on specialized corpora and a tendency to hallucinate when context is insufficient. Conditional questions are uniquely sensitive to missing or implicit constraints; the model must be aware of both the query and the specific conditions under which an answer is valid. Existing solutions, such as standard RAG (relying on keyword or similarity search) or purely symbolic Knowledge Graph (KG) approaches, often fail to retrieve critical logical dependencies or struggle with sparse graph connectivity.

Methodology: KGCaRe

The authors propose KGCaRe (Knowledge Graph Context aware Reasoning for complex Question Answering), a hybrid approach that integrates symbolic reasoning over LLM-generated Knowledge Graphs with neural retrieval from vector stores. The system operates through the following components:

1. Automatic Knowledge Graph Construction

KGCaRe constructs a KG from unstructured documents using a multi-prompt LLM extraction strategy stored in a Neo4j database. This process involves four progressive stages:

Stage 1 (Contextual Understanding): Extracts initial entities and relations, emphasizing temporal, causal, procedural, and conditional constructs.
Stage 2 (Conditional Expansion): Focuses on conditional logic, exceptions, and "if-then" structures to capture regulatory nuances and branching decision logic.
Stage 3 (Refinement & Logical Expansion): Validates completeness and uses transitive or implied logic to infer new triples (e.g., deriving that a Guardian can delegate consent to a Court if the Parents can delegate to the Court).
Stage 4 (Normalization): Formats and ingests the triples into the graph database.

2. Semantic Vector Indexing

In parallel, documents are embedded into high-dimensional vectors using an LLM-powered embedder and stored in a FAISS index. This enables fast, approximate nearest-neighbor search to retrieve relevant text passages based on semantic similarity to the user's query.

3. Iterative Graph Traversal (Symbolic Reasoning)

The core inference mechanism is an LLM-guided iterative graph traversal (Algorithm 1):

Initialization: Topic entities ( $E_{topic}$ ) are extracted from the question to anchor the traversal.
Depth-Bounded Search: The system traverses the graph up to a maximum depth $D$ $D$ .
- Depth 1: Uses partial matching (including subword matches) for broad exploration.
- Subsequent Depths: Uses exact matching for focused reasoning.
Clue Entity Mechanism: If the initial traversal yields insufficient context, the LLM is prompted to generate "clue entities" ( $E_{clue}$ ) to guide further traversal, inspired by the Tree-of-Traversal approach.
Pruning: An LLM-powered function ($PruneTriples$) filters extracted triples, retaining only those contextually useful for the specific question.
Termination: The process halts if sufficient context is gathered to answer the question, or if no new clues/triples are found.

4. Fused Answer Generation

The final answer is generated by feeding a custom prompt to the LLM. This prompt synthesizes:

Symbolic Context: Curated triples and traversal paths from the KG.
Neural Context: Top-ranked text passages from the vector store.
The prompt explicitly instructs the model to incorporate evidence from both sources and to state any conditions under which the answer holds, ensuring the output is both accurate and explainable.

Key Contributions

End-to-End Pipeline: Design and implementation of an automated pipeline for constructing KGs from long, complex documents using a multi-prompt LLM triple extraction approach.
Hybrid Architecture: Introduction of KGCaRe, which uniquely combines symbolic KG retrieval with vector-based semantic search to address complex conditional QA.
Empirical Validation: Demonstration that KGCaRe outperforms various baselines (Vanilla LLM, Code Prompt, Text Prompt, Think-on-Graph, Vanilla RAG, and HybridContextQA) across multiple LLMs (Mistral, Mixtral, GPT-3.5, GPT-4o).
Reproducibility: Public release of the software pipeline and associated prompts.

Experimental Results

The authors evaluated KGCaRe on two datasets: ConditionalQA (UK policy documents with complex conditional constraints) and HotpotQA (Wikipedia-based multi-hop questions).

Performance: KGCaRe consistently achieved the highest average F1 scores across all answer types (Yes/No, Span, Conditional) and all tested LLMs.
- On ConditionalQA with Mistral, KGCaRe achieved an F1 of 57.89, significantly outperforming Vanilla RAG (40.12) and HybridContextQA (45.17).
- On HotpotQA with GPT-4o, KGCaRe achieved an F1 of 80.21, surpassing HybridContextQA (73.37) in most categories, though the two methods achieved similar F1 scores in the Yes/No category.
Conditional Reasoning: The method showed particular strength in the "Conditional" category, where reasoning requires multi-hop steps and handling implicit constraints.
Ablation Study: When analyzing 106 Yes/No questions with single definitive answers, KGCaRe maintained a lead over baselines across all models, including high-capacity models like GPT-4o (88.67 vs. 85.85 for Text Prompt).
Baseline Comparison: Pure KG approaches (Think-on-Graph) performed poorly, often failing to retrieve necessary context from sparse graphs. Pure RAG approaches performed better but lacked the structured reasoning capabilities of the hybrid model.

Significance and Claims

The paper claims that KGCaRe validates the hypothesis that a hybrid context composed of both symbolic (structured) and neural (unstructured) representations improves performance on complex conditional QA tasks. The significance of the work lies in its ability to:

Enhance Explainability: By maintaining an explicit reasoning trace (traversed paths and pruned triples), the system provides human-understandable explanations for its answers, a critical feature for high-stakes domains like law and public policy.
Overcome Sparse Graph Limitations: Unlike methods that rely solely on dense graph connectivity, KGCaRe supplements sparse KGs with semantically retrieved text, allowing robust reasoning even when the KG is incomplete.
Generalize Across Models: The approach demonstrates consistent improvements regardless of the underlying LLM's capacity, from open-source models (Mistral) to proprietary models (GPT-4o).

The authors conclude that this framework offers a viable path toward more precise and transparent QA systems in specialized domains where answer faithfulness and traceability are paramount.