💬 NLP

Leveraging LLM-GNN Integration for Open-World Question Answering over Knowledge Graphs

This paper introduces GLOW, a hybrid system integrating a pre-trained Graph Neural Network with a Large Language Model to enable open-world Question Answering over incomplete Knowledge Graphs, alongside the GLOW-BENCH benchmark, achieving significant performance improvements by combining structural graph reasoning with semantic interpretation without requiring fine-tuning.

Original authors: Hussein Abdallah, Ibrahim Abdelaziz, Panos Kalnis, Essam Mansour

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hussein Abdallah, Ibrahim Abdelaziz, Panos Kalnis, Essam Mansour

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but your case file (the Knowledge Graph) is missing half the pages. You know the suspect's name, but the file doesn't say where they live or who their accomplice is.

In the old days, detectives (traditional AI systems) would say, "I can't solve this because the answer isn't in the file." They assumed the file was perfect and complete. But in the real world, files are always messy and incomplete.

This paper introduces a new detective team called GLOW (Graph-LLM for Open-World QA). GLOW is a partnership between two very different experts who cover each other's weaknesses:

The Two Partners

The Map Reader (The GNN):
- Who they are: A Graph Neural Network. Think of them as an expert who has memorized the entire layout of a city. They can look at a street corner and instantly guess, "Based on the shape of the neighborhood, the person living here is probably a doctor."
- Their weakness: They are terrible at reading the actual words on the signs. They know the structure but not the meaning. They might guess "Doctor" when the sign actually says "Dentist."
The Storyteller (The LLM):
- Who they are: A Large Language Model (like the AI you are talking to right now). They are brilliant at understanding language, context, and nuance. If you ask, "Who is the famous actor from that movie?", they know the answer.
- Their weakness: They don't have a map. If you ask them about a specific, obscure fact in a messy database, they might just make up a story (hallucinate) because they don't know what's actually in the file.

The Problem with Previous Teams

Before GLOW, other detective teams tried to solve these "missing page" mysteries in two ways:

The "Search Only" Team: They just asked the Storyteller to guess. Without the map, the Storyteller often guessed wrong or made things up.
The "Retrieval Only" Team: They tried to find the missing page by searching for similar pages. But if the page was completely missing, they hit a dead end.
The "Hybrid" Team (AskGNN): They tried to mix the Map Reader and Storyteller, but they forced the Storyteller to rely too much on the Map Reader's guesses. If the Map Reader was wrong, the whole team failed.

How GLOW Works: The Perfect Handshake

GLOW changes the game by letting the two partners talk to each other in a very specific way, without forcing them to relearn everything.

The Map Reader does a quick scan: When a question comes in (e.g., "What is the species of this protein?"), the Map Reader looks at the neighborhood in the graph and says, "I'm not 100% sure, but based on the neighbors, it's likely Species A, Species B, or Species C." It gives a shortlist of top guesses.
The Storyteller gets the context: The system takes that shortlist and the actual facts surrounding the protein (the "neighborhood" facts) and writes them down in a clear note.
The Storyteller makes the final call: The Storyteller reads the note: "Here are the top 3 guesses from the Map Reader, and here are the facts we know. Based on this, which one is the real answer?"

Because the Storyteller now has both the structure (the Map Reader's shortlist) and the facts (the context), it can reason much better. It doesn't have to guess blindly, and it doesn't have to rely solely on the Map Reader's potentially flawed intuition.

The "Open-World" Twist

The big innovation here is that GLOW is designed for Open-World problems.

Closed World: "The answer must be in the file. If it's not there, I give up."
Open World: "The answer might be missing from the file. I need to use logic and clues to infer what it should be."

GLOW is great at this because the Map Reader can predict missing links (like guessing a missing street name based on the surrounding blocks), and the Storyteller can verify if that guess makes sense linguistically.

The New Test Drive (GLOW-BENCH)

To prove this works, the authors built a new test track called GLOW-BENCH.

Imagine a driving test where they intentionally remove the stop signs and traffic lights from the road.
They asked 1,000 questions across different fields (medicine, movies, business) where the answer was deliberately hidden.
The Result: GLOW crushed the competition. It was up to 53% better than other systems. It was especially good at complex, multi-step questions (like "Who is the CEO of the company that bought the movie studio that made this film?").

Why This Matters

In the real world, knowledge is never perfect. Medical records are incomplete, scientific discoveries are ongoing, and business data is messy.

Old AI: "I can't answer that; the data isn't there."
GLOW: "The data isn't there, but based on the pattern and the context, here is the most logical answer."

By combining the "gut feeling" of a map expert with the "brain power" of a language expert, GLOW allows AI to navigate the messy, incomplete reality of the real world without getting lost or making things up.

1. Problem Statement

The paper addresses Open-World Question Answering (OW-QA) over Knowledge Graphs (KGs).

The Challenge: Traditional KGQA systems operate under a "closed-world assumption," where they assume all necessary facts exist within the graph. In real-world scenarios (e.g., biomedicine, finance), KGs are often incomplete, evolving, or contain missing links.
Limitations of Existing Approaches:
- LLMs alone: Excel at semantic understanding but lack structured reasoning and often hallucinate facts when the specific answer path is missing from the graph.
- GNNs alone: Model graph topology effectively but struggle with semantic interpretation of natural language questions.
- Existing Hybrid Systems: Methods like AskGNN or G-Retriever often rely on retrieving explicit paths or structural embeddings. When answer paths are missing (the core of OW-QA), these systems fail, leading to noisy retrieval or hallucinated paths. They often assume the graph is complete or rely on fine-tuning, which limits scalability and generalization.

2. Methodology: The GLOW Framework

The authors propose GLOW (Graph-LLM for Open-World QA), a hybrid system that synergistically combines a pre-trained Graph Neural Network (GNN) and a Large Language Model (LLM) without requiring fine-tuning of the LLM.

Core Architecture

GLOW operates on a four-stage pipeline:

Question Understanding & Linking:
- Extracts the main entity ( $Q_n$ ), its type, the target relation ( $Q_e$ ), and the KG name from the user's natural language question.
- Uses an LLM-based prompt to map these elements to KG schema URIs (Entity-Relation Linking).
- Resolves the question entity to a specific node ( $v_t$ ) in the KG via SPARQL queries.
Retrieval (KG Context):
- Retrieves a relevant subgraph (1-hop neighbors) connected to $v_t$ using a generated SPARQL query.
- Serializes these triples into text (Relational Context - RC) to provide grounded semantic context to the LLM.
Augmentation (GNN Prediction):
- A pre-trained GNN (trained specifically on the question pattern, e.g., "Protein $\to$ Species") predicts the top- $k$ candidate answers based on graph topology.
- These candidates are serialized and added to the prompt as structural guidance.
Generation:
- The LLM receives a structured prompt containing the question, the linked entities, the serialized KG context (RC), and the GNN-predicted candidates.
- The LLM reasons jointly over the linguistic semantics and structural signals to select the final answer.

System Variants

The authors explore three integration strategies:

GLOW-G: Uses only the KG subgraph context (RC).
GLOW-N: Uses only the GNN-predicted top- $k$ candidates.
GLOW-GN (Hybrid): Combines both RC and GNN predictions. This is the primary proposed method, allowing the LLM to cross-verify structural predictions with textual context.

Key Innovation: Unlike previous methods that rely on retrieving full answer paths (which may not exist), GLOW uses the GNN to predict missing links/candidates and the LLM to validate them against the available context. This decouples performance from the strict requirement of a complete graph.

3. Key Contributions

GLOW System: A novel, non-fine-tuned hybrid architecture that integrates GNN-driven candidate generation with LLM-based reasoning for open-world settings. It avoids the "retrieval failure" of path-based methods by predicting missing information.
GLOW-BENCH: A new benchmark consisting of 1,000 open-world questions across diverse domains (BioKG, CrunchBase, LinkedMDB, YAGO4).
- Features: Includes 1-hop and 2-hop reasoning tasks.
- Incompleteness: Gold answers are deliberately removed from the KG to simulate real-world incompleteness.
- Evaluation: Uses an "LLM-as-a-Judge" framework with both Exact Match and Hierarchical Match metrics to handle semantic variations.
Comprehensive Evaluation: Demonstrates significant improvements over state-of-the-art baselines (AskGNN, GCR, GoG, and LLM-only) across multiple LLMs (Qwen, DeepSeek, GPT-4o-mini, Granite).

4. Experimental Results

The paper evaluates GLOW on existing datasets (Arxiv, OGBN) and the new GLOW-BENCH.

Performance Gains:
- GLOW-GN consistently outperforms all baselines.
- On GLOW-BENCH, it achieves up to 53.3% improvement over the best baseline (AskGNN) and an average improvement of 38%.
- On standard datasets, it shows significant gains in 1-hop and 2-hop reasoning tasks.
Robustness:
- Weak GNNs: Even when the underlying GNN performs poorly (e.g., <50% accuracy), GLOW-GN outperforms the GNN alone by up to 12% by leveraging textual cues. In contrast, systems like AskGNN degrade significantly if the GNN is weak.
- Small LLMs: GLOW enables smaller models (e.g., Qwen3-8B) to achieve performance comparable to or better than larger commercial models (GPT-4o-mini) when augmented with structural signals.
Efficiency:
- Training: GLOW requires only ~1.7 hours for GNN training (decoupled from LLM), whereas joint training methods (AskGNN) take ~23.4 hours.
- Inference: GLOW-N is the fastest (8.2s/question) due to compact prompts. GLOW-GN balances speed and accuracy (11.5s).
Error Analysis:
- GLOW reduces "hallucination" of answer paths compared to GoG (which generates paths without schema constraints).
- The system is most effective in domain-specific tasks (e.g., BioKG) where LLMs lack pre-trained knowledge, relying on the GNN's structural inference.

5. Significance

Paradigm Shift: Moves KGQA from "retrieval-based" (finding existing paths) to "predictive reasoning" (inferring missing facts), which is critical for dynamic, real-world applications.
Scalability: By decoupling GNN training from LLM inference and avoiding LLM fine-tuning, GLOW is highly scalable and adaptable to new domains.
Reliability: The hybrid approach mitigates the weaknesses of both components: the GNN provides structural constraints to prevent LLM hallucinations, while the LLM provides semantic grounding to interpret graph structures.
Benchmarking: GLOW-BENCH fills a gap in the literature by providing a rigorous testbed for open-world reasoning, moving beyond the closed-world assumptions of traditional benchmarks like WebQSP.

In conclusion, GLOW demonstrates that integrating structured graph reasoning with semantic language understanding is the most effective path forward for answering questions over incomplete and evolving knowledge graphs.