SocraticKG: Knowledge Graph Construction via QA-Driven Fact Extraction

1. Problem Statement

The construction of Knowledge Graphs (KGs) from unstructured text using Large Language Models (LLMs) faces a fundamental trade-off between factual coverage and structural coherence:

• Direct Extraction: Methods that extract triples directly from raw text often capture surface-level facts but miss latent logical ties, resulting in fragmented subgraphs with weak semantic connectivity.
• Consolidation-Centric Strategies: Methods that prioritize entity structures first (e.g., GraphRAG, KGGen) often filter out contextual nuances and implicit relations to maintain graph organization, leading to information loss and incomplete coverage.
• The Core Challenge: Existing approaches struggle to externalize the full narrative logic of a document into a structured graph without sacrificing either the density of facts or the connectivity required for complex reasoning.

2. Methodology: SocraticKG

The authors propose SocraticKG, a multi-stage framework that treats Question-Answering (QA) not merely as a retrieval tool, but as a structured intermediate representation to systematically unfold document-level semantics before triple extraction. The pipeline consists of three main stages:

A. 5W1H-Guided QA Generation

Instead of extracting triples directly, the LLM first decomposes the document into discrete, self-contained QA pairs.

• 5W1H Framework: The model is prompted to generate questions based on Who, What, When, Where, Why, and How. This ensures the capture of surface entities as well as complex dependencies (causal rationales, procedural details).
• Contextual Independence: The LLM is instructed to generate answers that are fully understandable without referencing the original text. This involves replacing pronouns (e.g., "it," "they") with explicit entity names to resolve referential ambiguities early.

B. Triple Extraction from QA

The generated QA pairs serve as independent extraction units, reducing the complexity of parsing long, complex sentences.

• Atomic Decomposition: Each QA pair is broken down into fine-grained, atomic triples.
• Entity Clarity: All entities must be specific noun phrases; triples containing ambiguous pronouns are discarded.
• Simplified Relations: Predicates are distilled into concise verb phrases to facilitate later canonicalization.

C. Graph Construction & Canonicalization

The extracted triples are unified into a cohesive graph.

• Canonicalization: A cluster-then-refine process (based on Mo et al., 2025) is used. Semantic embeddings (via K-means) group similar entities and relations, followed by an LLM-based refinement step to resolve synonyms and surface-form variations, merging fragmented triples into a single canonical graph.

3. Key Contributions

1. QA-Mediated Semantic Scaffolding: The paper formalizes QA as an intermediate representation that explicitly articulates implicit connections and contextual nuances before structural extraction, acting as a "scaffold" for document understanding.
2. 5W1H-Guided Expansion: The introduction of a systematic 5W1H questioning strategy to surface latent causal and procedural dependencies that are typically overlooked in direct extraction pipelines.
3. Resolution of the Coverage-Connectivity Trade-off: The method demonstrates that it is possible to achieve high factual retention and high structural cohesion simultaneously, overcoming the limitations of previous approaches that forced a choice between the two.

4. Experimental Results

The method was evaluated on the MINE benchmark (for factual retention and structural metrics) and HotpotQA (for downstream multi-hop reasoning).

• Factual Retention (MINE Benchmark):

  • SocraticKG consistently outperformed state-of-the-art baselines (Direct Extraction, GraphRAG, KGGen) across five different LLM backbones (Qwen-2.5, GPT-4o-mini, GPT-4o, Gemini-2.5, Claude-4).
  • Peak Performance: Achieved 96.3% factual retention with Claude-4, significantly higher than Direct Extraction (86.8%) and KGGen (69.1%).
  • The "w/o 5W1H" variant still outperformed direct extraction, proving the value of the QA scaffold itself, but the full 5W1H version provided the highest gains.

• Structural Cohesion:

  • Connectivity: SocraticKG produced graphs with higher Average Degree (Deg) and lower Normalized Fragmentation Index (NFI) compared to baselines.
  • Volume: It extracted significantly more triples (e.g., ~14,849 triples with Claude-4 vs. ~4,417 for Direct Extraction) while maintaining lower fragmentation, indicating that the extra facts were meaningfully connected rather than redundant.

• Downstream Reasoning (HotpotQA):

  • On "Hard Bridge" samples requiring multi-hop reasoning, SocraticKG achieved the highest accuracy.
  • With Claude-4 at 3-hop depth, SocraticKG reached 56.38% accuracy, outperforming Naive RAG by 8.5 percentage points and Direct Extraction by 16.5 percentage points.
  • This confirms that the improved graph connectivity directly translates to better reasoning capabilities.

5. Significance and Conclusion

• Paradigm Shift: The paper argues that explicit semantic organization via QA generation is not just a preprocessing step but a critical component for maintaining graph fidelity in LLM-based KG construction.
• Robustness: The approach is robust across models of varying sizes and capabilities, suggesting that the 5W1H scaffold compensates for weaker reasoning in smaller models.
• Practical Impact: By resolving the tension between information density and graph structure, SocraticKG provides a more reliable foundation for document-grounded knowledge representation, enabling complex multi-hop reasoning tasks that previously failed with fragmented graphs.
• Limitations: The method incurs higher token consumption (2–3x more than direct extraction) due to the multi-stage pipeline, and performance relies on the LLM's ability to formulate specific, deep queries (failure analysis showed "What" questions can sometimes be too broad).

In summary, SocraticKG effectively bridges the gap between unstructured text and structured knowledge by using interrogative reasoning to "unpack" document semantics, resulting in knowledge graphs that are both factually complete and structurally coherent.