Scaling Knowledge Graph Construction through Synthetic Data Generation and Distillation

1. Problem Statement

Document-level Knowledge Graph (KG) construction faces a fundamental scaling challenge. Current approaches suffer from a trade-off between cost and quality:

• High-Cost/High-Quality: Methods relying on large, proprietary LLMs (e.g., GPT-4o) via zero-shot or few-shot prompting are economically unviable for large-scale corpora due to high inference costs and API limits.
• Low-Cost/Low-Quality: Smaller models often produce incomplete, inconsistent, or hallucinated graphs when used in a single-step prompting manner.
• Root Cause: The authors argue this limitation is not due to inherent model capabilities but rather a scarcity of high-quality, supervised training data for document-level, ontology-free KG construction. Existing datasets (like DocRED) are limited to specific schemas, and open-domain KGs lack large-scale training corpora.

2. Methodology

The paper proposes a two-stage, data-centric framework: SynthKG (data synthesis) and Distill-SynthKG (model distillation).

A. SynthKG: Multi-Step Data Synthesis Pipeline

SynthKG is a systematic pipeline designed to generate high-quality document-KG pairs using a large teacher model (Llama-3.1-70b). It decomposes the extraction task into reproducible stages to ensure consistency and coverage:

1. Chunking: Documents are split into semantically complete, non-overlapping text chunks (256 tokens) to prevent information loss in long contexts.
2. Decontextualization: A critical step where the LLM rewrites each chunk to resolve entity references (e.g., replacing "he" with "John Doe") based on the preceding chunk. This creates self-contained units, ensuring entity consistency across the document without requiring the full document context for every inference.
   • Validation: The authors verified that entities are typically re-mentioned within less than one chunk (average distance 0.9), making the "preceding chunk" strategy sufficient.
3. Entity & Proposition Extraction:
   • Entities: Extracted with types.
   • Propositions: Instead of just extracting raw triplets, the model generates natural language sentences (propositions) describing the relationship.
   • Triplets: Derived from propositions, formatted as (Head, Predicate, Tail).
   • Innovation: The inclusion of propositions serves as an intermediate "chain of thought," improving the coherence of the extracted triplets and providing rich, indexable retrieval units.

B. Distill-SynthKG: Single-Step Model Distillation

To overcome the latency and cost of the multi-step SynthKG pipeline at inference time:

• Distillation: The synthetic document-KG pairs generated by SynthKG are used to fine-tune a smaller student model (Llama-3-8b).
• Mechanism: The student model learns the pattern of the multi-step process (chunking, decontextualization, extraction) implicitly.
• Result: The distilled model (Distill-SynthKG) can process a full document in a single forward pass, generating high-quality KGs without the need for multiple API calls or complex prompting chains.

C. Evaluation Framework

The authors introduce a novel evaluation method for ontology-free KGs:

• Proxy Ground Truth: They repurpose multi-hop QA datasets (MuSiQue, 2WikiMultiHopQA, HotpotQA). Using GPT-4o, they convert QA pairs into "proxy triplets" where the answer is the head, tail, or predicate.
• Metrics:
  • Semantic Score: Cosine similarity between ground-truth and extracted triplets.
  • Triplet Coverage: Recall of critical triplets needed to answer questions.
  • F1 Score: Surface-level accuracy.
• Retrieval Framework: A Proposition-Entity Graph Retriever is proposed. It retrieves semantically similar propositions, traverses the graph to find connected entities, and uses an LLM for re-ranking. This leverages the rich context of propositions rather than sparse triplets.

3. Key Contributions

1. SynthKG Pipeline: A systematic, multi-step data synthesis engine that generates 100K high-quality, ontology-free document-KG pairs, addressing the training data scarcity.
2. Distill-SynthKG: A distilled 8B model that achieves performance comparable to or better than 70B models and GPT-4o, proving that smaller models can match large-model KG construction quality with the right synthetic data.
3. Evaluation Framework: A new methodology for evaluating document-level KGs using repurposed multi-hop QA datasets and semantic similarity metrics.
4. Graph-Based Retrieval: A novel retrieval framework (Graph+LLM) that utilizes the proposition-entity structure for superior multi-hop reasoning.
5. Cost Efficiency: Demonstrates a paradigm shift from scaling model parameters to scaling training data, achieving state-of-the-art results at a fraction of the inference cost.

4. Experimental Results

The experiments validate the approach across three dimensions: KG quality, retrieval, and Question Answering (QA).

• KG Quality (Coverage):

  • Distill-SynthKG-8b outperformed the untrained Llama-3-8b, the 70B model (Llama-3-70b), and even the multi-step SynthKG-70b pipeline in terms of triplet coverage and semantic scores.
  • It generated more triplets with higher recall than direct prompting, proving that the multi-step decomposition prevents information loss.
  • Precision: Manual inspection showed only ~4% incorrect triplets and ~5% meaningless ones, indicating high fidelity.

• Retrieval Performance:

  • The Graph+LLM retriever using Distill-SynthKG-8b achieved a 28.27 point average improvement in Hits@2 over the pre-trained Llama-3-8b.
  • It outperformed standard dense retrieval and dense retrieval with LLM re-ranking.
  • Performance was highly competitive with GPT-4o-based retrieval, despite the 8B model being significantly cheaper.

• Multi-Hop QA:

  • Distill-SynthKG-8b achieved the best results across all tested frameworks (LlamaIndex, Chain-of-Triplet, Graph+LLM).
  • In the Graph+LLM setting, it achieved a +15.2% absolute gain in Exact Match (EM) over Llama-3-8b and outperformed GraphRAG and HippoRAG (which use GPT-4o).

• Cost Analysis:

  • Inference Cost: Distill-SynthKG-8b costs approximately $0.00267 per document, compared to $0.031 for the multi-step SynthKG pipeline (using 70B models). This represents a ~91% reduction in cost (or ~8.6% of the original cost).
  • Training Cost: Fine-tuning the 8B model on 30K synthetic documents cost only $36.65.

5. Significance

This paper fundamentally shifts the paradigm for Knowledge Graph construction:

• From Model Scaling to Data Scaling: It demonstrates that the "intelligence" required for complex structured extraction can be transferred from large models to smaller, efficient models through high-quality synthetic data, rather than relying on massive parameter counts.
• Economic Viability: By enabling smaller models to perform tasks previously reserved for expensive API calls (GPT-4o), it makes large-scale, document-level KG construction economically feasible for enterprise applications.
• RAG Enhancement: The proposed retrieval framework shows that structured KGs with rich propositions significantly enhance multi-hop reasoning capabilities in RAG systems, outperforming both dense retrieval and existing KG-RAG baselines.
• Open Resource: The authors release the 100K synthetic document-KG dataset, fostering future research in data-centric knowledge extraction.