Technical Summary: GPTKB 2.0

Problem Statement

Automated Knowledge Base Construction (AKBC) has traditionally relied on curated resources like Wikipedia and Wikidata for entity canonicalization, or on surface string clustering for text-based extraction. However, recent approaches that generate knowledge bases directly from Large Language Models (LLMs) face a critical structural limitation: LLMs are trained on strings, not entities. Consequently, treating surface strings as unique identifiers leads to two complementary failure modes:

1. Homonymy: Distinct entities with identical labels (e.g., the city "Munich" and the film "Munich") are incorrectly merged.
2. Synonymy: Identical entities with different labels (e.g., "Île-de-France" and "Île-de-France region") are treated as distinct entities.

Existing disambiguation methods often rely on external inventories (e.g., linking to Wikidata IDs), which limits the scope of construction to pre-existing resources. Others use post-hoc clustering, which addresses synonymy but fails to resolve homonymy at scale. The core challenge is to construct a large-scale, open-domain knowledge base directly from an LLM's parameters while maintaining explicit internal canonicalization of entities, relations, and classes without relying on external identifiers.

Methodology: GPTKB 2.0

The authors propose GPTKB 2.0, a methodology for constructing disambiguated knowledge bases directly from LLMs. The system treats the LLM as the sole knowledge source, utilizing a recursive, context-guided pipeline that alternates between elicitation and consolidation along the subject–object frontier.

Data Model

The knowledge base is defined as a quadruple $(T, E, R, C)$ containing triples, entities, relations, and classes.

• Unique Identifiers: Every element is assigned a unique internal ID, decoupling identity from surface labels.
• Descriptions: Each element includes a generated textual description to support context-based disambiguation.
• Aliases: Synonymous labels are attached as aliases to a single canonical ID.
• Homonymy Handling: Distinct entities sharing a label are assigned different IDs, distinguished by their unique descriptions.

Construction Pipeline

The pipeline operates through three core phases, iteratively expanding the KB from seed entities:

1. Elicitation: The LLM is prompted to generate triples for a given entity. Crucially, the prompt includes the entity's description alongside its label. This context ensures that elicited facts pertain to the specific intended entity (e.g., distinguishing the film "Munich" from the city) rather than a homonymous one. The model is also instructed to generate instanceOf triples to capture entity types.
2. Named Entity Recognition (NER): The LLM classifies elicited objects as either literals or Named Entities (NEs). This step is context-guided, using the source triple to determine if an object (e.g., a code like "01099") is a literal or an entity.
3. Disambiguation (NED): This is the core consolidation phase, split into two rounds to balance accuracy and scalability:
   • Triple-Context NED: For NEs, the system retrieves top-$k$ candidates based on label embedding similarity. The LLM is prompted with the target label, the source triple as context, and the candidates' labels/descriptions. The model decides to match, select "None of the above" (new entity), or "Unsure."
   • Description Generation: Entities marked as "None of the above" are routed to a description generation step. The LLM generates a brief description based on the target label and source triple. This description serves as the entity's canonical definition.
   • Description-Context NED: For high-density homonyms (e.g., "Faculty of Law" appearing hundreds of times), the candidate set is restricted to entities sharing the exact label. The LLM then uses the generated description (rather than the source triple) to distinguish between candidates based on semantic similarity.

Scalability Mechanisms

To enable million-scale construction, the authors introduce Guarded Parallelization and Caching:

• Guarded Parallelization: To prevent the redundant creation of duplicate entities when processing concurrent requests for the same label, the system clusters pending NED requests by label. It processes only the earliest-elicited instance immediately; subsequent instances are deferred until the first is resolved, ensuring the entity exists as a candidate for later matches.
• Caching: Disambiguation decisions for frequent (relation, object) pairs are cached to bypass redundant NER and NED calls, significantly reducing API costs.

Key Contributions

1. Methodology: A parallelized, context-guided pipeline for KB construction that uses an LLM as the exclusive knowledge source, addressing both synonymy and homonymy through on-the-fly disambiguation without external identifiers.
2. Scale and Quality: The construction of a materialized KB containing 38.4 million triples and 1.6 million disambiguated entities. The system consolidated 2.3 million surface labels into 1.6 million canonical entities and correctly distinguished 131,990 homonymous entities sharing 41,275 labels.
3. Novelty: The first million-scale, LLM-native KB with explicit internal canonicalization outside the Wikimedia sphere. Approximately 36.8% of the constructed entities are novel to Wikidata, demonstrating the ability to capture long-tail knowledge.

Results and Evaluation

The authors evaluated the constructed KB using manual annotation and automated agent-based verification:

• Disambiguation Accuracy:
  • Merge Accuracy: 98% for homonyms and 91% for synonyms.
  • Split Accuracy: 100% for homonyms (no false splits in the sample) and 95% for synonyms.
  • Relations/Classes: Achieved 89–91% merge accuracy and 91–94% split accuracy.
• Factuality:
  • Triple Precision: 94.5% of manually sampled triples were verified as true; 92.8% in automated evaluation.
  • Entity Factuality: 96% of manually sampled entities were verifiable as real-world entities.
• Comparison to Baseline: A baseline using surface strings resulted in 2.3 million entities (vs. 1.6M in GPTKB 2.0) and failed to distinguish 131,990 homonymous entities. In a domain-specific test (Ancient Babylon), the baseline conflated distinct entities (e.g., different "Paris" figures), while GPTKB 2.0 maintained distinct identities.

Significance

The paper claims that GPTKB 2.0 represents a significant departure from prior Wikimedia-centric works. By demonstrating that a large, disambiguated, general-domain entity repository can be constructed directly from LLM parameters without external grounding, the work opens the possibility of exploring knowledge spaces beyond curated resources. The authors highlight that LLM-derived KBs are particularly effective at covering the "long-tail" of knowledge (entities appearing infrequently), where curated efforts are sparse. The resulting KB is made available for aggregate statistical analysis, enabling the study of biases, topical foci, and logical consistency at a scale previously unattainable for LLM-native knowledge bases.