Technical Summary: BLINKG – A Benchmark for LLM-Integrated Knowledge Graph Generation

Problem Statement

The construction of Knowledge Graphs (KGs) from heterogeneous data sources remains a labor-intensive process for knowledge engineers, primarily due to the difficulty of establishing semantic correspondences between input data schemas and ontology terms. While declarative solutions (e.g., RML, SPARQL-Anything) have generalized parts of this process, aligning schema elements with ontology concepts still requires intricate transformations and significant manual effort.

Although Large Language Models (LLMs) have been increasingly applied to automate KG construction and semantic alignment, the field lacks a standardized framework to evaluate their effectiveness. Existing studies utilize diverse scenarios, parameters, and metrics, making fair comparisons difficult. Furthermore, existing benchmarks like RODI focus primarily on relational databases and schema-level matching, while challenges like SemTab focus on annotating tables against pre-existing KGs rather than mapping to non-populated ontologies. There is a critical need for a benchmark that evaluates the explicit, declarative mapping capabilities of LLMs across diverse data formats and varying levels of schema alignment.

Methodology

The paper introduces BLINKG, a benchmark designed to assess the mapping capabilities of LLMs in constructing KGs. The methodology is structured around three progressively complex scenarios, a set of defined construction tasks, and a rigorous evaluation protocol.

1. Benchmark Scenarios

BLINKG defines three scenarios based on the degree of alignment between input data and the target ontology:

• Scenario 1 (Basic): Comprises eight atomic cases using simple inputs (CSV, JSON, XML) and a minimal ontology. These cases isolate core tasks such as class identification, subject generation, and simple joins to test fundamental mapping behaviors.
• Scenario 2 (Schema-Aligned): Focuses on the transportation domain using the GTFS-Madrid-Bench. The ontology is built bottom-up from the GTFS specification, resulting in high lexical and structural alignment. This scenario tests the ability of LLMs to handle ontology-specific modeling constructs (e.g., class hierarchies, controlled vocabularies/SKOS) and transformation functions in a realistic setting.
• Scenario 3 (Schema-Distant): Focuses on public procurement using the e-Procurement Ontology (ePO) and data from the Spanish CODICE platform. The input data (deeply nested XML) and the ontology are semantically related but structurally and lexically distant. This scenario challenges LLMs to apply deep domain expertise to establish mappings without direct schema alignment.

2. Defined Tasks

The benchmark decomposes KG construction into specific subtasks, moving beyond end-to-end black-box evaluation:

• Ontology Class Identification: Mapping input fields to ontology classes.
• Subject Generation: Creating unique URIs for instances (simple vs. complex/composite keys).
• Ontology Property Identification: Linking data fields to ontology properties.
• Data Reference Identification: Specifying how to access input data (e.g., column names, XPath).
• Related Entity Class and Joins: Defining conditions for object properties and joining multiple sources.
• Language & Datatype Annotation: Assigning correct language tags and XSD datatypes.
• Transformation Functions: Applying logic to normalize data (e.g., date formatting, enum-to-taxonomy mapping).

3. Evaluation Protocol

• Models Evaluated: Six state-of-the-art LLMs were tested, including DeepSeek-R1, Gemini 2.5 Pro, GPT-4 Omni, OpenAI o3, LLaMa 3.3 70B Instruct, and Mixtral 8x22B Instruct.
• Output Format: Models are prompted to generate structured tabular outputs listing mappings for each task, rather than raw code or natural language descriptions.
• Metrics: Performance is measured using Precision, Recall, and F-score. To address the limitations of strict string matching, the authors employ a similarity-based evaluation combining Levenshtein distance, raw text embedding cosine similarity, and ontology-verbalized embedding cosine similarity.
• Evaluation Types: Results are analyzed via Expert Evaluation (manual validation), Raw Evaluation (direct metric application), and Post-processed Evaluation (heuristic cleaning of outputs).

Key Contributions

The paper outlines five primary contributions:

1. C1: A comprehensive, domain-agnostic framework with multiple scenarios, gold standards, and metrics designed to assess LLM behavior in KG construction.
2. C2: Three progressively complex scenarios reflecting real-world challenges, covering a representative set of typical mapping tasks.
3. C3: Open and reusable resources (ontologies, datasets, gold standards) to foster reproducibility and community adoption.
4. C4: A comprehensive evaluation of six state-of-the-art LLMs, analyzing their performance and generalization capabilities with a focus on traceability and transparency.
5. C5: Practical recommendations and identified research directions for advancing (semi-)automated, LLM-driven KG construction.

Results

The experimental evaluation reveals distinct performance patterns across the scenarios:

• Scenario 1 (Basic): LLMs perform well in simple, lexically aligned tasks. Entity Class identification and Ontology Property identification achieve high similarity scores (often >0.95). However, generating Join Conditions remains challenging (scores ranging 0.37–0.69), and Language Annotation shows substantial variance between models.
• Scenario 2 (Schema-Aligned): Models handle entity and data reference alignment well (similarity >0.90). However, performance drops significantly for structural reasoning tasks. Join Conditions remain difficult (no model exceeds 0.60), and Function-related tasks (naming and output generation) perform poorly (often <0.20), indicating struggles with abstraction and domain-specific logic. Post-processing significantly improves scores for syntactic errors but does not resolve semantic gaps.
• Scenario 3 (Schema-Distant): Performance degrades sharply across all tasks. No task surpasses a mean similarity of 0.53. While Entity Class identification remains the strongest task, structurally complex tasks like Joins, Datatypes, and Functions score below 0.40. Prompt engineering (zero-shot vs. few-shot) improves performance on "local" tasks (e.g., selecting classes) but fails to trigger the complex reasoning required for joins and transformations.

Significance and Claims

The paper claims that BLINKG provides a necessary standardized framework to move beyond fragmented evaluations of LLMs in KG construction. The benchmark demonstrates that while current LLMs offer promising solutions for well-aligned, controlled scenarios, their performance is limited in complex, schema-distant environments requiring deep domain reasoning and structural logic.

Key takeaways emphasized by the authors include:

• Hybrid Approaches are Essential: General-purpose LLMs are unlikely to produce valid solutions in highly complex semantic environments alone. The authors suggest combining LLMs with symbolic reasoning (e.g., OWL-based inference) or human-in-the-loop validation.
• Traceability: Unlike end-to-end pipelines, BLINKG's task-level evaluation provides transparency into where systems succeed or fail, highlighting that errors in complex tasks (like joins) are not merely formatting issues but fundamental reasoning gaps.
• Limitations of Current Benchmarks: The authors acknowledge that existing benchmarks often rely on memorization of public datasets. They propose that future iterations of BLINKG should utilize synthetic ontologies and data to better isolate genuine reasoning capabilities from data leakage.

The paper concludes that while LLMs can generate initial mapping drafts, human expertise remains essential for validating mappings and ensuring semantic soundness, particularly in complex, real-world applications.