🤖 machine learning

A Unified Benchmark for Evaluating Knowledge Graph Construction Methods and Graph Neural Networks

This paper introduces a unified, reproducible benchmark in the biomedical domain that jointly evaluates the robustness of Graph Neural Networks on noisy, text-derived graphs and the effectiveness of various knowledge graph construction methods by comparing them against a high-quality expert-curated reference.

Original authors: Othmane Kabal, Mounira Harzallah, Fabrice Guillet, Hideaki Takeda, Ryutaro Ichise

Published 2026-05-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Othmane Kabal, Mounira Harzallah, Fabrice Guillet, Hideaki Takeda, Ryutaro Ichise

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand the complex world of medicine. To do this, you give the robot a "map" called a Knowledge Graph. This map connects medical terms (like "Heart" or "Gene") with relationships (like "treats" or "causes").

The problem is that most of these maps are built by humans, which is slow and expensive. So, scientists started using AI to build these maps automatically from text. But, just like a child drawing a map from a storybook, the AI often makes mistakes: it might connect the wrong dots, miss important landmarks, or draw lines that don't make sense. This creates a "noisy" map.

This paper introduces a new testing ground (a benchmark) to solve two big problems at once:

How good is the map-maker? (Does the AI that builds the map do a good job?)
How good is the map-reader? (Can the AI that learns from the map still work well even if the map is messy?)

Here is how they built this testing ground, explained simply:

1. The Three Maps

To test things fairly, the researchers created three different maps based on the exact same pile of medical text (abstracts from medical journals):

The "Gold Standard" Map (The Reference): This is a perfect, clean map built by human experts using a massive medical dictionary (UMLS). It's the "answer key." It shows us the best possible performance a robot could ever hope to achieve.
The "Robot Map A" (GT2KG): This map was built by one type of AI that reads sentences and tries to pull out facts. It's a bit messy, with some broken connections.
The "Robot Map B" (KGGen): This map was built by a newer, more powerful AI (a Large Language Model). It has more details but is also very fragmented and full of confusing errors.

The Magic Trick: Even though these three maps look different and have different levels of quality, the researchers forced them to share the same 1,032 key medical terms. This is like giving three different drivers the same set of 1,000 specific street names to navigate, but giving them three different maps (one perfect, two messy) to do it.

2. The Test Drive (The Evaluation)

The researchers then asked a group of "map-reading" robots (Graph Neural Networks) to perform a simple task: Classify the medical terms.

Imagine the robots are given a few labeled examples (e.g., "This is a gene," "This is a disease") and asked to guess the labels for the rest of the terms on the map.

Scenario A: They test the map-makers. They see how much the "Robot Map A" and "Robot Map B" hurt the performance compared to the "Gold Standard." If the robot gets a low score on the messy map, it tells us the map-maker did a bad job.
Scenario B: They test the map-readers. They see which robot is the toughest. Can it still guess correctly even when the map is full of holes and wrong turns?

3. The Results: Who Won?

The study found some interesting things about how these robots handle bad maps:

The "Specialist" Robots Won: Robots designed specifically to understand different types of connections (like "causes" vs. "treats") performed much better.
The "Geometric" Robots were the Toughest: The robots that used a special "geometry" approach (thinking about the map like a 3D shape rather than just a flat drawing) were the most resilient. Even when the map was very noisy and broken, these robots could still find their way.
The "Simple" Robots Struggled: Robots that just looked at neighbors without caring about the specific type of connection got confused easily when the map was messy.

Why This Matters

Before this paper, it was hard to tell if a robot was failing because it was "stupid" or because the map it was reading was "broken."

This new benchmark acts like a controlled crash test. It allows scientists to say, "Okay, we know the map is messy, so let's see which robot survives the crash best." It also helps map-makers see exactly how their errors hurt the final result.

In short: The paper provides a standardized "gym" where both map-builders and map-readers can be tested fairly, ensuring that future medical AI is built on maps that are actually useful, not just pretty.

Technical Summary: A Unified Benchmark for Evaluating Knowledge Graph Construction Methods and Graph Neural Networks

Problem Statement

Knowledge graphs (KGs) automatically constructed from unstructured text are increasingly vital for real-world applications. However, these text-derived graphs inherently suffer from noise, fragmentation, and semantic inconsistencies, which significantly degrade the performance of Graph Neural Networks (GNNs) on downstream tasks. A critical gap exists in the literature: it is often unclear whether performance limitations stem from the learning model itself or from the poor quality of the constructed graph.

Existing benchmarks fail to address this duality effectively. Early academic datasets (e.g., Citeseer, Cora) are too small and monorelational for modern GNNs. Conversely, large-scale, high-quality resources (e.g., Wikidata, UMLS) are curated by experts and lack the specific noise patterns found in automatically extracted graphs. Furthermore, no existing dataset simultaneously provides (i) a raw textual corpus, (ii) multiple automatically constructed KGs, and (iii) a high-quality reference graph. This absence prevents the systematic analysis of how extraction errors propagate and affect downstream learning, making it difficult to quantify performance degradation caused specifically by graph construction errors.

Methodology

The authors introduce a dual-purpose benchmark built on the biomedical domain, utilizing the MedMentions corpus and the UMLS (Unified Medical Language System) thesaurus. The methodology involves three core components:

1. Benchmark Construction

Reference Graph ( $G_{ref}$ ): A high-quality, expert-curated subgraph derived from the UMLS Metathesaurus, specifically filtered to the NCI Thesaurus. To ensure suitability for GNN training, the authors applied source filtering (English only), edge pruning (removing inverse edges and rare relations), and explicit integration of semantic types (adding type nodes linked via is-a relations). This graph serves as an upper performance bound.
Text Corpus: The MedMentions dataset (4,392 PubMed abstracts) was used as the source text. A normalization step was applied to replace surface-level mentions with UMLS preferred terms and align semantic types with the reference graph schema to ensure a common set of entities across all graphs.
Automatically Constructed Graphs: Two distinct graphs were generated from the same normalized corpus using different extraction pipelines:
- GT2KG: Constructed using an OpenIE pipeline followed by syntactic cleaning and LLM-based validation. This graph exhibits structural weaknesses, including weak connectivity and various edge/node-level noise (e.g., incorrect predicates, overly generic terms).
- KGGen: Constructed using a fully LLM-based pipeline (DeepSeek-Chat) involving prompt-based extraction and entity resolution. While larger in scale, it remains fragmented and suffers from open-schema noise, including self-relations and logical connectors misidentified as semantic predicates.

2. Evaluation Protocol

The benchmark supports two complementary evaluation scenarios using a standardized, reproducible pipeline based on PyTorch Geometric:

Scenario 1 (KG Construction Evaluation): New construction methods can be evaluated by generating a graph from the MedMentions corpus and measuring its performance on a downstream task compared to the reference graph. This quantifies the performance loss attributable to construction errors.
Scenario 2 (GNN Robustness Evaluation): New GNN models can be tested across the three graph variants (Reference, GT2KG, KGGen) to assess their robustness to noise and fragmentation.

The downstream task is semi-supervised node classification on 1,032 aligned nodes spanning eight semantic types. The dataset is split into training (10%), validation (10%), and test (80%) sets, stratified by class. Evaluation metrics include Accuracy, Macro F1, Precision, and Recall.

3. Baselines

The authors implemented several representative GNN architectures to establish baselines:

GCN & GAT: Standard convolution and attention-based models.
RGCN: Relation-aware model designed for multi-relational graphs.
TransGCN / RotatE-GCN: Models integrating geometric relation transformations.
All models were trained under identical protocols (Adam optimizer, fixed hyperparameter search, PLM-based initialization using all-MiniLM-L6-v2) to ensure fair comparison.

Key Results

Experimental results highlight the interplay between graph quality and model architecture:

Impact of Graph Quality: Performance dropped significantly when moving from the clean Reference graph to the noisy, automatically constructed graphs. For instance, RGCN achieved an accuracy of 0.712 on the Reference graph but dropped to 0.508 on GT2KG.
Model Robustness: Relation-aware and geometric models demonstrated superior robustness to noise.
- RGCN performed best on the clean Reference graph but was impractical on the KGGen graph due to the high number of unique relations (5,787), leading to over-parameterization and memory issues.
- Geometric Models (TransGCN, RotatE-GCN) maintained the highest performance on noisy graphs. RotatE-GCNattn achieved the best accuracy on GT2KG (0.565) and KGGen (0.594). The authors attribute this to their ability to model geometric relations and utilize bidirectional message passing, which improves information flow in sparse or fragmented graphs.
Structural vs. Semantic Noise: The results indicate that while semantic quality is important, structural properties (connectivity, component size) critically affect message propagation and model effectiveness.

Key Contributions

Dual-Purpose Benchmark: A novel framework designed to jointly evaluate (i) the impact of KG construction methods on downstream tasks and (ii) the robustness of GNNs on text-derived, noisy graphs under controlled conditions.
The Benchmark Dataset: A unique collection comprising a textual corpus, two automatically constructed KGs (GT2KG and KGGen) from different pipelines, and a high-quality reference graph (UMLS-NCI). Crucially, all graphs share a common set of annotated nodes, enabling direct, controlled comparisons.
Reproducible Evaluation Framework: A standardized pipeline providing data loaders, training protocols, and baseline implementations, facilitating the integration of new extraction methods and learning models.

Significance and Limitations

The paper claims that this benchmark fills a critical gap by providing a controlled environment to disentangle the effects of graph construction quality from model architecture performance. It enables the precise quantification of performance degradation caused by structural and semantic noise and supports the systematic study of GNN robustness.

The authors acknowledge specific limitations:

Domain Restriction: The benchmark is currently limited to the biomedical domain due to the high cost and difficulty of obtaining reliable, clean reference graphs and annotated corpora in other fields.
Task Scope: Evaluation is currently restricted to semi-supervised node classification. Extending to tasks like link prediction or knowledge graph completion remains challenging in open-schema settings where relation types vary significantly across graphs.

The work concludes by emphasizing the benchmark's extensibility, allowing for future integration of new extraction methods and deeper analysis of how structural and semantic characteristics influence learning performance.