🤖 machine learning

PROBE-Web: An Interactive System for Probing Evaluation Landscapes of Knowledge Graph Completion Models

This paper introduces PROBE-Web, an interactive system that enables users to flexibly probe and evaluate Knowledge Graph Completion models across diverse perspectives, specifically focusing on predictive sharpness and popularity-bias robustness, through a user-friendly interface offering conventional toolkits, perspective-aware analysis, explainable case studies, and landscape exploration.

Original authors: Sooho Moon, Yunyong Ko

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sooho Moon, Yunyong Ko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a chef. You have a list of dishes (facts) you want them to cook. To judge who is the best chef, you usually just look at a single score: "How many dishes did they get right?" and "How fast did they get them?" This is how most people currently judge Knowledge Graph Completion (KGC) models—computer programs that try to guess missing facts in a giant web of information.

However, the paper argues that this "one-size-fits-all" score is misleading. Just like different people have different tastes, different users need different things from these models.

User A (The Doctor): Needs a model that is extremely confident. If the model guesses wrong, it could be dangerous. They want "sharp" predictions where mistakes are heavily punished.
User B (The Explorer): Wants to find new, rare connections. They don't care about the famous, well-known facts; they want the model to dig up obscure, low-popularity entities.

The paper introduces PROBEWeb, a new interactive tool that lets you stop using a single ruler and instead use a "multi-dimensional map" to find the perfect model for your specific needs.

The Two Dials of PROBEWeb

Think of PROBEWeb as a sound mixing board with two main knobs that change how you judge the chefs:

The "Sharpness" Knob (Predictive Sharpness):
- Low Sharpness: You are lenient. If the chef gets the dish right but it's the 10th best option, you still give them a decent score. You care about general accuracy.
- High Sharpness: You are strict. If the chef doesn't put the correct dish at the very top (Rank 1), you give them a zero. You only care about absolute confidence.
- Analogy: It's the difference between a teacher who gives you a B for getting the answer mostly right, versus a teacher who only gives you an A if you are 100% perfect.
The "Popularity" Knob (Popularity-Bias Robustness):
- Low Robustness: You ignore the "famous" facts. If the model guesses a connection between two very popular people (like "Elvis" and "USA"), you don't give them extra credit because that's easy to guess.
- High Robustness: You reward the model for guessing rare things. If it connects two obscure entities that no one usually talks about, you give them a huge bonus.
- Analogy: It's the difference between a trivia game where you get points for knowing "Who is the President?" (easy, popular) versus points for knowing "Who was the mayor of a tiny village in 1920?" (hard, rare).

What PROBEWeb Actually Does

The system offers four main features to help you navigate this landscape:

The Standard Toolkit: It comes pre-loaded with the "old school" scores (like MRR and Hits@K) so you can see how models look under traditional rules. It's like checking the standard report card.
The Interactive Mixer: You can slide those two knobs (Sharpness and Popularity) back and forth. As you move them, the ranking of the models changes in real-time. You might see that Model A is the best when you want "High Sharpness," but Model B takes the lead when you want "High Popularity Robustness."
The "Why" Detective: If you wonder why a model's ranking dropped when you turned a knob, PROBEWeb shows you case studies. It breaks down the data to show, for example, "Model A failed because it kept guessing the top 5 popular entities, while Model B found the rare ones."
The 3D Landscape Map: Instead of a flat list, the system builds a 3D mountain range. The X and Y axes are your two knobs, and the height of the mountain is the model's score. You can see that Model A has a high peak in one corner of the map, while Model B has a high peak in a different corner. This helps you see exactly where each model shines and where it falls flat.

The Bottom Line

The paper claims that PROBEWeb doesn't just tell you which model is "best." Instead, it helps you understand which model is best for your specific goal. It moves the conversation from "Who won?" to "Who wins when we care about X and Y?" By visualizing these different "evaluation landscapes," users can stop picking suboptimal models and start choosing the one that fits their specific needs, whether that's high confidence for medical safety or high creativity for discovering new knowledge.

Technical Summary: PROBEWeb

Problem Statement

Knowledge Graph Completion (KGC) models are predominantly evaluated using fixed, rank-based metrics such as Mean Reciprocal Rank (MRR) and Hits@K. However, the definition of a "good" KGC model varies significantly depending on the user and the target application. For instance, practitioners in drug discovery may prioritize models with high confidence to avoid costly trial failures, whereas researchers may value models capable of discovering novel knowledge regarding rare entities over well-known facts. Existing evaluation protocols fail to accommodate these diverse perspectives, often leading users to select suboptimal models that do not align with their specific objectives.

Methodology

The paper introduces PROBEWeb, an interactive web-based system designed to probe the evaluation landscapes of KGC models. The system is built upon the PROBE framework, which generalizes the standard rank-based evaluation protocol through three stages: prediction, transformation, and aggregation. PROBEWeb allows users to flexibly adjust two critical evaluation perspectives:

Predictive Sharpness (P1): Controlled by a factor $\alpha$ , this perspective determines how strongly inaccurate predictions are penalized. The system employs a Rank Transformation (RT) function that converts predicted ranks into scores. Higher values of $\alpha$ assign substantially lower scores to incorrect predictions, enforcing a stricter evaluation, while lower values result in a more generous evaluation.
Popularity-Bias Robustness (P2): Controlled by a factor $\beta$ , this perspective addresses the bias toward high-popularity entities. The system utilizes a Rank Aggregation (RA) function that assigns weights to predictions based on the popularity of the target entity in the training set. Higher values of $\beta$ assign lower weights to high-popularity entities, thereby increasing the contribution of low-popularity entities to the final score.

PROBEWeb accepts user-uploaded prediction results (rank scores for test queries) and Knowledge Graph (KG) information (entity/relation dictionaries and training triples) in a predefined format. It then computes evaluation scores by applying the RT and RA functions with user-specified $\alpha$ and $\beta$ parameters.

Key Contributions

The paper presents PROBEWeb as a system with four primary functionalities:

Conventional Evaluation Toolkit: Provides a standard benchmark view using widely adopted metrics (MRR, Hits@1, Hits@3) across six pre-computed state-of-the-art KGC models (RotatE, ComplEx, HousE, TuckEr, pLogicNet, RNNLogic) and three popular KGs (FB15k-237, WN18RR, YAGO3-10). This allows immediate comparison without requiring user data.
Flexible Perspective-Aware Evaluation: Enables users to interactively adjust $\alpha$ and $\beta$ via a graphical user interface (GUI). The system recomputes evaluation scores and updates model rankings in real-time, allowing users to observe how relative model performance shifts under different evaluation perspectives.
Explainable Evaluation via Case Studies: Offers diagnostic tools to explain ranking changes.
- For P1, it analyzes prediction quality across different rank groups (e.g., 1st, 2nd–5th, 6th–20th) to determine if a model excels at top-rank predictions or maintains consistent moderate accuracy.
- For P2, it provides query-level case studies showing the popularity of specific queries and the ranks assigned by different models, helping users understand model behavior on low-popularity entities.
Evaluation Landscape Exploration: Visualizes model performance in a 3-D space where the $x$ and $y$ axes represent $\alpha$ and $\beta$ , and the $z$ axis represents the normalized evaluation score. This creates a "performance surface" for each model, allowing users to identify regions where specific models dominate. The system also partitions this space into four representative quadrants for fine-grained comparison.

Results and System Features

The paper does not present new experimental results regarding model performance improvements but rather demonstrates the system's capabilities through its interface and pre-computed benchmarks. Key system features highlighted include:

Accessibility: A user-friendly GUI that requires no additional programming effort.
Portability: Independence from underlying KGC model architectures, training frameworks, or execution environments, provided inputs follow the predefined format.
Interactivity: Real-time updates of rankings and visualizations as users modify evaluation parameters.

Significance and Claims

The authors claim that PROBEWeb bridges the gap between diverse user evaluation perspectives and existing fixed evaluation protocols. By allowing users to probe diverse evaluation landscapes, the system helps users not only identify which KGC model performs best but also understand why and under what specific evaluation perspectives a model should be selected. The system aims to facilitate better model selection for downstream applications such as question answering, recommender systems, and drug discovery by aligning evaluation metrics with specific application requirements. The source code and a demonstration video are made publicly available to support reproducibility and further exploration.