Embedding Ontologies via Incorporating Extensional and Intensional Knowledge

This paper proposes EIKE, a novel ontology embedding framework that simultaneously captures extensional knowledge via geometry-based modeling and intensional knowledge via pretrained language models, significantly outperforming state-of-the-art methods in triple classification and link prediction tasks.

The Gist

Imagine you are trying to teach a robot how to understand the world. You give it a giant encyclopedia (an Ontology) that contains two very different types of information:

1. The "Who's Who" (Extensional Knowledge): This is the list of specific people, places, and things. For example, "Fido is a dog," "John is a human," and "Fido belongs to John." It's about concrete instances.
2. The "Rulebook" (Intensional Knowledge): This is the abstract logic and definitions. It says, "All dogs are animals," "Animals have fur," and "Humans are smarter than dogs." It's about concepts and their inherent properties.

The Problem

Existing methods for teaching robots to understand this encyclopedia were like trying to describe a complex city using only a flat map, or only a list of street names.

• Some methods were great at mapping the concrete instances (the "Who's Who") but missed the deep meaning of the rules.
• Others were great at understanding the abstract rules but got confused when trying to place specific people or things into the picture.

They tried to force both types of knowledge into a single "box," and the result was a messy, confused understanding.

The Solution: EIKE (The Two-Room House)

The authors of this paper propose a new method called EIKE (Extensional and Intensional Knowledge Embedding). Instead of one messy room, they build a two-room house for the robot's brain.

Room 1: The "Geometric Playground" (Extensional Space)

Think of this room as a 3D playground with shapes.

• Concepts are shapes: A concept like "Dog" isn't just a word; it's a giant, stretchy bubble (an ellipsoid).
• Instances are dots: A specific dog named "Fido" is a dot inside that bubble.
• The Logic: If Fido is a dog, his dot must be inside the Dog bubble. If "Dog" is a type of "Animal," the Dog bubble must be inside the Animal bubble.
• Why it works: This is perfect for visualizing who belongs to what group. It's like a Venn diagram that the robot can navigate physically.

Room 2: The "Library of Meanings" (Intensional Space)

Think of this room as a high-tech library where every book is a concept.

• The Tool: The authors use a Pre-trained Language Model (like a super-smart AI that has read the entire internet).
• How it works: Instead of just looking at shapes, the AI reads the description of a concept. If the concept is "Danish Male Film Actors," the AI understands the meaning of those words. It creates a "fingerprint" (a vector) based on the text.
• The Logic: This room captures the vibe and characteristics of a concept. It knows that "Dog" and "Puppy" are similar because the words are related, even if the geometric shapes in Room 1 are far apart.

The Magic Bridge

Here is the genius part: EIKE connects these two rooms.

• It takes the "dot" (Fido) from the Geometric Playground and translates it into a "fingerprint" in the Library.
• It takes the "bubble" (Dog) from the Playground and matches it with the "book" (Dog) in the Library.

By training the robot to look at both the shape (where Fido lives) and the meaning (what a Dog is), the robot gets a much clearer picture of the world.

The Results: A Smarter Robot

The researchers tested this on three massive datasets (like giant encyclopedias of real-world data).

• The Test: They asked the robot to guess missing facts (e.g., "If Fido is a dog, and all dogs are animals, is Fido an animal?") or to spot fake facts.
• The Outcome: EIKE crushed the competition. It was significantly better at guessing the right answers than previous methods.

The Analogy Summary

Imagine you are trying to find a specific book in a library.

• Old Methods: You either look at the color of the spine (Geometry/Instances) OR you read the title and summary (Text/Concepts). You might get lost.
• EIKE: You look at the color of the spine to find the right shelf AND you read the summary to confirm it's the right book. You find what you need instantly.

Why This Matters

This approach is a big step forward because it respects the fact that knowledge comes in two flavors: specific examples and general rules. By giving each flavor its own dedicated space and then linking them, we create AI that understands the world more like a human does—seeing both the trees and the forest at the same time.

Technical Summary

1. Problem Statement

Ontologies contain two distinct types of knowledge:

• Extensional Knowledge (Assertional): Describes concrete instances, their types, and relationships (e.g., "John is a Person," "John hasPet Fido").
• Intensional Knowledge (Terminological): Describes the inherent properties, characteristics, and semantic associations of concepts (e.g., "A Person is a subclass of Animal," definitions of concepts).

The Gap: Existing ontology embedding methods fail to simultaneously model both types of knowledge effectively.

• Geometry-based methods (e.g., TransC, TransEllipsoid) excel at modeling extensional knowledge and structural hierarchies but ignore the textual/lexical characteristics of concepts, failing to capture intensional knowledge.
• Text-based/Random Walk methods (e.g., Concept2Vec, OWL2Vec*) capture textual semantics but often struggle to distinguish clearly between concepts and instances or fail to model the geometric containment required for extensional knowledge (e.g., an instance belonging to a concept).

The authors propose that a unified framework is needed to represent ontologies by capturing both structural (geometric) and textual (semantic) information simultaneously.

2. Methodology: The EIKE Framework

The authors propose EIKE (Extensional and Intensional Knowledge Embedding), which represents ontologies in two distinct but jointly trained spaces: Extensional Space and Intensional Space.

A. Knowledge Representation

1. Extensional Space (Geometric Modeling):

   • Concepts: Represented as ellipsoidal regions $G(c, b)$ in a vector space. This captures the anisotropy of concept embeddings.
   • Instances: Represented as point vectors.
   • Logic: An instance is considered part of a concept if its point vector lies within the concept's ellipsoid region. This space primarily models InstanceOf and SubClassOf relations based on set inclusion.

2. Intensional Space (Semantic Modeling):

   • Concepts: Represented as point vectors generated by a Pre-trained Language Model (PLM) (specifically Sentence-BERT). The input is the concept name or the concatenation of the name and its description. This captures inherent properties and textual semantics.
   • Instances: Represented as virtual instance vectors. Since PLMs do not natively encode specific instances in the same way as concepts, instance vectors in this space are derived by mapping the extensional instance vectors through a transformation matrix ($M_{ei}$).

B. Scoring Functions & Loss

The model defines specific scoring functions and loss terms for three types of triples, combining signals from both spaces:

1. InstanceOf Triples $(i, r_e, c)$:

   • Extensional Loss: Measures the distance of the instance point from the center of the concept ellipsoid (must be inside).
   • Intensional Loss: Measures the cosine similarity between the virtual instance vector and the concept vector in the intensional space.
   • Total Loss: A weighted sum of both.

2. SubClassOf Triples $(c_i, r_c, c_j)$:

   • Extensional Loss: Models the geometric containment where the ellipsoid of the subclass ($c_i$) is a sub-region of the superclass ($c_j$).
   • Intensional Loss: Enforces high similarity between the concept vectors and ensures the superclass vector has a larger magnitude (representing broader, more abstract concepts).

3. Relational Triples $(h, r, t)$:

   • Modeled only in Extensional Space using the standard TransE translation-based scoring function ($\|h + r - t\|^2$), as these relations typically involve instances.

C. Training

The model is trained using a margin-based ranking loss for all three triple types simultaneously. The objective is to minimize the overall loss function $L = L_{sub} + L_{ins} + L_{rel}$, ensuring that positive triples score better than negative (corrupted) triples.

3. Key Contributions

1. Dual-Space Framework: First to propose embedding extensional and intensional knowledge into two distinct, specialized spaces (geometric for extensional, PLM-based for intensional) and jointly learning them.
2. Hybrid Representation: Successfully integrates geometric region modeling (for instance-concept containment) with pre-trained language model embeddings (for concept semantics), addressing the limitations of purely structural or purely textual approaches.
3. Virtual Instance Mapping: Introduces a mechanism to map extensional instance vectors into the intensional space via a transformation matrix, enabling joint training despite the lack of native textual descriptions for instances.
4. Comprehensive Evaluation: Demonstrates that capturing both knowledge types leads to superior performance in both structural reasoning (SubClassOf) and instance classification.

4. Experimental Results

The authors evaluated EIKE on three datasets: YAGO39K, M-YAGO39K (transitivity-enhanced), and DB99K-242. They compared against 13 state-of-the-art baselines (including TransE, TransC, TransEllipsoid, Concept2Vec, etc.).

• Triple Classification:

  • EIKE significantly outperformed all baselines in InstanceOf and SubClassOf triple classification across all datasets.
  • The variant EIKE-PRE-EYE (unif) (using a pre-trained model and identity mapping) achieved the best results, confirming the value of PLMs for intensional knowledge.
  • On Relational Triple classification, EIKE achieved competitive results comparable to the best existing methods (e.g., ~93.4% F1-score on YAGO39K).

• Link Prediction:

  • EIKE showed significant improvements in Hits@10, Hits@1, and MRR for relational triples, indicating that the dual-space representation benefits instance-level embeddings.

• Ablation Studies:

  • Using a Pre-trained Model (PRE) for intensional encoding yielded better results than random initialization (UNP), especially on datasets with rich descriptions (YAGO).
  • Using an Identity Matrix (EYE) for mapping instances to the intensional space generally performed better than a learnable normal matrix (MAT), suggesting a fixed mapping is sufficient for bridging the spaces.

5. Significance

The paper demonstrates that separating but jointly learning extensional and intensional knowledge is crucial for high-quality ontology embedding.

• Comprehensiveness: It provides a more representative view of the domain by capturing both the "what" (instances/structure) and the "why/definition" (concepts/semantics).
• Performance: The approach sets a new state-of-the-art for ontology embedding tasks, particularly in scenarios requiring accurate classification of instances and hierarchical reasoning.
• Future Potential: The framework opens avenues for incorporating more complex relations (e.g., Domain/Range) and exploring Graph Neural Networks (GNNs) for even more expressive extensional modeling.