Improving Continual Learning of Knowledge Graph Embeddings via Informed Initialization

Technical Summary: Improving Continual Learning of Knowledge Graph Embeddings via Informed Initialization

Problem Statement
Knowledge Graphs (KGs) are dynamic structures that frequently evolve through the addition of new triples. In scenarios such as recommender systems, these updates are often small and frequent, necessitating that Knowledge Graph Embeddings (KGEs) adapt without requiring full retraining. Continual learning (CL) techniques address this by learning embeddings for new entities while updating existing ones. However, a critical bottleneck in current CL methods is the initialization of new embeddings.

Standard practice, inherited from static training, involves randomly initializing new entity embeddings (e.g., via Xavier or He initialization). This approach presents two significant issues:

1. Training Efficiency: Random initialization places new embeddings far from their optimal positions, requiring more training epochs and time to converge.
2. Catastrophic Forgetting: When new triples include previously existing entities, the large training loss caused by poorly initialized new embeddings forces significant adjustments to the entire triple during optimization. This disrupts previously learned representations, leading to the loss of old knowledge (catastrophic forgetting).

Existing CL-specific initialization methods (e.g., in LKGE or incDE) attempt to infer new positions based on the geometric constraints of specific translational models (like TransE). However, these approaches are often model-specific, relying on invertible scoring functions that do not exist for many modern KGE models (e.g., semantic matching models or neural network-based models like ConvE), rendering them inapplicable to a broad range of architectures.

Methodology
The authors propose a model-agnostic, schema-based informed initialization strategy for new entity embeddings in continual learning scenarios. The method leverages the KG schema (entity classes) and the distribution of previously learned embeddings to generate initial representations.

The core algorithm proceeds as follows:

1. Class Centroids: For each entity class $c$ defined in the KG schema, a centroid embedding $v_c$ is computed as the mean of the embeddings of all existing entities belonging to that class.
2. Intra-class Variation: The standard deviation $\sigma_c$ of embeddings within each class is calculated to capture intra-class diversity.
3. Initialization: For a new entity $e$ belonging to a set of classes $C_e$, its initial embedding is calculated as the average of the centroids of its classes, augmented with a stochastic perturbation:
   $$\mathbf{e} = \frac{1}{|C_e|} \sum_{c \in C_e} (\mathbf{v}_c + \gamma \cdot \boldsymbol{\sigma}_c \odot \mathbf{r}_c)$$
   Where $\gamma$ is a hyperparameter controlling the perturbation magnitude, $\mathbf{r}_c$ is a random vector, and $\odot$ denotes element-wise multiplication.

This approach ensures that new entities start with latent semantic information derived from their class memberships, positioning them closer to their likely final representations. The stochastic component prevents degenerate cases where entities with identical class memberships receive identical embeddings. Once initialized, any standard continual learning technique (e.g., fine-tuning, EWC, rehearsal) can be applied to update the embeddings using the new triples.

Key Contributions

1. Informed Initialization Strategy: A novel method for initializing KGEs in continual learning that utilizes the KG schema and existing embeddings, applicable across diverse KGE models without requiring architectural modifications.
2. Empirical Validation of Efficiency and Performance: Demonstration that the proposed strategy significantly reduces training time (epochs) while improving predictive performance compared to random and model-specific initialization baselines.
3. New Evaluation Metrics: The introduction of adapted metrics ($\Omega_{base}$ and $\Omega_{new}$) to quantify knowledge retention and acquisition in KGE continual learning, derived from incremental task learning in neural networks.
4. Comprehensive Evaluation: Extensive experiments evaluating the method across various continual learning algorithms (Fine-tuning, EWC, EMR, LKGE, incDE) and KGE model families (Translational and Semantic Matching).

Experimental Results
The authors evaluated their approach on three datasets (FBinc-S, FBinc-M, FBinc-L) derived from FB15K-237, featuring incremental updates of varying sizes. Key findings include:

• Performance Gains: The schema-based initialization consistently outperformed random initialization and, in many cases, model-specific initialization. It improved both knowledge retention ($\Omega_{base}$) and acquisition ($\Omega_{new}$). For instance, in non-regularized methods like Fine-tuning and EMR, schema initialization increased knowledge retention by up to 30% and new knowledge acquisition by up to 65% on smaller datasets.
• Training Acceleration: The informed initialization significantly reduced the number of epochs required to reach convergence. Compared to random initialization, the time to convergence was reduced by a factor between 2.16 and 2.67 across most methods.
• Model Agnosticism: The method proved effective across both translational models (TransH, RotatE, TransR) and semantic matching models (DistMult, HolE, ProjE). In models prone to catastrophic forgetting under random initialization (e.g., RotatE, TransR), the schema initialization mitigated these effects.
• Computational Cost: The initialization process itself is computationally negligible (orders of magnitude faster than a single training epoch) and scales efficiently, as it only requires maintaining class centroids and standard deviations between updates.

Significance and Claims
The paper claims that informed initialization is a fundamental, yet often overlooked, step in continual learning for KGEs. By providing a semantically grounded starting point, the method:

• Accelerates Knowledge Acquisition: New information is learned faster, reducing the computational cost of incremental updates.
• Mitigates Catastrophic Forgetting: By minimizing the initial distance to optimal positions, the method reduces the disruptive adjustments required for existing embeddings during the optimization of new triples.
• Enhances Generalizability: Unlike previous CL-specific methods, this approach is not tied to the geometric constraints of specific KGE models, making it a versatile plug-in for existing continual learning pipelines.

The authors conclude that this strategy improves the predictive performance of downstream tasks and enhances the efficiency of continual learning, making it a valuable addition to the toolkit for managing dynamic Knowledge Graphs.