💬 NLP

ReaLM: Residual Quantization Bridging Knowledge Graph Embeddings and Large Language Models

ReaLM is a novel framework that bridges Knowledge Graph Embeddings and Large Language Models by discretizing continuous KG embeddings into learnable tokens via residual vector quantization and incorporating ontology-guided constraints, thereby achieving state-of-the-art performance in Knowledge Graph Completion.

Original authors: Wenbin Guo, Xin Wang, Jiaoyan Chen, Lingbing Guo, Zhao Li, Zirui Chen

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wenbin Guo, Xin Wang, Jiaoyan Chen, Lingbing Guo, Zhao Li, Zirui Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Two Languages That Don't Mix

Imagine you have two brilliant experts trying to solve a puzzle together, but they speak completely different languages.

Expert A (The Knowledge Graph): This expert knows the world as a massive, structured map of facts. They think in precise coordinates and numbers. If they know "Iron Man is married to Pepper Potts," they store this as a specific mathematical connection between two points.
Expert B (The Large Language Model): This expert is a master storyteller who speaks human language. They understand context, nuance, and how words flow together. However, they don't naturally understand the precise mathematical coordinates of Expert A's map.

The Conflict: When you ask the Storyteller (LLM) to guess a missing fact from the Map (Knowledge Graph), they often get confused.

If you show them the map as text, they might guess a word that sounds right (like "Pepper") but isn't the exact entity in the database.
If you try to force them to learn every single name in the map (there are tens of thousands), their brain gets too full, they get slow, and they start making mistakes.

The Solution: ReaLM (The Universal Translator)

The authors created a system called ReaLM to bridge this gap. Think of it as a translation device that turns the Map's complex coordinates into a simple, short code that the Storyteller can read and write perfectly.

Here is how it works, step-by-step:

1. Turning Maps into "Zip Codes" (Residual Vector Quantization)

Instead of trying to teach the Storyteller the entire map, ReaLM takes the complex mathematical "address" of every entity (like Iron Man) and compresses it into a short, unique sequence of numbers.

The Analogy: Imagine you have a giant library with millions of books. Instead of giving the librarian the whole book to read, you give them a short barcode (e.g., 7-48-109-586).
How it works: The system breaks the complex math down into layers (like peeling an onion). It finds the closest match in a "dictionary of codes" for the first layer, calculates what's missing, finds a match for the second layer, and so on. The result is a compact list of numbers that perfectly represents the original complex idea.
The Benefit: The Storyteller (LLM) doesn't need to memorize millions of names. It just needs to learn a manageable set of these "barcodes."

2. Teaching the Storyteller the New Codes

Once the entities are converted into these number codes, ReaLM adds them to the Storyteller's vocabulary.

The Analogy: Imagine the Storyteller's dictionary suddenly gets a new section called "Quantized Codes." Now, when they see the code 7-48-109, they know exactly which character it refers to, without needing to guess the spelling.
The Benefit: The Storyteller can now generate the exact correct answer (the specific entity) rather than a vague guess.

3. The "Rule Book" Check (Ontology-Guided Constraints)

Sometimes, even with the codes, the Storyteller might make a logical error. For example, they might guess that a "Car" is the wife of a "Person."

The Analogy: ReaLM brings in a Rule Book (Ontology). Before the Storyteller gives their final answer, the Rule Book checks: "Wait, the question asks for a 'wife'. The answer must be a 'Person', not a 'Car'."
How it works: The system predicts the category (class) of the answer first. If the Storyteller guesses an entity that doesn't fit the category, the system filters it out and picks the next best option that does fit.
The Benefit: This ensures the answers aren't just mathematically close, but also logically and semantically correct.

What Did They Find?

The researchers tested this system on two major "puzzle books" (datasets) containing facts about movies and words.

The Result: ReaLM became the champion of these puzzles. It outperformed all previous methods, including other AI models that tried to combine maps and language.
Why it won: By turning complex math into simple codes, the Storyteller could finally "see" the map clearly. By adding the Rule Book check, it stopped making silly logical mistakes.

Summary

ReaLM is like a translator that turns a complex, high-tech map into a simple set of barcodes that a language expert can read. It then adds a "logic check" to ensure the answers make sense. This allows Artificial Intelligence to combine the best of both worlds: the precise structure of a knowledge database and the powerful reasoning of a language model.

Technical Summary of ReaLM: Residual Quantization Bridging Knowledge Graph Embeddings and Large Language Models

Problem Statement
While Large Language Models (LLMs) have demonstrated strong reasoning capabilities for Knowledge Graph Completion (KGC), a fundamental misalignment exists between the continuous embedding spaces of Knowledge Graph (KG) models and the discrete token spaces of LLMs. Traditional approaches that convert KG triples into natural language often generate outputs that are semantically related but do not correspond to exact entities defined in the KG, leading to mapping inconsistencies. Conversely, directly expanding the LLM vocabulary to include all KG entities is computationally prohibitive due to the vast and evolving nature of entity sets (e.g., WN18RR contains over 40,000 entities, exceeding the 32,000-token vocabulary of LLaMA-2). Existing methods attempting to bridge this gap, such as direct projection or partial integration, often suffer from training instability, inability to support ranking tasks, or a failure to capture rich structural semantics.

Methodology
The proposed framework, ReaLM, addresses these challenges through a multi-stage process that discretizes KG embeddings into compact code sequences compatible with LLM tokenization, while enforcing semantic consistency via ontology.

Semantic Extraction: The framework utilizes RotatE to pretrain high-quality entity and relation embeddings in a complex vector space. RotatE is selected for its superior ability to capture relational patterns compared to other embedding methods like TransE or DistMult.
Residual Vector Quantization (RVQ): To bridge the continuous-discrete gap, ReaLM employs Residual Vector Quantization. Pretrained complex embeddings are mapped to real-valued vectors and approximated through a sequence of $S$ stages (empirically set to 32). In each stage, a residual vector is quantized by selecting the nearest codeword from an independent codebook. This process transforms a high-dimensional continuous embedding into a compact discrete sequence of $S$ code indices. The quantization is optimized using a loss function combining reconstruction loss, codebook loss, and a commitment loss to prevent codebook collapse.
LLM Integration and Fine-tuning: The discrete code indices are treated as new tokens and added to the LLM vocabulary. The embeddings for these new tokens are initialized by summing the vectors of the corresponding codewords across all quantization stages. The model is then fine-tuned using a dual adaptation strategy:
- Token Embedding Optimization: The embeddings of the newly added quantized tokens are trained via standard gradient descent.
- Parameter-Efficient Adaptation: The internal parameters of the LLM are adapted using Low-Rank Adaptation (LoRA), allowing the model to internalize the discrete token format without retraining the entire network.
Ontology-Guided Class Constraints: To ensure semantic correctness, ReaLM integrates ontology knowledge. Classes are treated similarly to entities: they are embedded, quantized, and added to the vocabulary. During inference, the model predicts both the missing entity and its corresponding class. If the predicted entity does not belong to the predicted class, the prediction is refined by selecting an alternative candidate that satisfies the class constraint.

Key Contributions

Residual Vector Quantization Framework: ReaLM introduces a method to transform high-dimensional continuous KG embeddings into compact discrete code sequences. This enables seamless integration with LLM token spaces while preserving rich relational and semantic information, avoiding the need for uncontrolled vocabulary expansion.
Ontology-Guided Class Constraints: The framework incorporates class-level semantic constraints to enforce consistency between predicted entities and their corresponding classes, addressing the issue of semantically plausible but class-inconsistent predictions.
Empirical Validation: The approach is validated on standard benchmark datasets (FB15k-237 and WN18RR), demonstrating significant improvements over both traditional embedding-based models and existing LLM-based KGC methods.

Experimental Results
Extensive experiments were conducted on link prediction and triple classification tasks:

Link Prediction: On the WN18RR dataset, ReaLM achieved a Mean Reciprocal Rank (MRR) of 0.608 and Hits@1 of 0.575, outperforming the strongest LLM-based baseline (SSQR-LLaMA3.1) by 1.6–3.5%. On FB15k237, it achieved an MRR of 0.467, surpassing all baselines.
Triple Classification: On the FB15k-237N dataset, ReaLM reached an accuracy of 0.814 and an F1-score of 0.825, improving upon the SSQR-LLaMA3.1 baseline by 1.4 percentage points in F1-score.
Ablation Studies: Removing the ontology-guided component resulted in consistent performance degradation across all metrics (e.g., a 3.38% drop in F1-score for triple classification), confirming the critical role of class constraints in filtering type-incoherent predictions.
Parameter Analysis: Experiments indicated that a codebook size ( $K$ ) of 1000–1500 and 32 quantization stages ( $S$ ) offer the optimal trade-off between representational capacity and computational efficiency.

Significance
The paper claims that ReaLM effectively bridges the gap between structured knowledge and generative language models. By discretizing embeddings into quantized codes, ReaLM allows LLMs to internalize structured KG semantics without sacrificing linguistic fluency or requiring massive vocabulary expansions. The integration of ontology-guided constraints further ensures that the model's predictions remain semantically consistent with the underlying knowledge graph structure. The authors conclude that ReaLM represents a state-of-the-art approach for KGC, offering a robust solution to the challenges of semantic misalignment and vocabulary scalability in LLM-based knowledge reasoning.