💻 computer science

Multimodal Cultural Heritage Knowledge Graph Extension with Language and Vision Models

This paper introduces WJoconde, a multimodal French cultural heritage knowledge graph, and proposes a novel framework leveraging Large Language Models and Vision-Language Models to automatically extend such graphs with high reliability through a specialized validation pipeline.

Original authors: Yang Zhang, Nada Mimouni, Jean-Claude Moissinac, Fayçal Hamdi

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yang Zhang, Nada Mimouni, Jean-Claude Moissinac, Fayçal Hamdi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly detailed library of French art and history. Right now, this library is organized like a giant spreadsheet: it lists paintings, sculptures, and artifacts, and it tells you who made them, when, and where they are kept. This is what the authors call a Knowledge Graph. It's a structured map of facts.

However, this spreadsheet has two big problems:

It's incomplete: It's missing huge chunks of information because the real world is messy and complex.
It's boring: It only has text. It doesn't "see" the pictures of the art, even though the pictures are right there.

The authors of this paper wanted to fix this. They built a new, super-charged version of this library called WJoconde, and they invented a clever robot team to help fill in the missing blanks.

Here is how they did it, explained simply:

1. The New Library: WJoconde

Think of the original French museum database (Joconde) as a dusty archive. The authors took the best parts of this archive and connected them to Wikidata (a giant, free online encyclopedia of facts).

They created a new version called WJoconde. What makes it special?

It's Multimodal: Imagine a library card that doesn't just have a description of a painting, but also shows you the painting. WJoconde links the text description directly to the actual image of the artwork.
It's a Benchmark: They cleaned this data up and made three different versions of it (one raw, one cleaned, and one with both text and images) so other scientists can use it to test their own AI tools. It's like giving everyone a standardized "exam" to see who builds the best knowledge graph.

2. The Problem: The "Closed World" vs. The "Open World"

Most AI systems that try to fill in missing facts work like a multiple-choice test. They look at the existing list of answers and guess which one is missing.

The Problem: If the answer isn't on the list, the AI says, "I don't know."
The Reality: In art history, the "right answer" might be something nobody has ever written down before. The AI needs to be able to invent new facts, not just pick from a menu. This is called the Open World Assumption.

3. The Solution: The Robot Detective Team

To solve this, the authors built a pipeline (a workflow) using two types of super-smart AI robots:

The Reader (LLM): A Large Language Model (like a super-smart text reader) that reads the descriptions of the art.
The Viewer (VLM): A Vision-Language Model (a robot that can "see" and understand images) that looks at the paintings.

How the team works together:

The Assignment: The system takes a painting and asks, "What is happening in this picture?"
The Guess: The Reader looks at the text, and the Viewer looks at the image. They both shout out their guesses (e.g., "It shows a horse!" or "It shows a battle!").
The Double-Check (The Magic Step): This is the most important part. Since AI robots sometimes "hallucinate" (make things up), the authors built a validation pipeline.
- They check if the Reader's guess matches the Viewer's guess.
- They check if the guess is actually a new fact or just a synonym for something they already know (e.g., making sure they don't add "Curtains" if "Curtain" is already in the database).
- They even ask the Viewer to look at the image again to confirm, "Yes, I definitely see a horse in this picture."

4. The Results: A Bigger, Better Map

The results were impressive. By using this robot team:

They successfully added 61% more facts to the database.
They added thousands of new details, like specific actions (jousting), objects (swords), and scenes (cityscapes) that were previously missing.
Quality Control: When humans checked the work, they found that the robots were correct about 92% of the time.

The Bottom Line

The authors didn't just build a bigger database; they built a system that can look at a painting and its description, understand the story, and add new, accurate facts to a digital map without needing a human to type everything in.

They proved that by combining text-reading AI and image-seeing AI, and then having them check each other's work, we can automatically expand our knowledge of cultural heritage in a way that is both fast and reliable. They also made all their code and data free for anyone to use, so other researchers can try to do even better.

Technical Summary: Multimodal Cultural Heritage Knowledge Graph Extension with Language and Vision Models

Problem Statement

The preservation and interpretation of cultural heritage increasingly rely on Knowledge Graphs (KGs) to structure vast amounts of data. However, the construction and expansion of these KGs face significant challenges due to the diverse, complex, and often unstructured nature of cultural heritage information (e.g., text descriptions and images found in museum archives or Wikipedia).

Existing KGs, even large-scale ones like DBpedia or Freebase, are inherently incomplete. This incompleteness is exacerbated in the cultural heritage domain, where historical data is often hidden behind unstructured resources. Traditional Knowledge Graph Completion (KGC) methods, which are predominantly embedding-based (e.g., TransE, RESCAL), operate under the Closed World Assumption (CWA). This assumption limits predictions to predefined sets of entities, making these methods insufficient for introducing new knowledge or entities that are not already present in the graph. Furthermore, existing benchmarks often lack the operational complexity, ambiguity, and multimodal richness (text and images) characteristic of real-world cultural heritage collections.

Methodology

The authors propose a comprehensive approach involving the creation of a new dataset and a novel framework for extending KGs under the Open World Assumption (OWA).

1. The WJoconde Dataset

The authors introduce WJoconde, a new multimodal knowledge graph focused on French cultural heritage, derived from the Joconde database (a reference database of French museum collections) and Wikidata.

Construction: The pipeline involves extracting a context graph from Wikidata using entities linked to the Joconde ID (P347), cleaning redundancies, and enriching entities with external text and images from open heritage sources, Wikimedia Commons, and Wikipedia.
Variants: Three variants were created to facilitate research:
- WJoconde (Raw): Contains 28,524 entities and 483,938 triples.
- WJoconde_cleaned: A filtered version removing inverse, redundant, and sparse relations to prevent artificial performance inflation in KGC tasks.
- WJocondeMM: A full multimodal subset where entities possess both text descriptions and visual images.
Benchmarking: The dataset serves as a new benchmark for KGC, evaluated against various embedding models (TransE, ComplEx, TuckER, etc.). Results indicate that translation-based models struggle with the dataset's complexity, while deep neural models like TuckER perform best.

2. Multimodal KGC Framework under OWA

To extend the KG with new entities, the authors propose a pipeline leveraging Large Language Models (LLMs) and Vision-Language Models (VLMs) in a zero-shot manner. The process consists of three stages:

Information Collection: For a given triple with a missing entity, the system retrieves unstructured text descriptions and visual images associated with the known entities or relations.
Entity Extraction:
- LLM (e.g., LLaMA 2): Processes text inputs using prompt templates designed to define the KG task, assign a role, and enforce JSON output formatting.
- VLM (e.g., BLIP): Processes image inputs similarly to extract visual concepts.
- Both models generate lists of candidate entities without prior training on the specific dataset.
Joint Validation and Grounding: To ensure reliability and novelty, a rigorous validation pipeline is applied:
- Entity Matching: A Word2Vec model calculates similarity between generated candidates and existing KG entities. Candidates with high similarity (synonyms) or existing entities are filtered out using a threshold ( $\tau=0.4$ ).
- Cross-Validation: The VLM is used to verify if entities generated by the LLM are visually present in the image (and vice versa), reducing hallucinations.
- Model Validation: The models are tested on their ability to recognize existing entities from the provided context to establish a baseline of credibility.

Key Contributions

The paper outlines three primary contributions:

WJoconde Dataset: A novel, high-quality, multimodal knowledge graph for French cultural heritage containing over 483k triples. It integrates text and images, serving as a resource for multi-modal research and a new benchmark for KGC.
OWA Extension Pipeline: A novel framework for extending KGs under the Open World Assumption. By combining LLMs and VLMs with a specialized grounding and validation pipeline, the method successfully introduces new, reliable entities to the graph without requiring model fine-tuning.
Benchmarking and Evaluation: Extensive evaluation of existing KGC models on WJoconde and its variants, alongside a new evaluation methodology for OWA scenarios that relies on human evaluation and cross-model validation rather than traditional ranking metrics.

Results

The proposed pipeline was applied to the WJocondeMM_en dataset:

Entity Expansion: The method successfully introduced 61.36% more entities to the dataset. Specifically, 4,394 new final entities were added, expanding the entity count from 7,749 to 12,143.
Reliability:
- Model Credibility: The combined LLM and VLM pipeline achieved a 90.2% recall and 94.8% F1 score in recovering ground truth entities (existing entities), demonstrating high reliability in entity recognition.
- Human Evaluation: Two annotators reviewed generated entities for the "depicts" relation. The combined approach achieved a 92.0% precision, indicating that the majority of newly added entities were correct and meaningful.
Semantic Quality: Analysis of the added entities showed a shift from "unknown" categories (55.4% in the original) to semantically rich, visually grounded categories such as "Object" (22.5%), "Nature" (17.5%), and "Person" (15.6%). The added entities captured interpretable concepts (e.g., "jousting," "arch," "armor") relevant to art historians.

Significance

The paper claims that this work represents the first creation of a comprehensive multimodal benchmarking dataset in the French cultural heritage domain that combines textual and visual information. Its significance lies in:

Advancing OWA in Cultural Heritage: Demonstrating that multimodal AI models can effectively extend KGs with new knowledge, overcoming the limitations of the Closed World Assumption which restricts traditional KGC methods.
Domain-Specific Utility: Providing a structured, multimodal resource that facilitates sophisticated querying, analytics, and research in cultural heritage, addressing the complexity of managing extensive museum collections.
Methodological Robustness: Establishing a validation pipeline that ensures the reliability of AI-generated knowledge, making it feasible to automate the enrichment of cultural heritage data with high confidence.

The authors note that while the pipeline is effective, future work could explore advanced models, few-shot learning, and more sophisticated fine-tuning scenarios. All code, datasets, and interactive access points are open-sourced to facilitate further research.