💻 computer science

Knowledge Graphs Generation from Cultural Heritage Texts: Combining LLMs and Ontological Engineering for Scholarly Debates

This paper introduces ATR4CH, a systematic five-step methodology that combines Large Language Models with ontological engineering to automatically convert unstructured Cultural Heritage texts into structured, queryable Knowledge Graphs, demonstrating high extraction accuracy and cost-effective deployment through a case study on authenticity assessment debates.

Original authors: Andrea Schimmenti, Valentina Pasqual, Fabio Vitali, Marieke van Erp

Published 2026-04-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Andrea Schimmenti, Valentina Pasqual, Fabio Vitali, Marieke van Erp

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library filled with thousands of dusty, complex books about history, art, and ancient artifacts. These books are full of fascinating stories, but they are written in a messy, human way. They contain arguments, disagreements, and "maybe" statements. For example, one page might say, "This ancient vase looks real," while another page argues, "No, the paint is wrong; it's a fake from the 1800s."

Currently, if you ask a computer database (like a digital library catalog) about this vase, it usually gives you a simple, boring answer like: "Status: Fake." It strips away the entire story, the evidence, and the debate. It's like summarizing a heated courtroom trial by just saying, "Guilty," without telling you why or who said what.

This paper introduces a new way to fix that. The authors created a system called ATR4CH (which sounds like a robot butler for history books) that uses AI (specifically Large Language Models) to read these messy books and turn them into a structured, searchable "Knowledge Graph."

Here is how they did it, explained with some everyday analogies:

1. The Problem: The "Flat" Map vs. The "3D" City

Think of current digital libraries as flat paper maps. They show you where things are (the title, the date, the author), but they miss the terrain. They don't show you the hills (the strong evidence), the valleys (the weak arguments), or the fact that two people are arguing about the same street corner.

The authors wanted to build a 3D interactive model of the city. They wanted to capture not just what happened, but who said it, why they said it, and what evidence they used.

2. The Solution: The "Five-Step Recipe" (ATR4CH)

Instead of just throwing a computer at the text and hoping for the best, the authors cooked up a five-step recipe to make sure the AI didn't get confused:

Step 1: The Blueprint (Foundational Analysis): Before building, they looked at the "ingredients" (the text) and the "blueprint" (the rules of how knowledge should be organized). They asked: "What are we actually looking for?"
Step 2: The Training Wheels (Annotation): They taught a human team how to highlight the important parts of the text (like who is arguing and what their evidence is). This created a "gold standard" to train the AI.
Step 3: The Assembly Line (Pipeline): They built a machine with three stations.
- Station 1: Find the objects (e.g., "The Donation of Constantine").
- Station 2: Find the people arguing (e.g., "Lorenzo Valla").
- Station 3: Connect the dots (e.g., "Valla says the object is a fake because the language is wrong").
Step 4: The Quality Control (Refinement): They ran the machine, checked the results, and tweaked the settings to make sure it wasn't making things up.
Step 5: The Final Exam (Evaluation): They tested the system against human experts to see if it got the story right.

3. The "Brain" of the Operation: The AI Models

They tested three different AI "brains" to see which one was best at this task:

The Giant Brain (Claude Sonnet 3.7): Very smart, very careful, but expensive to run.
The Medium Brain (Llama 3.3 70B): A good balance of smarts and speed.
The Tiny Brain (GPT-4o-mini): Surprisingly good! It's smaller and cheaper, but it did a great job finding the people and arguments.

The Surprise: The "Tiny Brain" performed almost as well as the "Giant Brain." This is huge news because it means museums and libraries don't need to spend a fortune to use this technology.

4. What Did They Find?

The system was incredibly successful at turning messy text into a structured web of knowledge:

Metadata: It was nearly perfect (99% accuracy) at finding basic facts like titles and dates.
The Debate: It was very good (around 70-80% accuracy) at figuring out who was arguing and what they were arguing about.
The Evidence: It was excellent at spotting the specific reasons (e.g., "the ink is wrong") behind the arguments.

5. Why Does This Matter?

Imagine you are a historian researching a disputed painting.

Before: You have to read 50 different Wikipedia pages and academic papers, manually taking notes on who said what.
After ATR4CH: You ask the computer, "Show me all the arguments about this painting." The computer instantly pulls up a visual map showing:
- Person A says it's real because of the canvas.
- Person B says it's fake because of the paint.
- Person C agrees with Person B but adds new evidence about the frame.

It turns a chaotic pile of papers into a clear, organized conversation that you can search, filter, and understand instantly.

The Bottom Line

This paper proves that we can use modern AI to save the "human side" of history—the debates, the doubts, and the evidence—rather than just the cold facts. It gives museums and libraries a tool to turn their dusty archives into living, breathing, searchable conversations, all without breaking the bank.

1. Problem Statement

The paper addresses a critical gap in the Digital Humanities and Cultural Heritage (CH) domain: the inability to systematically convert unstructured, interpretative scholarly discourse into structured, queryable Knowledge Graphs (KGs).

The Representation Gap: While existing CH KGs (e.g., Wikidata, DBpedia) effectively handle metadata and structured catalogues, they fail to capture complex scholarly debates. These debates often involve conflicting hypotheses, evidential reasoning, and subjective interpretations (e.g., "Is this artifact a forgery?"). Current representations reduce these nuanced arguments to simple categorical assertions (e.g., "Forgery"), stripping away the underlying evidence and the spectrum of scholarly opinion.
Epistemological Challenge: Humanities scholarship is often "capta" (actively constructed knowledge) rather than "data" (passively recorded facts). Traditional KGs struggle to represent uncertainty, competing viewpoints, and the provenance of specific claims without becoming unmanageably complex.
Scalability Barrier: Manually extracting this complex reasoning from thousands of documents is prohibitively expensive and labor-intensive, creating a bottleneck for institutions with vast textual resources.

2. Methodology: ATR4CH

The authors propose ATR4CH (Adaptive Text-to-RDF for Cultural Heritage), a systematic, five-step iterative methodology designed to coordinate Large Language Model (LLM) extraction with ontological frameworks. Unlike traditional linear pipelines, ATR4CH treats annotation, ontology alignment, and extraction as interdependent processes.

The Five Steps:

Foundational Analysis: Analyzing the corpus and target ontology to identify Core Ontological Patterns (COPs)—the essential semantic structures needed to answer specific Competency Questions (CQs).
Minimal Working Annotation (MWA) Development: Iteratively developing an annotation schema using a pilot corpus. This step ensures the schema is practical for both manual annotation and automated extraction, mapping directly to the ontology.
Pipeline Architecture Development: Designing a computational pipeline. The authors utilize a sequential, modular approach where tasks are decomposed (metadata $\to$ entity recognition $\to$ opinion extraction $\to$ evidence mining $\to$ hypothesis extraction) to minimize error propagation.
Integration and Refinement: Harmonizing the annotation model, ontology, and pipeline into a production-ready system. This involves refining mapping algorithms and adding error handling.
Knowledge Extraction and Evaluation: Executing the pipeline on test data and evaluating results using both technical metrics (Precision/Recall/F1) and domain-expert validation (G-EVAL).

Case Study Implementation:

Domain: Authenticity assessment debates (e.g., historical forgeries, disputed artifacts).
Ontology: The SEBI (Scholarly Evidence Based Interpretation) ontology, which uses RDF-star to reify claims, allowing for the representation of concurrent, conflicting, and contextualized scholarly opinions.
Models Tested: The pipeline was evaluated using three LLMs of varying scales:
- Claude Sonnet 3.7 (Large)
- Llama 3.3 70B (Medium)
- GPT-4o-mini (Small/Lightweight)
Pipeline Components:
- GliNER: For lightweight Named Entity Recognition (NER).
- LLMs: For structured JSON extraction using In-Context Learning (ICL) and Chain-of-Thought (CoT) reasoning.
- Rule-based Linking: For entity resolution via Wikidata.

3. Key Contributions

ATR4CH Framework: The first systematic methodology for coordinating LLM-based extraction with Cultural Heritage ontologies, providing a replicable blueprint adaptable to various CH domains.
SEBI Ontology Application: A practical demonstration of using RDF-star to encode multi-perspectival scholarly debates, preserving the provenance of claims and the nuance of evidential reasoning rather than flattening them into binary facts.
Empirical Validation: A comprehensive evaluation of LLM performance on complex humanities tasks, challenging the assumption that larger models are always superior for domain-specific reasoning.

4. Results and Performance

The methodology was validated on a corpus of 581 Wikipedia articles regarding historical forgeries and authenticity controversies.

Quantitative Performance (F1-Scores):

Metadata Extraction: Exceptional performance across all models (0.96–0.99 F1). All models successfully extracted alleged item metadata (author, date, location).
Entity Recognition (Cognizers): Moderate performance (0.70–0.80 F1). GPT-4o-mini achieved the highest recall (91.2%), identifying the most scholarly agents, while Claude showed higher precision but lower recall.
Evidence Extraction: High accuracy (95.3%–96.8%). All models effectively captured multi-dimensional evidential reasoning (feature, evaluation, polarity) when the correct entity was identified.
Hypothesis Extraction: Moderate performance (0.62–0.75 F1). This was the most challenging task. GPT-4o-mini led in overall hypothesis extraction (0.749), while Claude excelled in geographic and temporal hypotheses.
Discourse Representation (G-EVAL): The overall quality of the generated KGs in representing the debate was rated around 0.60 (on a 0-1 scale). This indicates acceptable but improvable fidelity.

Key Findings on Model Comparison:

Size vs. Performance: Larger models (Claude) did not consistently outperform smaller models (GPT-4o-mini). In fact, GPT-4o-mini (the smallest model) often achieved the best balance of recall and precision for entity identification, leading to better downstream extraction.
Error Propagation: A critical bottleneck was identified: if an entity (Cognizer) is misidentified in the early stages, errors propagate through the pipeline, resulting in false positives in evidence and hypothesis extraction.
Cost-Effectiveness: Smaller models demonstrated competitive performance, suggesting that resource-constrained CH institutions can deploy cost-effective solutions without sacrificing significant accuracy.

5. Significance and Implications

Transforming CH Knowledge: ATR4CH enables Cultural Heritage institutions to convert vast, unstructured textual resources into queryable KGs. This supports automated metadata enrichment and allows researchers to query complex relationships (e.g., "Show all scholars who argued for the authenticity of X based on linguistic evidence").
Preserving Scholarly Nuance: By using RDF-star and the SEBI ontology, the system preserves the constructivist nature of humanities scholarship, capturing uncertainty, conflicting views, and the evolution of consensus, which are typically lost in standard LOD implementations.
Deployment Strategy: The paper argues for a hybrid human-machine approach. While LLMs can automate the bulk of extraction, human oversight remains necessary for post-processing and resolving ambiguities, particularly in entity linking and hypothesis validation.
Future Directions: The authors highlight the need for multilingual capabilities (currently English-only), fine-tuning for specific Cognizer classification, and the development of user-friendly tools to facilitate human-in-the-loop customization.

In conclusion, the paper demonstrates that combining ontological engineering with sequential LLM pipelines offers a viable, scalable path to structuring the complex, interpretative knowledge inherent in Cultural Heritage texts, moving beyond simple metadata to capture the substance of scholarly debate.