💬 NLP

From Prompt to Graph: Comparing LLM-Based Information Extraction Strategies in Domain-Specific Ontology Development

This study evaluates three Large Language Model-based strategies—pre-trained prompting, in-context learning, and fine-tuning—for automating domain-specific ontology construction in casting manufacturing, ultimately identifying the most effective approach to build and validate a knowledge graph with expert input.

Original authors: Xuan Liu, Ziyu Li, Mu He, Ziyang Ma, Xiaoxu Wu, Gizem Yilmaz, Yiyuan Xia, Bingbing Li, He Tan, Jerry Ying Hsi Fuh, Wen Feng Lu, Anders E. W. Jarfors, Per Jansson

Published 2026-02-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuan Liu, Ziyu Li, Mu He, Ziyang Ma, Xiaoxu Wu, Gizem Yilmaz, Yiyuan Xia, Bingbing Li, He Tan, Jerry Ying Hsi Fuh, Wen Feng Lu, Anders E. W. Jarfors, Per Jansson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of old, messy technical books about casting (the process of pouring molten metal into molds to make things). These books are full of valuable knowledge, but it's buried in paragraphs of text, written in different styles, and scattered everywhere.

To make this knowledge useful for computers, engineers need to build a digital map (called an ontology). This map organizes the information into clear categories (like "Materials," "Defects," or "Equipment") and draws lines connecting related ideas (like "Temperature controls Solidification Time").

The problem? Building this map by hand is like trying to read every single page of those books, highlight the important words, and draw the connecting lines yourself. It takes forever, costs a lot of money, and is very boring.

This paper asks: Can we use a super-smart AI (a Large Language Model or LLM) to do the heavy lifting for us?

The researchers tested three different ways to ask the AI to read these books and build the map. Here is how they did it, explained with simple analogies:

The Three "AI Assistants" Tested

1. The "General Knowledge" Assistant (Pre-trained LLM)

How it works: You give the AI a set of instructions (a prompt) like, "Read this text and tell me the important words and how they connect." You don't show it any examples of what you want; you just hope it already knows enough from its general training.
The Result: It's like asking a very well-read librarian to guess what you need. It gets some things right, but it misses a lot of details. It found about 77% of the right words but missed many connections. It's fast and cheap, but not detailed enough for a perfect map.

2. The "Show Me an Example" Assistant (In-Context Learning or ICL)

How it works: Before asking the AI to read the new text, you show it a few examples of what you want. It's like saying, "Here is a sample page where I highlighted 'Metal' and drew a line to 'Mold' saying 'is made of.' Now, please do the same for this new page."
The Result: This is like giving the librarian a cheat sheet. It found way more words than the first method (almost 3 times as many!). However, it had a weird habit of being inconsistent. Sometimes it called a connection "processed by," and other times "made by," even when they meant the same thing. It also struggled to spot when two different words meant the exact same thing (synonyms).

3. The "Intern Who Studied Your Notes" Assistant (Fine-Tuning)

How it works: Instead of just showing examples, you take the AI and "train" it specifically on your small set of labeled data. You feed it the examples over and over until it learns the specific rules and style you want. It's like hiring a new employee and spending a week teaching them exactly how your company organizes its files.
The Result: This was the clear winner. Because it was specifically trained on your style, it was incredibly accurate. It found the right words 94% of the time and the right connections 87% of the time. It was also much better at spotting that "Melting Point" and "Melting Temperature" are the same thing.

The Final Outcome

The researchers took the best method (the "Intern" who was fine-tuned) and used it to build a real Casting Ontology.

They fed it thousands of sentences from technical books.
The AI extracted the key terms and relationships.
They cleaned up the duplicates and organized everything into a digital graph database (Neo4j).
The Proof: Human experts (real casting engineers) reviewed the AI's work. They confirmed that 97% of the concepts and 93% of the relationships were correct.

The Bottom Line

The paper concludes that while you can just ask a smart AI to do the job, it's better to teach it specifically for your field.

General AI: Good for a quick first draft, but messy.
Example-based AI: Good at finding things, but messy with labels.
Trained AI: The most accurate and reliable, capable of building a high-quality, professional knowledge map even with a relatively small amount of training data.

This study proves that with the right training, AI can turn messy, unstructured engineering documents into a clean, organized, and usable digital knowledge base, saving experts from doing the tedious work of manual annotation.

Technical Summary: From Prompt to Graph: Comparing LLM-Based Information Extraction Strategies in Domain-Specific Ontology Development

Problem Statement
Ontologies are critical for structuring domain knowledge to facilitate interoperability, data sharing, and reasoning, particularly in complex industrial sectors like casting manufacturing. However, traditional ontology construction is labor-intensive, costly, and relies heavily on manual annotation of fragmented, unstructured documentation. While Large Language Models (LLMs) offer potential for automating knowledge extraction, existing research lacks side-by-side comparisons of different LLM-based approaches. Furthermore, many existing manufacturing ontologies are informal, lacking complete concept-relation triples, and prior studies often assume access to large, well-structured datasets, which is not typical in traditional manufacturing environments.

Methodology
This study proposes a semi-automated ontology learning framework to extract terms and relations from domain-specific texts (academic papers and technical books) using limited data. The research evaluates and compares three distinct LLM-based strategies:

Pre-trained LLM-driven Method: Utilizes a general pre-trained model (ChatGPT 4.1 mini) guided by optimized prompts based on Chain-of-Thought (CoT) methodology. This method extracts terms and relations directly without human-annotated examples, relying on the model's inherent knowledge.
In-Context Learning (ICL) Method: A two-step process (term extraction followed by relation extraction) that uses $k$ -shot examples from a small training set to guide the LLM. To ensure semantic relevance, the study employs the Universal AnglE Embedding (UAE)-Large-V1 model to retrieve the top $k$ (set to 16) most similar training samples for each test input.
Fine-tuning Method: A two-step supervised fine-tuning approach where two customized models (one for term extraction, one for synonym/relation extraction) are trained on the entire labeled dataset. This method updates model parameters to adapt the base model (OpenAI GPT-4.1-mini) to the specific casting domain without requiring $k$ -shot examples during inference.

Experimental Setup
The study focused on the casting domain, guided by six top-level concepts: materials, casting process, product property, casting parameter, casting defect, and casting equipment.

Data: A training dataset of 173 short texts and a testing dataset of 30 long paragraphs were curated.
Annotation: Human experts manually labeled 538 terms and 321 relation triples in the training set, and 590 terms and 481 relations in the test set.
Evaluation: Performance was measured using Precision, Recall, and F1-score. The extracted knowledge was structured as triples (subject, object, relation) and implemented in a Neo4j graph database.

Key Results

Pre-trained LLM: Achieved approximately 77.3% precision in term extraction and 79.5% in relation extraction. However, it extracted significantly fewer entities (142 terms, 36 relations) compared to the ground truth and lacked positional information, preventing Recall and F1-score calculation. It was prone to hallucinations and irrelevant classifications.
ICL Method: Extracted 421 terms (Precision: 79.8%, Recall: 56.9%, F1: 66.4%) and 424 relation triples (Precision: 63.9%, Recall: 56.3%, F1: 59.9%). While effective for long texts, it suffered from inconsistent relation naming (e.g., varying synonyms for the same relationship) and poor synonym recognition (identifying only 1 of 22 annotated pairs).
Fine-tuning Method: Demonstrated the superior performance across all metrics. It achieved a term extraction F1-score of 90.6% (Precision: 93.8%, Recall: 87.6%) and a relation extraction F1-score of 79.5% (Precision: 86.7%, Recall: 73.4%). It showed greater consistency in relation naming and significantly better synonym recognition (identifying 10 of 22 pairs).

Ontology Construction
The fine-tuned method was selected to construct the final casting ontology. After consolidating synonyms and removing duplicates, 273 unique concepts and 620 validated relations were retained. Expert validation confirmed a concept precision of 97% and a relation precision of 93%. The resulting ontology was visualized in Neo4j, demonstrating a hierarchical organization of casting terminology.

Significance and Contributions
The paper claims three primary contributions:

A comparative evaluation of three LLM-based methods (pre-trained prompting, ICL, and fine-tuning) for term and relation extraction in a domain-specific setting with limited data.
The proposal of a prompt-based relation extraction strategy designed to improve ontology construction in manufacturing applications.
The development of a high-quality, validated casting ontology derived from academic and technical literature, addressing the gap of informal ontologies in manufacturing.

The study concludes that while pre-trained models serve as a useful starting point and ICL offers cost-efficiency, the fine-tuning approach is the most effective for high-quality ontology development when resources permit. The authors note that the study is limited by the small dataset size and suggest future work should validate these methods on larger, more diverse corpora to assess scalability in real-world industrial settings.

The Three "AI Assistants" Tested

The Final Outcome

The Bottom Line

More like this