Technical Summary: Leveraging Large Language Models for Generating Research Topic Ontologies

Problem Statement

Ontologies and taxonomies are fundamental to organizing scientific knowledge, facilitating the classification, dissemination, and retrieval of research outputs (e.g., articles, patents, datasets). However, the creation and maintenance of these Knowledge Organization Systems (KOSs) are expensive, time-consuming, and rely heavily on the coordinated effort of domain experts. Consequently, existing ontologies often suffer from uneven disciplinary coverage, weak inter-disciplinary connectivity, and infrequent updates that fail to reflect rapid scientific progress. While previous attempts to automate ontology generation have utilized linguistic tools and machine learning, they struggle to produce large-scale, fine-grained representations due to the inherent complexity and specialization of research domains. This study addresses the gap in systematically evaluating the capability of Large Language Models (LLMs) to identify semantic relationships between research topics, a core task in constructing and refining academic ontologies.

Methodology

The study investigates the performance of 12 open-weight LLMs (ranging from 3 billion to 27 billion parameters) across three distinct optimization strategies:

1. Zero-Shot Learning (ZSL): Evaluating models without prior training on the specific task using two prompting strategies:
   • Standard Prompting: A predefined template defining the task and output format.
   • Bidirectional Chain-of-Thought (CoT): A two-step prompting technique where the model first defines topics and constructs a sentence, then classifies the relationship, with the process repeated with swapped topic positions to resolve discrepancies.
2. Fine-Tuning: Adapting models using Low-Rank Adaptation (LoRA) on a novel dataset.
3. Evaluation Scenarios: The study employs a modular experimental design testing:
   • Discipline-specific: Training and testing within the same domain.
   • Cross-discipline: Training on one domain and testing on another.
   • Multidisciplinary: Training and testing on a combined dataset.

The PEM-Rel-8K Dataset

To support this analysis, the authors introduced PEM-Rel-8K, a novel benchmark containing over 8,000 semantic relationships extracted from three major taxonomies:

• IEEE Thesaurus: Engineering and Computer Science (~3,200 relationships).
• PhySH (Physics Subject Headings): Physics (~875 relationships).
• MeSH (Medical Subject Headings): Biomedicine (~4,000 relationships).

The dataset categorizes relationships into four mutually exclusive classes:

• Broader: Topic A subsumes Topic B.
• Narrower: Topic A is subsumed by Topic B.
• Same-as: Topics are semantically equivalent.
• Other: No semantic relationship exists (negative examples).

The dataset construction involved manual expert supervision, particularly for identifying "same-as" relations, which are not always explicitly defined in source taxonomies (e.g., distinguishing synonyms from alternative labels).

Key Contributions

1. Comprehensive LLM Analysis: A systematic evaluation of 12 LLMs across three scientific disciplines (Engineering, Physics, Biomedicine) to identify semantic relations.
2. Methodological Comparison: A rigorous comparison of Zero-Shot, Chain-of-Thought, and Fine-Tuning approaches for ontology generation tasks.
3. Cross-Discipline Transferability Study: An investigation into whether models fine-tuned on one discipline can generalize to others, revealing the potential for unified multi-disciplinary models.
4. PEM-Rel-8K Benchmark: The introduction and public release of a modular, multi-disciplinary dataset designed specifically for training and evaluating models on research topic relationship identification.
5. Open Source Release: Provision of the complete codebase, datasets, and prompt templates to ensure reproducibility.

Results

The experiments yielded the following key findings:

• Superiority of Fine-Tuning: Fine-tuning LLMs on PEM-Rel-8K significantly outperformed both Zero-Shot and Chain-of-Thought prompting. The best-performing model, gemma-2-27b-it, achieved an F1-score of 93.5% on the multidisciplinary test set.
• Performance of CoT: While Bidirectional CoT outperformed standard prompting in Zero-Shot settings (improving F1 by ~5.1%), it still lagged significantly behind fine-tuned models.
• Cross-Discipline Generalization: Models fine-tuned on a single discipline demonstrated strong transferability. For instance, a model trained on Physics data achieved competitive results when tested on Biomedicine data. Crucially, a model fine-tuned on the combined PEM-Rel-8K dataset achieved an average F1-score across the three disciplines that was only 1.2% lower than models fine-tuned exclusively on discipline-specific datasets.
• Model Variability: While gemma-27b was the top performer overall, other models like phi-4 and mistral-22b also showed strong capabilities, with performance varying by discipline (e.g., phi-4 performed best on MeSH, while gemma-9b excelled on PhySH).
• Error Analysis: The most challenging relation to classify was "same-as" (average F1: 88.5%), often confused with hierarchical relations due to lexical overlap and inconsistent definitions across ontologies. The "other" category (non-related pairs) was the easiest to identify.

Significance and Claims

The paper claims that this study represents the first large-scale investigation into using LLMs for identifying semantic relationships between research topics across a wide range of scientific disciplines. The primary significance lies in demonstrating that fine-tuning LLMs on a curated, multi-disciplinary dataset can produce robust models capable of generalizing across different scientific fields.

The authors assert that this approach offers a viable path to overcoming the fragmentation and uneven coverage of current research ontologies. By showing that a single model can effectively handle engineering, physics, and biomedicine with minimal performance loss compared to specialized models, the study suggests a shift away from the costly, manual curation of separate ontologies toward automated, unified knowledge organization systems. The results indicate that LLMs can significantly reduce the barrier to creating and maintaining high-quality, up-to-date ontologies, thereby improving the discoverability and analysis of scientific literature.

The authors remain modest regarding the current limitations, noting that while LLMs are powerful assistive tools, they still face challenges in consistency and accuracy that may require human oversight, particularly for complex semantic relations like "same-as." Future work is proposed to expand to social sciences and humanities and to develop systems for ontology matching and evolution.