Benchmarking Resource-Efficient LLMs for Research Topic Ontology Generation in the Biomedical Field

Technical Summary: Benchmarking Resource-Efficient LLMs for Biomedical Topic Ontology Generation

Problem Statement
Knowledge Organization Systems (KOSs), such as ontologies and taxonomies, are critical for structuring scientific knowledge and enabling efficient data retrieval in digital libraries. However, the manual curation of these systems creates a persistent bottleneck, resulting in uneven coverage and infrequent updates that fail to keep pace with rapidly emerging research areas. While Large Language Models (LLMs) offer a scalable alternative for automated ontology generation, their ability to classify complex, domain-specific semantic relationships—particularly within the biomedical field—requires systematic evaluation. Existing automated approaches often struggle with the granular complexity of scientific terminologies, and current LLM applications frequently rely on human-in-the-loop verification rather than autonomous generation.

Methodology
To address these challenges, the authors conducted a systematic evaluation of five small, open-source LLMs (ranging from 3.2B to 9.2B parameters) on the task of identifying semantic relationships between pairs of biomedical research topics. The study focused on four specific classification categories: broader (parent topic), narrower (child topic), same-as (synonymous), and other (unrelated).

The research employed the following methodological components:

1. Dataset Construction (MeSH-Rel-4K): The authors introduced a new dataset comprising 4,000 semantic relationships extracted from the Medical Subject Headings (MeSH) ontology (January 2025 release). The dataset was partitioned into training (2,800), validation (400), and test (800) subsets with a 7:1:2 ratio, ensuring strict isolation of research concepts to prevent node leakage.
2. Adaptation Strategies: Three distinct strategies were evaluated:
   • Standard Prompting: Using predefined templates to instruct models on the classification task.
   • Chain-of-Thought (CoT) Prompting: A two-way, sequential prompting mechanism where models first define topics and construct sentences before classifying relationships, with bidirectional verification to resolve discrepancies.
   • Fine-Tuning: Parameter-Efficient Fine-Tuning (PEFT) using Low-Rank Adaptation (LoRA) on the MeSH-Rel-4K dataset. All models were 4-bit quantized and trained using the Unsloth library on consumer-grade GPUs.
3. Models Evaluated: The study utilized mistral-7b, llama-3.2-3b-instruct, gemma-2-9b-it, phi-3.5-mini-instruct, and zephyr-7b.

Key Results
The experiments yielded significant findings regarding the efficacy of fine-tuning versus prompting:

• Prompting Performance: While CoT prompting provided moderate improvements over standard prompting (an average F1-score increase of 5.4 percentage points), it was insufficient to bridge the reasoning gaps in smaller models. For instance, the smallest model, llama-3b, saw only a marginal 0.5 percentage point gain with CoT.
• Fine-Tuning Impact: Direct fine-tuning proved to be the most effective strategy, increasing the average F1-score by 34.1 percentage points across all models compared to prompting baselines.
  • The largest model, gemma-9b, achieved the highest overall performance with an F1-score of 91.6%.
  • The smallest model, llama-3b, demonstrated the most dramatic relative improvement, rising from an F1-score of 25.7% (with CoT) to 86.2% after fine-tuning—a 60.5 percentage point increase.
• Error Analysis: Fine-tuned models excelled at identifying unrelated topics (other category), achieving near-perfect precision and recall. The primary challenge remained distinguishing same-as relationships from hierarchical ones (broader/narrower). This confusion was inversely proportional to model size; gemma-9b minimized these errors, while llama-3b frequently misclassified synonymous terms as hierarchical relations due to dense lexical overlaps in the MeSH ontology.

Significance and Contributions
The paper makes three primary contributions to the field of automated ontology engineering:

1. Dataset Release: The introduction of MeSH-Rel-4K, a specialized dataset of 4,000 biomedical semantic relationships, which is openly available to support future research in ontology generation.
2. Empirical Validation of Resource-Efficiency: The study demonstrates that parameter-constrained models (under 10B parameters) can achieve near-state-of-the-art performance in domain-specific semantic classification when subjected to targeted fine-tuning, effectively bypassing the reasoning bottlenecks typically associated with smaller LLMs.
3. Methodological Framework: The authors provide a reproducible pipeline for constructing and evolving specialized biomedical ontologies, confirming that direct fine-tuning offers a more accurate and automated methodology than prompting-based approaches alone.

The authors conclude that while prompting strategies offer baseline utility, explicit fine-tuning is necessary to unlock the full potential of small, open-source LLMs for the construction and evolution of specialized scientific knowledge graphs. Future work is proposed to extend this analysis to other academic disciplines (Humanities, Social Sciences) and to develop pipelines for ontology matching and evolution.