Automating Categorization of Scientific Texts with In-Context Learning and Prompt-Chaining in Large Language Models

Technical Summary: Automating Categorization of Scientific Texts with In-Context Learning and Prompt-Chaining in LLMs

1. Problem Statement

The exponential growth of scientific literature (e.g., 3.3 million articles in 2023 alone) has created a significant challenge for knowledge discovery and navigation. Traditional methods for categorizing research—such as manual tagging by librarians or subject experts—are time-consuming, unscalable, and prone to human error. While automated supervised learning models (like BERT or BiLSTM) have been used, they require extensive labeled training data and struggle with the deep, nested hierarchies and emerging interdisciplinary topics found in modern taxonomies like the Open Research Knowledge Graph (ORKG). There is a critical need for automated, scalable, and "off-the-shelf" solutions that can navigate complex hierarchical classification schemes without extensive fine-tuning.

2. Methodology

The researchers conducted a systematic evaluation of several contemporary Large Language Models (LLMs) to determine their ability to classify scientific texts (titles and abstracts) according to the three-level hierarchical ORKG taxonomy: Domain (1st level), Subject (2nd level), and Topic (3rd level).

• Dataset: The study utilized the Field of Research Classification (FoRC) dataset, consisting of 24,911 cleaned scientific texts (titles and abstracts) annotated by human experts.
• Models Evaluated: Four open-source LLMs were tested via the Ollama framework:
  • Llama 3.1 (70B)
  • Gemma 2 (27B)
  • Mistral Nemo (12B)
  • Phi-3 (3.82B)
• Prompt Engineering Strategies:
  • In-Context Learning (ICL): Tested via Zero-shot (no examples), One-shot (one example), and Few-shot (multiple examples) prompting.
  • Prompt Chaining: A top-down iterative approach where the model first predicts the Domain, then uses that result to select a Subject from a constrained list, and finally selects a Topic. This reduces complexity by narrowing the search space at each step.
• Hyperparameter Tuning: The researchers experimented with the Temperature setting (ranging from 0.2 to 1.0) to find the optimal balance between determinism and creativity.
• Evaluation Metrics: To account for "near misses" (semantic similarities that aren't exact string matches), they used:
  • Exact Match (EM): Binary comparison.
  • String Distance (SD): Normalized Levenshtein distance (threshold > 0.7).
  • Embedding Distance (ED): Cosine similarity of BERT sentence embeddings (threshold > 0.7).

3. Key Contributions

• Novel Application: The first systematic study applying off-the-shelf LLMs to the specific task of hierarchical research area classification using the ORKG taxonomy.
• Methodological Framework: Introduction of a hierarchical prompt chaining approach that decomposes complex taxonomy navigation into manageable, iterative steps.
• Comparative Analysis: A rigorous comparison between ICL and Prompt Chaining, as well as a benchmark against traditional models (BERT/BiLSTM) and various temperature settings.

4. Results

• Superiority of Prompt Chaining: Prompt chaining significantly outperformed pure In-Context Learning, particularly for the nested structure of the ORKG taxonomy.
• Model Performance: Llama 3.1 (70B) emerged as the top performer.
• Accuracy by Level:
  • Domain (1st level): High success, reaching up to ~90.1% accuracy.
  • Subject (2nd level): Strong performance, reaching up to ~80.5% accuracy.
  • Topic (3rd level): Significant drop in performance, reaching only approximately 50% accuracy.
• Optimal Hyperparameters: The best results were consistently achieved at a temperature of 0.8.
• Limitations Identified: LLMs struggle with "multi-label" or "multivalent" subjects (where a text belongs to several overlapping categories), often defaulting to a single, simplified label.

5. Significance

This research demonstrates that while LLMs are not yet fully capable of granular, three-level topic classification, they are highly effective at broad domain and subject categorization. The findings suggest that Prompt Chaining is a vital technique for handling complex, hierarchical data structures. This work provides a foundation for developing advanced research information systems in universities and industry, enabling more efficient automated tagging, expert identification, and knowledge organization in digital archives.