Large Language Models for Biomedical Article Classification

1. Problem Statement

The paper addresses the challenge of classifying biomedical articles for systematic literature reviews, specifically focusing on Drug Class Reviews (DCR). In this domain, experts must decide whether to include or exclude articles based on abstracts and full texts.

• The Challenge: Traditional machine learning (ML) classifiers require large amounts of labeled training data, which is expensive and time-consuming to produce in specialized medical domains.
• The Proposed Solution: Investigating the utility of Large Language Models (LLMs) as text classifiers. LLMs offer the potential to perform classification with zero-shot (no examples) or few-shot (few examples) prompting, leveraging their pre-trained knowledge rather than requiring extensive retraining.
• Specific Gap: Prior research often focused only on basic accuracy/F1 scores using simple prompts and direct text output. This paper argues that such evaluations are insufficient because they ignore prediction confidence (probabilities), which is crucial for handling class imbalance and adjusting decision thresholds in high-stakes medical applications.

2. Methodology

The study conducted a systematic, multi-faceted experimental investigation using 15 binary classification datasets derived from PubMed abstracts related to different drug groups (e.g., ACE Inhibitors, Statins, Opioids). The datasets are characterized by severe class imbalance (positive class percentages ranging from 2.2% to 34.5%).

2.1 Models Evaluated

• Open Source: Llama 3.1 (8B and 70B), Mistral 7B v0.3, Gemma 2 9B.
• Closed Source: OpenAI GPT-4.1 mini and GPT-4.1 nano.
• Baselines: Naïve Bayes, Random Forest (with and without hyperparameter tuning), and fine-tuned Transformer encoders (DeBERTa, SciDeBERTa-v2).

2.2 Experimental Dimensions

The authors systematically varied four key dimensions to identify optimal configurations:

1. Prompt Design: Tested 6 prompts varying in complexity (simple, medium, complex) and structure (presence of system messages, direct questions, task descriptions).
2. Output Processing: Compared methods for extracting class predictions and confidence scores:
   • Word/JSON: Direct text output (e.g., "yes"/"no" or JSON).
   • Score: Prompting the model to output a numeric score (0–5).
   • Token Probabilities: Calculating class probabilities based on the probability distribution of the first generated token.
3. Few-Shot Strategies:
   • Count: Tested 0 (zero-shot) to 4 examples.
   • Selection: Compared random selection vs. semantic similarity-based selection (using embeddings from MiniLM, DistilRoBERTa, and MPNet) and class-balanced selection.
4. Advanced Techniques: Evaluated Chain of Thought (CoT), Tree of Thoughts (ToT), and text Chunking.

2.3 Evaluation Metrics

Performance was measured using stratified 5-fold cross-validation with a focus on:

• AUPRC (Area Under the Precision-Recall Curve): The primary metric, chosen because it is robust to class imbalance and requires numeric probability scores.
• MCC (Matthews Correlation Coefficient): For discrete class predictions.
• Standard metrics: Accuracy, Precision, Recall, F1.

3. Key Contributions

1. Systematic Output Processing Analysis: The paper is one of the first to rigorously compare token probability-based confidence estimation against direct text prompting. It demonstrates that token probabilities yield significantly better AUPRC scores than asking the model to generate a numeric score or text label.
2. Comprehensive Few-Shot Investigation: It provides empirical data on the diminishing returns of adding more than 3–4 examples and confirms that semantic similarity-based selection outperforms random selection, while class rebalancing during selection offers no benefit.
3. Advanced Technique Evaluation: It empirically tests advanced reasoning techniques (CoT, ToT) and chunking, finding them ineffective for this specific task compared to simpler few-shot setups, despite higher computational costs.
4. Benchmarking: Provides a direct comparison between LLMs and conventional ML/Transformer baselines on a challenging, real-world biomedical dataset.

4. Key Results

• Performance vs. Baselines:
  • Conventional algorithms (Random Forest with tuning, Naïve Bayes) and fine-tuned SciDeBERTa generally achieved the highest AUPRC scores (approx. 0.50–0.55).
  • LLMs came close: The best few-shot LLM configurations achieved an average AUPRC of ~0.47–0.49, approaching the performance of conventional models.
  • Zero-shot performance: Even without examples, LLMs achieved an average AUPRC of >0.40, demonstrating non-trivial utility.
• Output Processing:
  • Token Probabilities were the superior method for obtaining class probabilities, significantly outperforming "Score" prompting and direct text output.
• Prompting & Examples:
  • Few-shot is critical: Adding just one example drastically improved performance over zero-shot.
  • Diminishing Returns: Increasing examples from 1 to 4 yielded marginal gains; 3 examples were often sufficient.
  • Prompt Structure: Including a direct question about the class in the user message (e.g., "Should this be included?") was more effective than prompt complexity alone.
• Model Size & Type:
  • No consistent correlation was found between model size and performance. Surprisingly, Llama 3.1 8B slightly outperformed the 70B variant in AUPRC.
  • Open-source models performed comparably to commercial models (GPT-4.1 mini/nano).
• Advanced Techniques: Chain of Thought, Tree of Thoughts, and Chunking failed to improve AUPRC scores and increased computational overhead.

5. Significance and Recommendations

The study validates that LLMs are viable alternatives to conventional classifiers for biomedical text classification, particularly in scenarios where labeled data is scarce.

Practical Recommendations derived from the study:

1. Use Few-Shot Prompting: Even a single example significantly boosts performance.
2. Select Examples by Similarity: Use semantic similarity (embeddings) to choose few-shot examples rather than random selection.
3. Leverage Token Probabilities: Do not rely on the model's generated text for confidence; instead, calculate probabilities based on the first token's likelihood to enable threshold tuning.
4. Keep Prompts Simple: Complex reasoning techniques (CoT/ToT) are not necessary for this task; a direct question in the prompt is sufficient.
5. Cost-Effectiveness: Smaller open-source models (e.g., 8B parameters) can match the performance of larger commercial models for this specific task, offering a cost-effective solution.

Conclusion: While LLMs did not strictly surpass fine-tuned transformers or tuned Random Forests in this study, they achieved comparable results with significantly less data preparation. The ability to generate reliable probability estimates via token processing makes them a powerful tool for medical literature screening, especially when labeled training data is limited.