Curation and Extraction of Drug-Related Entities from Reddit Platform

Technical Summary: Curation and Extraction of Drug-Related Entities from Reddit Platform

Problem Statement
Physicians primarily learn about illicit drugs through clinical overdose cases, which limits their understanding of real-world usage patterns, dosages, and effects. Conversely, drug users frequently share first-hand experiences on social media platforms like Reddit, offering insights into novel psychoactive substances (NPS) and vernacular terminology that often diverge from formal medical language. While previous studies have utilized social media for tracking substance use, there is a significant lack of standardized, curated datasets that annotate not only drug names but also reported doses and effects. Existing Named Entity Recognition (NER) datasets are often limited to specific substances, rely on structured clinical narratives rather than unstructured user-generated text, or lack annotations for dosage and adverse effects. This gap hinders the development of NLP models capable of bridging the divide between clinical terminology and colloquial online discourse to identify emerging public health threats.

Methodology
To address these challenges, the authors introduced REDOSE (REddit Drug DOSe and Effect), a comprehensive dataset and benchmarking framework.

• Data Curation: The dataset comprises 6,435 unique documents collected from seven drug-related subreddits (e.g., r/fentanyl, r/heroin, r/suboxone). Data was extracted using the Reddit API (praw), with metadata such as usernames and geolocation removed to protect privacy. Duplicate posts were filtered out.
• Annotation Process: A board-certified medical toxicologist annotated the training and test sets, while two medical science students annotated the validation set. The annotation scheme focused on three entity types:
  • DRUG: Drugs, vitamins, or herbs (excluding neurotransmitters).
  • DOSE: Quantity and units of the substance.
  • EFFECT: Physical or mental state changes attributed to the substance.
  • Inter-annotator agreement (IAA) was calculated using a method adapted for multi-entity documents, yielding a score of 0.75.
• Model Development and Benchmarking: The study evaluated three categories of models:
  1. BERT-based Models: Fine-tuned BaseBERT, BioBERT, and BiomedBERT. Additionally, CRF and LSTM layers were added to BaseBERT and BioBERT to assess performance gains.
  2. Large Language Models (LLMs): GPT-4 and Llama-3 (8B and 70B variants) were tested using one-shot prompting.
  3. Retrieval-Augmented Generation (RAG): A retrieval-based approach was developed for LLMs to extract the top 5 most similar examples from the training corpus to append to the prompt, aiming to overcome the limitations of static few-shot examples.
• Evaluation: Metrics included span-level precision, recall, and F1-scores. For LLMs, document-level metrics were used due to tokenization challenges with numerical doses.

Key Results

• Dataset Statistics: REDOSE contains 6,267 annotated entities (4,784 DRUG, 750 EFFECT, 733 DOSE) across 15,469 sentences.
• BERT Performance:
  • BiomedBERT achieved the highest F1-score for DRUG extraction (0.843) with high precision (0.907).
  • BaseBERT achieved the highest recall for DRUG (0.873) and the highest overall micro-averaged F1-score (0.826) among BERT variants.
  • EFFECT Extraction: All BERT-based models struggled significantly with EFFECT extraction, with F1-scores ranging from 0.043 (BiomedBERT) to 0.298 (BaseBERT). Adding CRF+LSTM layers improved EFFECT F1-scores slightly (up to 0.291 for BioBERT+CRF+LSTM) but did not significantly outperform simpler models in other categories.
• LLM Performance:
  • Llama-3 70B outperformed GPT-4 in DRUG extraction (F1 = 0.79 vs. 0.72).
  • GPT-4 demonstrated superior performance in EFFECT extraction, achieving a recall of 0.41 compared to Llama-3 70B's 0.04, resulting in the best overall F1-score for EFFECT (0.12).
  • RAG Impact: The RAG approach significantly improved Llama-3 8B's DRUG extraction recall (from 44% to 80%) and F1-score (from 0.55 to 0.75), though gains for EFFECT extraction were modest.
• Comparative Analysis: Contrary to some expectations, fine-tuned BERT models (specifically BaseBERT) outperformed LLMs in overall recall and F1-scores for drug extraction. The authors attribute this to the alignment of Reddit's user-generated text with general language corpora rather than strict medical jargon.

Significance and Contributions
The paper claims three primary contributions:

1. REDOSE Dataset: The introduction of the first dataset of its kind collected from online platforms with detailed annotations of drugs, doses, and effects. It captures patient-curated narratives and vernacular terms (e.g., "fent," "SEA #4") that are often missed by formal medical records, providing a resource to train models for emerging substance detection.
2. Benchmarking: The study provides a fair comparison between traditional fine-tuned BERT models and modern LLMs (both one-shot and RAG-based) for medical NER tasks in a social media context.
3. Model Analysis: The research offers a deep analysis of model behaviors, highlighting that while LLMs possess large knowledge bases, they may struggle with the specific frequency distributions and strict boundary definitions required for medical NER without fine-tuning. It also identifies that simpler models may be better suited for user-generated content that deviates from domain-specific language.

The authors conclude that REDOSE serves as a critical resource for advancing NLP in substance use research, enabling the detection of emerging trends and informing public health guidelines, while noting that future work should address dataset biases and explore multimodal integration.