Employing General-Purpose and Biomedical Large Language Models with Advanced Prompt Engineering for Pharmacoepidemiologic Study Design

1. Problem Statement

The field of pharmacoepidemiology faces increasing pressure to generate rapid evidence for regulatory and clinical decision-making, particularly for populations underrepresented in clinical trials. Study design in this domain is complex, requiring the precise formulation of interrelated elements such as temporal windows, inclusion/exclusion criteria, exposure definitions, and ontology-code mapping (e.g., ICD, SNOMED, ATC).

While Large Language Models (LLMs) have shown promise in general biomedical tasks, their reliability in pharmacoepidemiologic study design remains uncharacterized. Key uncertainties include:

• Whether general-purpose LLMs (trained on broad data) or biomedical fine-tuned LLMs (specialized on medical literature) perform better for structured study design tasks.
• How prompt engineering strategies (e.g., Chain-of-Thought, Least-to-Most) influence the logical consistency and relevance of model outputs.
• The extent to which LLMs can accurately map natural language concepts to standardized medical ontologies, a critical step for reproducibility in real-world evidence (RWE) studies.

2. Methodology

Data Sources

The study evaluated LLM performance using 46 real-world pharmacoepidemiologic protocols from three major sources:

• HMA-EMA Catalogue: 16 DARWIN EU® protocols and 15 non-DARWIN expert-developed protocols (2023–2024).
• Sentinel System: 15 protocols (2022–2024) classified as "drug and outcome analysis" or "drug use."

Evaluation Framework

The study design was operationalized into 10 specific tasks based on the HARPER checklist:

1. Study Design Identification
2. Index Date Definition
3. Inclusion/Exclusion Criteria
4. Assessment Windows
5. Exposure Definition
6. Outcome Definition
7. Follow-up Period
8. Covariates
9. Covariate Assessment Windows
10. Ontology-Code Mapping (converting concepts to codes like ICD-10, ATC, RxNorm)

Models and Prompt Strategies

• Models Evaluated:
  • General-Purpose: GPT-4o and DeepSeek-R1.
  • Biomedical Fine-Tuned: QuantFactory/Bio-Medical-Llama-3-8B-GGUF, mradermacher/DeepSeek-r1-Medical-Mini-GGUF, Plaban81/gemma-medical_qa-Finetune, and Irathernotsay/qwen2-1.5B-medical_qa-Finetune.
• Prompt Engineering: Seven strategies were tested in a pre-assessment phase, including Basic, Synthetic, Active Prompting, Plan-and-Solve, Least-to-Most (LTM), Tree-of-Thought, and Decomposition.
• Selection: Based on a pre-assessment of 3 protocols, the top 5 combinations were selected for the main analysis:
  • GPT-4o + LTM
  • GPT-4o + Active Prompt
  • DeepSeek-R1 + LTM
  • Bio-Medical-Llama-3-8B + LTM
  • qwen2-1.5B-medical + LTM

Metrics and Gold Standard

• Gold Standard: Two human experts extracted the correct answers from the protocols.
• Primary Outcome (Relevance): Agreement between LLM output and human expert answers, measured on a 5-point Likert scale (1 = completely inaccurate, 5 = highly accurate).
• Secondary Outcome (Logic of Justification): A 5-point scale assessing the coherence, methodological rigor, and support for the reasoning.
• Tertiary Outcome (Ontology-Code Agreement): Percentage agreement between predicted codes and reference codes.
• Error Analysis: Qualitative review of error types (e.g., irrelevant auto-generated questions, lack of justification, prompt echoing).

3. Key Contributions

• Comparative Performance: The first systematic comparison of off-the-shelf general-purpose LLMs versus biomedical fine-tuned LLMs specifically for study design (not just knowledge retrieval).
• Prompt Strategy Impact: Demonstrated that Least-to-Most (LTM) prompting significantly enhances performance in complex, multi-step reasoning tasks compared to Active Prompting or basic prompts.
• Ontology Limitations: Identified a critical gap where LLMs perform well on narrative design tasks but fail significantly at ontology-code mapping, highlighting the need for deterministic support (dictionaries/crosswalks) rather than relying on LLM inference for coding.
• Protocol Sensitivity: Showed that the clarity and structure of the source protocol (e.g., HMA-EMA vs. DARWIN EU®) significantly impact LLM performance, suggesting that standardization is a deployment issue, not just a reporting one.

4. Results

Relevance and Logic of Justification

• General-Purpose Superiority: GPT-4o and DeepSeek-R1 (both general-purpose) consistently outperformed all biomedical LLMs.
  • GPT-4o + LTM achieved the highest stability, reaching a median relevance score of 4 in 8 out of 9 questions for HMA-EMA protocols.
  • Biomedical LLMs showed lower relevance scores and frequently failed to provide sufficient justification, often generating "irrelevant auto-generated questions" or lacking reasoning entirely.
• Prompt Strategy: Least-to-Most (LTM) prompting yielded higher relevance and better justification scores than Active Prompting, particularly for study design and exposure tasks. However, LTM introduced a risk of error propagation (early errors affecting later steps).
• Protocol Source Effect:
  • HMA-EMA protocols yielded the highest and most stable scores.
  • Sentinel protocols showed strong performance but higher variability.
  • DARWIN EU® protocols resulted in the lowest and least stable performance, particularly for outcome and follow-up definitions.

Ontology-Code Agreement

• Performance was markedly lower for coding tasks compared to narrative tasks.
• Agreement was observed mainly for ATC (Anatomical Therapeutic Chemical) codes and some procedure-based systems (HCPCS).
• Performance was limited or negligible for ICD, RxNorm, and SNOMED systems.
• The highest agreement recorded was 28.6% (DeepSeek-R1-LTM on ATC) and 22.2% (GPT-4o-LTM on HCPCS).

Error Patterns

• Biomedical LLMs exhibited recurrent structured failures, most notably:
  • Irrelevant auto-generated questions (n=189).
  • Lack of justification (n=105).
  • Overuse of conditional language (e.g., "might," "could"), leading to uncertainty.
• General-purpose models were more prone to occasional low-end outliers but maintained better overall adherence to the algorithm.

5. Significance and Conclusion

Conclusion:
Contrary to the assumption that domain-specific fine-tuning is superior for medical tasks, this study demonstrates that off-the-shelf general-purpose LLMs (specifically GPT-4o and DeepSeek-R1) combined with Least-to-Most prompting currently offer superior support for pharmacoepidemiologic study design. They excel in methodological reasoning and instruction following, whereas biomedical LLMs struggled with structured logic and justification.

Implications:

1. Workflow Integration: LLMs can effectively assist in the drafting and logical structuring of study protocols but should not be relied upon for ontology-code mapping without human verification or deterministic rule-based validation.
2. Prompt Engineering: The choice of prompt strategy (LTM) is as critical as the model selection. Decomposing complex design tasks into sequential subtasks improves reasoning stability.
3. Standardization: The variability in performance based on protocol source suggests that standardizing protocol reporting formats is essential for the successful deployment of AI in regulatory and research settings.
4. Future Research: While general-purpose models are currently superior, future work should explore hybrid approaches where general reasoning is combined with specific, curated biomedical knowledge bases to address the coding limitations.

This study provides a robust benchmark for the use of AI in regulatory science, emphasizing that for complex methodological tasks, broad semantic understanding and instruction adherence are currently more valuable than narrow biomedical specialization.