💬 NLP

Optimizing Large Language Models for Causality Assessment in Pharmacovigilance: Developing a Performance Metric as Objective for Bayesian Hyperparameter Optimization

This study demonstrates that while no universal temperature optimum exists for Large Language Models in pharmacovigilance, developing a novel Entropy-Weighted Agreement and Cosine Similarity Score (EWACS) metric enables Bayesian hyperparameter optimization to significantly improve LLM-expert agreement on Naranjo causality assessments, particularly for doubtful cases.

Original authors: Nicole Sonne Heckmann, Arnault-Quentin Vermillet, Søren Norlin Mølgaard, Manuela Del Castillo Suero, Lars Melskens, Gerard Ompad, Maurizio Sessa

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nicole Sonne Heckmann, Arnault-Quentin Vermillet, Søren Norlin Mølgaard, Manuela Del Castillo Suero, Lars Melskens, Gerard Ompad, Maurizio Sessa

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: Did a specific medicine cause a patient's illness?

In the real world, this job is done by human safety experts. They read thousands of messy reports (called "Individual Case Safety Reports" or ICSRs) and use a strict checklist called the Naranjo Algorithm to decide if the drug is the culprit. It's a slow, tiring job.

Enter Large Language Models (LLMs)—super-smart AI chatbots. Researchers hoped these AIs could do the detective work automatically. But there was a problem: the AI was often wrong, especially on the tricky parts of the checklist where human judgment is needed.

This paper is about a team of researchers who tried to fix the AI's "brain settings" to make it a better detective. Here is the story of what they found, explained simply.

1. The Problem: The AI's "Creativity Dial"

Think of the AI as a student taking a test. The researchers could turn a dial called "Temperature" on the AI's settings.

Low Temperature (0): The AI is very strict, logical, and boring. It always gives the same answer.
High Temperature (2): The AI is wild, creative, and unpredictable. It might try many different answers.

The researchers wanted to know: Is there a "Goldilocks" setting (not too low, not too high) that makes the AI agree with human experts the most?

2. The Challenge: Measuring "Good" Answers

To find the perfect dial setting, they needed a way to grade the AI. But grading is hard because:

Sometimes the AI gets the final score right but gives a weird reason.
Sometimes the AI gets the reason right but the score wrong.
Most of the time, the human experts all agree on the easy questions, so the AI just needs to copy them.

The researchers built four different "report cards" (metrics) to grade the AI. They wanted one that could talk to a computer algorithm (Bayesian Optimization) to automatically tweak the temperature dial until it found the best setting.

3. The Big Discovery: There is No "One Size Fits All"

After running thousands of tests, they found a surprising truth: There is no single "perfect" temperature for every case.

The Analogy: Imagine you are trying to find the perfect volume for a radio. If you are listening to a jazz station, you might want the volume loud. If you are listening to a news broadcast, you might want it quiet.
The Result: The AI's performance didn't depend on the temperature dial itself. Instead, it depended entirely on what was written in the patient's report.
- If the report was clear and detailed, the AI got it right no matter what the temperature was.
- If the report was vague or missing information, the AI struggled, regardless of the setting.

4. The Twist: Fixing the AI "Case by Case"

Even though there was no universal "best setting," the researchers found a clever workaround. They realized that different cases needed different settings.

The Strategy: Instead of picking one temperature for the whole day, they let the computer pick a specific temperature for each individual case based on how tricky that specific case was.
The Result: This "custom tuning" was a huge success.
- Before: The AI agreed with humans on the final verdict only 45% of the time.
- After: With the custom tuning, agreement jumped to 72%.
- The Magic Spot: The biggest improvement happened with the "Doubtful" cases (the messy, hard-to-solve mysteries). The AI stopped guessing and started thinking more like a human expert for those specific tricky reports.

5. The "Report Card" Winner

Out of the four different ways they tried to grade the AI, one method called EWACS worked the best.

Why? It was smart enough to ignore the easy questions (where everyone agrees) and focus its energy on the hard questions (where the AI usually fails). It was like a teacher who stops grading the spelling of simple words and focuses entirely on the complex essay questions.

6. The Final Takeaway

The AI is getting better: The new AI model (GPT-5.2) was already better at this job than previous specialized medical AIs, even before they tweaked the settings.
The Data is the Key: The quality of the AI's answer depends mostly on how much information is in the patient report, not on the AI's settings.
The Solution: You don't need to find one perfect setting for the whole world. You just need to let the computer pick the right setting for each specific case. This simple trick turned a mediocre AI detective into a much more reliable one.

In short: The researchers didn't find a magic switch to make the AI perfect. Instead, they taught the AI how to adjust its own "mood" (temperature) depending on the specific mystery it was solving, which made it much smarter at spotting drug side effects.

Technical Summary: Optimizing Large Language Models for Causality Assessment in Pharmacovigilance

Problem Statement
The volume of Individual Case Safety Reports (ICSRs) in the FDA Adverse Event Reporting System (FAERS) has surged, creating an urgent need for scalable, automated causality assessment. While Large Language Models (LLMs) show promise, their performance on clinically demanding tasks—specifically the Naranjo algorithm's assessment of temporal plausibility, alternative causes, and objective evidence—remains suboptimal for routine use. Previous studies utilizing biomedical LLMs achieved moderate agreement with human experts but failed to resolve specific challenges like listedness detection or complex clinical judgment. Furthermore, the potential of inference-time hyperparameter optimization, specifically temperature tuning, to improve LLM performance in this domain had not been investigated. A critical barrier to such optimization is the lack of a Gaussian Process (GP)-compatible performance metric: existing measures (binary agreement, ordinal ratings, or raw cosine similarity) either fail to capture the nuances of LLM output quality or introduce structural biases specific to the sparse, template-driven nature of spontaneous reporting data.

Methodology
The study employed a multi-stage approach to develop an optimization framework and evaluate its efficacy:

Data Curation: A stratified subset of 723 FAERS cases was selected from a larger pool of 55,177 cases (derived from Q1-Q2 2025 data). Cases were mapped to Anatomical Therapeutic Chemical (ATC) classifications and MedDRA High-Level Group Terms (HLGTs) to ensure diversity.
Expert Gold Standard: Four assessors (three MSc candidates and one medical doctor) evaluated the 723 cases using the Naranjo Algorithm, providing both numeric scores and reasoning text. Disagreements were resolved via consensus to establish a human gold standard.
Automated Assessment: OpenAI's GPT-5.2 was evaluated using Chain-of-Thought (CoT) prompting. The model generated scores and reasoning for all Naranjo questions.
Metric Development: Four composite, GP-compatible metrics were engineered to serve as optimization objectives:
- Weighted Cosine Similarity (WCS): Combines reasoning similarity and score agreement with equal weighting.
- Information-Weighted Agreement Score (IWAS): Weights binary score agreement by normalized Shannon entropy, down-weighting questions with no score variability.
- Entropy-Weighted Agreement and Cosine Similarity Score (EWACS): Integrates reasoning similarity with entropy-based weighting, retaining the cosine component for informative questions.
- Consensus-Weighted Cosine Similarity (CWCS): A bounded, multiplicative metric designed specifically for GP stability, combining soft score agreement with rescaled reasoning similarity.
Bayesian Optimization: A Gaussian Process surrogate model with a Matérn 5/2 kernel and Probability of Improvement (PoI) acquisition function was used to search the temperature space ( $T \in [0, 2]$ ). Experiments included grid sweeps, within-case optimization, and cross-validation using a Tree-structured Parzen Estimator (TPE).

Key Contributions

Metric Formulation: The paper introduces four novel metrics tailored to the constraints of pharmacovigilance data. It identifies EWACS as the optimal metric for this context, demonstrating that entropy-weighting is necessary to prevent optimization objectives from being dominated by questions with zero expert score variability (questions 6–9).
Hyperparameter Investigation: It is the first study to investigate inference-time temperature optimization for LLM-based causality assessment, moving beyond static baseline configurations.
Performance Benchmarking: The study establishes a new baseline for GPT-5.2, showing it outperforms previous biomedical LLMs (e.g., MedLlama3-8B) on the Naranjo algorithm, even before optimization.

Results

Baseline Performance: At baseline ( $T=0$ ), GPT-5.2 achieved 74.1% agreement on Question 5 (alternative causes) and 65.4% on Question 10 (objective evidence), outperforming prior biomedical models. However, overall classification agreement was 45.0%.
Entropy Analysis: Analysis of the 723 cases revealed that Questions 7–9 had zero expert entropy (uniform scores), and Questions 6–9 had near-identical expert reasoning templates. Consequently, Questions 5 and 10 were the only informative targets for optimization.
Temperature Effects:
- Population Level: No systematic population-level effect of temperature on agreement was found ( $\beta \approx 0, p > 0.9$ ). The relationship between temperature and output quality is not universal; a single optimal temperature does not exist for all cases.
- Case-Specific Optimization: Despite the lack of a universal optimum, EWACS-guided Bayesian optimization improved overall causality classification agreement from 45.0% to 72.0% (+27 percentage points).
- Targeted Gains: The improvement was most pronounced for "Doubtful" cases (+42.9 pp) and "Possible" cases (+18.5 pp), effectively correcting systematic biases where the baseline model ( $T=0$ ) over-classified cases as "Probable."
Metric Comparison: While CWCS provided a smoother response surface for the GP, EWACS yielded superior classification results because it better aligned with the specific endpoint of Naranjo classification by leveraging entropy to focus on informative questions.

Significance and Claims
The authors claim that the study demonstrates the viability of case-specific temperature optimization as a strategy to enhance LLM-assisted pharmacovigilance, even in the absence of a universal temperature optimum. The findings suggest that LLM performance in this domain is driven primarily by the content of the case narrative rather than the stochasticity of generation. By directing the optimization search at the individual case level, the framework can correct systematic reasoning biases (e.g., over-estimating temporal plausibility or under-estimating objective evidence) that plague deterministic baselines.

The paper posits that the performance gap previously attributed to the need for domain-specific training has been largely closed by advances in general-purpose LLMs (GPT-5.2 outperformed specialized biomedical models). The proposed framework is presented as a model-agnostic approach that can be extended to other hyperparameters and richer data sources (such as clinical trial data where therapy timelines and dechallenge records are present), offering a pathway toward hybrid decision-support workflows where automated tools handle clear-cut cases and flag ambiguous ones for human review.