A large language model-assisted workflow for generating a living evidence base for climate-sensitive foodborne disease

Technical Summary: LLM-Assisted Workflow for Living Evidence on Climate-Sensitive Foodborne Disease

Problem Statement
Climate change is altering environmental conditions that drive foodborne disease transmission, creating a rapidly expanding and heterogeneous evidence base. Traditional systematic reviews, while rigorous, are too slow and resource-intensive to keep pace with this growth. National-scale assessments, such as the UK Health Security Agency's 2023 climate-health review, require screening tens of thousands of records, a process that limits the ability of public health agencies to generate rapid, repeatable, and policy-relevant "living" evidence for adaptation strategies. While machine learning offers potential solutions, few studies have tested Large Language Model (LLM) workflows within operational public health settings or designed them specifically for living reviews.

Methodology
The authors developed and evaluated an LLM-assisted screening workflow embedded within the SysRev platform, utilizing GPT-4-Turbo as the auto-labeller. The study focused on foodborne pathogens of public health importance in England (e.g., Campylobacter, Salmonella, Norovirus, Cryptosporidium, Listeria, VTEC, Yersinia) and their links to climate drivers (e.g., temperature, precipitation, seasonality).

The workflow followed a hybrid human-LLM approach:

1. Data Collection: A structured PubMed search (2010–2023) yielded a dataset of abstracts.
2. Gold-Standard Creation: Three human reviewers independently screened an initial set of 365 abstracts against eligibility criteria (requiring human disease outcomes explicitly linked to climate/weather). Disagreements were resolved by consensus to create a gold-standard dataset.
3. Iterative Refinement: The auto-labeller was initially configured with prompts derived from inclusion criteria. The authors implemented an agent-driven refinement process:
   • The model screened the gold-standard set.
   • Misclassified abstracts (false positives/negatives) were sampled.
   • These examples were fed back to the LLM to update and refine the inclusion criteria prompts.
   • The process repeated until model performance stabilized.
4. Evaluation: The final model was tested on a separate set of 1,044 abstracts. Performance was measured using recall, precision, specificity, accuracy, and balanced accuracy, compared against human screening.

Key Results

• Model Performance: The refined inclusion model achieved 89.2% recall, 59.2% precision, 84.5% specificity, and 85.4% accuracy.
• Screening Output: Across the evaluation set, the model identified 436 studies (41.7%) as meeting the refined criteria.
• Post-Hoc Analysis: Abstracts excluded by the model but initially included by humans were re-evaluated against the final refined criteria and judged not to meet them. This suggests that the iterative refinement process improved the consistency of screening definitions, rather than the model simply missing valid studies.
• Evidence Characteristics: The included studies predominantly focused on rainfall, temperature, seasonality, and humidity. The most common pathogens were norovirus, Salmonella, Campylobacter, and Cryptosporidium. Geographically, studies were concentrated in Europe, Asia, and North America.

Key Contributions

• Operational Workflow: The study demonstrates a scalable, auditable, and reproducible workflow that integrates structured searching, human expertise, and iterative LLM refinement within a secure institutional environment.
• Iterative Prompt Engineering: Unlike static prompt evaluations, this work explicitly uses misclassified examples to update inclusion criteria, showing how model performance can be improved within an operational pipeline.
• Living Evidence Foundation: The approach addresses the limitation of static reviews by offering a method to generate evidence bases that can be rapidly updated as new literature emerges, supporting horizon scanning and climate-health risk assessment.
• Strategic Trade-off: The workflow prioritizes high recall (sensitivity) over precision, a deliberate trade-off in systematic review screening to minimize false negatives while retaining manageable volumes of false positives for downstream human review.

Significance and Claims
The authors claim that this LLM-assisted workflow provides a low-cost, scalable alternative to traditional manual screening, capable of reducing the manual burden while maintaining high sensitivity to relevant literature. The study positions the workflow not as a replacement for human judgment or full systematic reviews for causal inference, but as a tool to augment decision-making by narrowing candidate evidence sets and standardizing eligibility criteria.

The paper asserts that such systems are particularly vital for climate-sensitive infectious diseases, where non-stationary relationships and rapid evidence growth render static reviews quickly outdated. By supporting repeatable and auditable processes, this approach offers a foundation for "living" evidence systems that can inform surveillance priorities, risk assessments, and policy responses under uncertainty. The authors note that while formal time savings were not quantified, the workflow is expected to significantly reduce the number of records requiring full manual screening.

Limitations Acknowledged
The authors modestly note that the study focused on abstract screening rather than full-text assessment, which may limit generalizability to later synthesis stages. The evaluation was conducted on a single domain-specific dataset using one LLM model, and the observed precision indicates that human oversight remains necessary to filter false positives. Furthermore, the study addresses study identification but does not yet automate downstream tasks such as data extraction, risk-of-bias assessment, or meta-analysis.