Prospective validation of NeutrinoReview for LLM-assisted systematic review screening: an indoor air quality case study

Technical Summary: Prospective Validation of NeutrinoReview for LLM-Assisted Systematic Review Screening

Problem Statement
Systematic reviews are critical for synthesizing evidence, but the screening process—specifically title and abstract screening—imposes a substantial workload. To mitigate human error and selection bias, standard guidelines recommend independent screening by at least two reviewers. However, this "double-screening" requirement significantly increases the time and cost of conducting reviews. While Large Language Models (LLMs) offer potential for automating or pre-filtrating this process, existing evidence is largely retrospective. Retrospective evaluations suffer from limitations such as potential data contamination (if the model was trained on the benchmark data) and the inability to prevent post-hoc optimization of prompts or thresholds. There is a lack of prospective evidence from live review workflows demonstrating that LLM-based pre-filtration can reduce workload without excluding records that are ultimately deemed relevant by human consensus.

Methodology
This study conducted a prospective validation of NeutrinoReview, an open-source research prototype for LLM-assisted screening, in parallel with an ongoing systematic review of measured indoor air pollutant concentrations in Italy.

• Study Design: The evaluation was designed as a non-interventional, prospective study. The NeutrinoReview screening outputs and the evaluation protocol were locked and uploaded to a repository (Zenodo) before the manual consensus ground truth was available to the analysis team. This "locked-output" design prevents bias and ensures the evaluation reflects genuine screening behavior rather than optimized responses to known labels.
• Data Source: The source review yielded 1,383 deduplicated candidate records. The final performance analysis dataset consisted of 1,344 records (112 finally included, 1,232 finally excluded) based on the manual consensus of three human reviewers.
• LLM Configuration: Screening was performed using a self-hosted LLaMA 3.1-8B instruction model. Six distinct screening configurations were evaluated:
  1. 5-tier Algorithm Variants:
     • MaxSensitivity: Excluded only the lowest relevance category (Category 5).
     • Balanced: Excluded Categories 4 and 5.
     • Full Automation: Retained only the top two categories (1 and 2) for human review.
  2. CAL-X Algorithm Variants: This algorithm uses next-token likelihood scores to retain low-confidence exclusions. Three thresholds were tested corresponding to calibrated sensitivity targets:
     • CAL-95: Threshold 63 (95% sensitivity target).
     • CAL-98: Threshold 80 (98% sensitivity target).
     • CAL-99: Threshold 89 (99% sensitivity target).
• Comparators: The performance of NeutrinoReview was benchmarked against:
  • The manual consensus ground truth (final inclusion status).
  • Simulated single-human screening workflows (individual reviewer decisions).
  • Simulated double-human screening workflows (pairs of reviewers).
• Metrics: Primary metrics included Sensitivity (proportion of finally included records retained) and Workload Reduction (proportion of records filtered out before human screening). Cost simulations were also performed based on observed screening times.

Key Results

• Sensitivity: All six NeutrinoReview configurations achieved 100% sensitivity against the manual consensus ground truth. No record that was finally included by the human reviewers was excluded by any of the LLM configurations.
• Workload Reduction: Workload reduction varied significantly by configuration:
  • CAL-99 (Most Conservative): Reduced workload by 37.1% (retained 846 records).
  • 5-tier Full Automation (Least Conservative): Reduced workload by 74.0% (retained 350 records).
  • Intermediate configurations (CAL-98, CAL-95, 5-tier Balanced, 5-tier MaxSensitivity) achieved reductions ranging from 45.6% to 66.4%.
• Human Comparator Performance:
  • Single-human workflows missed between 5 and 7 finally included records (Sensitivity: 93.8%–95.5%).
  • Double-human workflows missed between 0 and 3 finally included records (Sensitivity: 97.3%–100.0%).
  • NeutrinoReview's performance in this specific case was comparable to or better than the observed double-human workflows regarding missed records.
• Cost Implications: Simulations indicated that using NeutrinoReview as a pre-filtration step could save between CHF 571 and CHF 2,280 (at CHF 100/hour) depending on the configuration and the number of screeners involved, primarily by reducing the number of records requiring manual title/abstract screening.

Significance and Claims
The paper claims that this study provides prospective evidence that LLM-based pre-filtration can reduce the screening burden in systematic reviews without compromising the retrieval of relevant studies, at least within the specific context of this review and model implementation.

• Credibility of Findings: The authors emphasize that the "locked-output" design strengthens the credibility of the 100% sensitivity finding, as it rules out the possibility that the model or analysis was tuned to the known ground truth.
• Practical Application: The authors conclude that NeutrinoReview is most defensible as an adjunct to human screening rather than a replacement. Specifically, they recommend the CAL-99 configuration for cautious practical use, as it combines the prospective findings with stronger prior retrospective evidence for high sensitivity.
• Limitations and Scope: The paper explicitly states that these results should not be interpreted as general validation across all review topics, models, or settings. The zero missed records in this single case study does not guarantee safety in all future deployments. The authors note that the study was limited to title/abstract screening and did not account for full-text screening, conflict resolution, or infrastructure costs.
• Future Direction: The authors assert that further prospective evaluations and field tests across multiple reviews and domains are necessary before LLMs can be considered for unsupervised replacement of human screeners. Until such evidence is available, the tool should be used in supervised or parallel workflows.