Validation of a two-stage automated screening pipeline for medical systematic reviews: a stratified concordance study with three independent human reviewers, using PABAK and Gwet's AC1

Technical Summary: Validation of a Two-Stage Automated Screening Pipeline for Medical Systematic Reviews

Problem Statement
Systematic reviews (SRs) face a growing bottleneck in title–abstract screening due to the exponential expansion of biomedical literature. While Large Language Models (LLMs) offer a potential route for automation, their validation in this domain is often compromised by inappropriate metrics. Standard measures like Cohen's kappa are mathematically unstable under the low-prevalence and unbalanced class distributions typical of SRs. Furthermore, existing validation studies frequently lack a combination of critical design elements: a hybrid two-stage pipeline (rules + LLM), a three-way classification scheme including a separately sampled "uncertain" class, stratified sampling, and the use of prevalence-robust agreement coefficients (PABAK and Gwet's AC1). Additionally, the validity of comparisons is often questioned when the pipeline designer also serves as a human validator.

Methodology
This study presents a prospective agreement study validating a two-stage automated screening pipeline against three independent human reviewers within a medical systematic review context (High on-treatment Platelet Reactivity, aspirin resistance, and carotid plaque).

• Dataset: 904 records retrieved from multiple databases (PubMed, Scopus, Europe PMC, OpenAlex, arXiv).
• Pipeline Architecture:
  • Stage 1 (Rule-Based): A Python script using PICO-derived keyword heuristics (no ML/LLM) classifies records into INCLUDE, EXCLUDE, or UNCERTAIN.
  • Stage 2 (LLM Adjudication): Records labeled UNCERTAIN by Stage 1 are processed by DeepSeek V4 Pro (zero-shot, temperature = 0) via a fixed prompt to resolve them into INCLUDE or EXCLUDE.
• Validation Design:
  • Sampling: A stratified sample of 185 records was drawn: 50 from each of the three Stage 1 strata (INCLUDE, EXCLUDE, UNCERTAIN) and 35 from the Stage 2 LLM output.
  • Reviewers: Three physicians judged the sample independently and blind to automated labels:
    1. E.P. (Pipeline designer, familiar with the corpus).
    2. D.C. (External to the project).
    3. P.P.B. (External to the project).
  • Metrics: The study utilized sensitivity, specificity, PABAK, and Gwet's AC1 (with $q=3$) to measure concordance. It also reported Work Saved over Sampling (WSS) and Number Needed to Read (NNR) in response to editorial guidance for ML/GenAI evaluation in SRs. Majority labels (2 of 3) were computed as a supplementary panel but did not replace pairwise tables as the primary reference for sensitivity.

Key Contributions
The paper claims to be the first study to simultaneously combine five specific methodological dimensions:

1. A medical domain application.
2. A hybrid two-stage pipeline (rules + LLM).
3. A three-way classification with a separately sampled uncertain class.
4. Stratified sampling with three independent human reviewers.
5. Dual reporting of prevalence-robust coefficients (PABAK and Gwet's AC1) alongside workload metrics.

The study explicitly addresses the limitation of "designer-as-validator" by including two external reviewers and separating the analysis of the primary exclusion decision from the secondary fine-classification of included records.

Results

• Primary Target (Stage 1 Automatic Exclusion): The pipeline demonstrated near-perfect concordance with human reviewers on the decision to automatically exclude records.
  • Stage 1-EXCLUDE Stratum: The pipeline excluded 670/904 records (74.1%).
  • Agreement: PABAK and AC1 were 1.000 versus E.P. (48/48) and D.C. (50/50). Versus P.P.B., agreement was PABAK = 0.940 / AC1 = 0.959 (48/50 EXCLUDE, 2 UNCERTAIN).
  • Human Agreement: Inter-human agreement was highest between the two external readers (PABAK = 0.854) and lower when E.P. was involved, suggesting a potential familiarity effect, though this was not a measured variable.
• Secondary Target (Fine Classification/INCLUDE): The pipeline performed poorly on identifying "INCLUDE" records.
  • Overall Sensitivity: 0.800 vs. E.P., 0.500 vs. D.C., and 0.833 vs. Berti.
  • False Negatives: The pipeline missed human-identified INCLUDE records, particularly in Stage 2. For the four records where a majority of humans agreed on "INCLUDE," the machine labeled them as UNCERTAIN or EXCLUDE (0/4 recovery).
  • Stage 2 Performance: The LLM adjudication of the 94 uncertain records resulted in 11 INCLUDE and 83 EXCLUDE, but sensitivity against human labels remained low.
• Workload Metrics: The WSS of 74.1% reflects the proportion of records removed by the Stage 1 filter, validated by the sampled EXCLUDE stratum.

Significance and Claims
The paper modestly defines its scope and significance:

• Validated Capability: The study validates the Stage 1 automatic-exclusion decision as a reliable method to remove approximately 74% of records (the "obviously off-topic" papers) from a systematic review workflow. Three independent physicians confirmed this exclusion slice.
• Unvalidated Capability: The study explicitly states that fine classification of INCLUDE records is not validated. Humans themselves showed low agreement on which residual records were "INCLUDE" (triple intersection was only one record), and the machine failed to recover the few records humans agreed were included.
• Methodological Implication: The results suggest that while automation can effectively handle the bulk exclusion task, the residual task of identifying relevant studies remains a human, domain-experienced task. The study cautions that the results are specific to this PICO and pipeline and do not automatically generalize to other topics.
• Metric Recommendation: The authors reinforce the necessity of using PABAK and Gwet's AC1 over Cohen's kappa for SR screening validation due to prevalence instability.

In conclusion, the paper establishes that a rule-based filter can safely offload the majority of screening workload in this specific medical context, but it does not claim that LLMs can currently replace human judgment for the critical task of identifying relevant studies.