Evaluating a Locally Deployed 20-Billion Parameter Large Language Model for Automated Abstract Screening in Systematic Reviews

This study demonstrates that a locally deployed 20-billion parameter LLM, utilizing a sensitivity-enhanced prompting strategy, can significantly accelerate systematic review abstract screening with high accuracy and zero data privacy risks, though its performance varies by domain and is best used as a second screener alongside human experts.

The Gist

Imagine you are a librarian trying to find the perfect books for a very specific reading list. You have a mountain of 16,000 new books (abstracts) that just arrived, and you need to pick out the few dozen that are actually relevant. Traditionally, you'd need two librarians to read every single book cover-to-cover, argue about the tricky ones, and then have a head librarian make the final call. It takes weeks, it's exhausting, and by the time you finish, new books have already arrived.

This paper is about testing a super-smart robot librarian (an AI) to see if it can help speed up this process without missing any important books.

Here is the story of their experiment, broken down simply:

1. The Problem: The "Cloud" vs. The "Safe Room"

Most people use cloud-based AI (like asking a question on a website) to do this work. But for medical research, that's like reading your private patient notes out loud in a crowded town square. It's risky for privacy.

• The Solution: The researchers built their own "Safe Room." They installed a powerful AI (a 20-billion parameter model called GPT-OSS) directly on their own computers. This way, the data never leaves their building, ensuring total privacy and control.

2. The Strategy: "Better Safe Than Sorry"

The researchers gave the robot a very specific rule: "When in doubt, INCLUDE the book."

• Why? Imagine you are looking for a rare coin. If you accidentally throw away a coin that might be the one you want (a "False Negative"), you can never get it back. But if you accidentally keep a fake coin (a "False Positive"), you can just look at it later and throw it away.
• So, they told the AI to be overly cautious. It's better to bring 100 books to the human librarian and say, "Hey, check these," than to throw away one book that was actually important.

3. The Test: Three Different Libraries

They tested this robot on three different "libraries" (Systematic Reviews):

1. Tech Library: About AI in pediatric surgery. (Very clear rules: Does it use a robot? Yes/No.)
2. Digital Records Library: About AI in hospital records. (Also very clear rules.)
3. Emotion Library: About parents' stress and caregiver burden. (Very fuzzy rules: Is this "stressful"? It's hard to define.)

4. The Results: The Robot's Performance

The robot was incredibly fast. It did the work of a human in 5 hours that took a human 26 hours. That's nearly 5 times faster.

But how accurate was it?

• In the Tech Libraries: The robot was a superstar. It found 100% of the relevant studies. It didn't miss a single one.
• In the Emotion Library: It was good, but not perfect. It missed about 14% of the relevant studies.
  • Why? Because "stress" and "burden" are subjective. A robot is great at spotting concrete things like "surgery" or "software," but it struggles with fuzzy human feelings.

5. The Big Twist: The Robot Caught the Humans!

Here is the most interesting part. Usually, we assume humans are the "Gold Standard" and the robot is the student. But in this experiment, they had a blind expert judge every time the human and the robot disagreed.

The expert found that the humans were wrong 11 times!

• The humans had thrown away important studies that the robot correctly saved.
• The robot had thrown away 13 important studies that the humans correctly saved.

The Analogy: Think of it like two security guards at a gate. One guard is a human who sometimes gets tired and misses a VIP. The other is a robot that is hyper-vigilant but sometimes stops a delivery truck that was actually allowed in. When they work together, they catch each other's mistakes.

6. The Conclusion: Teamwork, Not Replacement

The researchers aren't saying "Fire the humans and let the robot do it all." They are saying: "Let's use the robot as a second pair of eyes."

• The New Workflow: A human reads the abstracts. The robot reads them too. If they agree, great! If they disagree, a senior expert steps in to decide.
• The Benefit: This keeps the quality high (because the robot catches what the human misses) but cuts the workload in half.

Summary

This paper proves that if you put a smart AI in a private "safe room" and tell it to be overly cautious, it can do the boring, time-consuming work of sorting research papers incredibly fast. It's not perfect at understanding human emotions, but it's amazing at spotting technical facts. The best approach isn't to replace humans, but to let the robot and the human dance together, catching each other's mistakes to ensure no important discovery is ever lost.

Technical Summary

1. Problem Statement

Systematic reviews (SRs) are fundamental to evidence-based medicine but are hindered by the resource-intensive nature of abstract screening. Traditional methodology requires at least two independent human reviewers to screen thousands of abstracts, a process that is slow, costly, and prone to delays that can render reviews outdated upon publication.

While Large Language Models (LLMs) offer a potential solution for automation, their adoption in research settings is limited by:

• Data Privacy & IP: Cloud-based APIs require sending sensitive data to third parties.
• Reproducibility: Lack of version control in proprietary cloud models.
• Performance Concerns: Uncertainty regarding sensitivity, particularly in complex or subjective domains.

2. Methodology

The study employed a methodological design to evaluate a locally deployed, open-source LLM across three distinct systematic reviews.

• Model & Deployment:
  • Model: GPT-OSS:20B (a 20-billion parameter open-source model).
  • Framework: Deployed locally using Ollama to ensure data never leaves the institutional environment, guaranteeing privacy and full version control.
• Datasets: Three independent systematic reviews totaling 16,646 abstracts:
  1. SR1: AI applications in pediatric surgical pathology (3,350 abstracts).
  2. SR2: LLM applications in electronic health records (4,326 abstracts).
  3. SR3: Parental stress/caregiver burden in surgically treated children (8,970 abstracts).
• Input/Output: The model received the article title, abstract, and specific inclusion/exclusion criteria. It output a structured JSON object containing a binary decision (Include/Exclude) and reasoning.
• Prompting Strategy: A sensitivity-enhanced prompt was used, explicitly instructing the model: "When in doubt, INCLUDE the abstract for further review." This prioritizes minimizing False Negatives (missing relevant studies) over False Positives, as False Positives can be filtered out later, whereas False Negatives are irreversible.
• Validation Protocol:
  • Blind Expert Adjudication: All cases where the LLM and the original human reviewer disagreed were reviewed by a senior expert blinded to the source of the decision.
  • Error Analysis: The study quantified False Negatives (FN), False Positives (FP), and crucially, Human Errors (cases where the expert sided with the LLM against the original human decision).

3. Key Contributions

• Local Deployment Framework: Demonstrates a viable workflow for running 20B parameter models locally, addressing privacy and reproducibility concerns that plague cloud-based AI solutions.
• Bidirectional Error Analysis: Unlike many studies that treat human screening as the absolute "gold standard," this study used expert adjudication to reveal that humans also make screening errors. The LLM successfully identified 11 relevant studies that human reviewers had incorrectly excluded.
• Domain-Specific Performance Benchmarking: Provides empirical data showing that LLM performance is highly dependent on the objectivity of inclusion criteria (technology-focused vs. psychosocial).
• Efficiency Metrics: Quantifies the speed advantage of AI screening over human reviewers.

4. Results

After expert adjudication, the performance metrics were as follows:

Metric	SR1 (Tech/Pediatric Surgery)	SR2 (Tech/EHR)	SR3 (Psychosocial)	Overall
Sensitivity	100%	95.7%	85.7%	91.7%
Specificity	97.9%	88.7%	99.99%	96.7%
Human Errors Detected	3	4	4	11
LLM Errors (FN)	0	2	11	13

• Efficiency: The LLM screened 3,350 abstracts in 5.58 hours, compared to 26 hours for human reviewers. This represents a 4.7-fold increase in speed, with an average processing time of ~6 seconds per abstract.
• Error Distribution:
  • The LLM missed 13 relevant studies (False Negatives) across the board.
  • The LLM correctly identified 11 relevant studies that the original human reviewers had missed.
  • The model performed exceptionally well on objective, technology-focused criteria (SR1 & SR2) but showed reduced sensitivity on subjective, psychosocial criteria (SR3).

5. Significance and Conclusion

• Human-AI Collaboration: The study concludes that neither humans nor AI are perfect; they make errors on different abstracts. Therefore, the optimal strategy is collaborative, not replacement.
• Proposed Workflow: The authors recommend a "Second Screener" model where:
  1. A human screens all abstracts.
  2. The LLM screens independently.
  3. Discordant cases are sent to expert adjudication.
     This maintains the rigor of dual screening while significantly reducing the workload.
• Limitations & Future Work:
  • Performance varies by domain; models struggle with subjective criteria (e.g., stress, burden).
  • The current model lacks confidence scores, making it difficult to flag "borderline" cases for human review.
  • Validation is needed across diverse institutions and for full-text screening stages.

Final Takeaway: Locally deployed LLMs with sensitivity-enhanced prompting are a powerful tool for accelerating systematic reviews, particularly in technical fields. However, they should currently be deployed as a safety net to catch human errors and reduce workload, rather than as a fully autonomous replacement for human reviewers.