Cost-Efficient Large Language Model-Assisted Title­­ and Abstract Screening for Systematic Reviews: Method Evaluation and Validation

Every year, the world produces a staggering flood of new scientific papers. In fields ranging from medicine to environmental health, researchers are publishing more studies than ever before, creating a deluge of information that no single team of humans can possibly read in a lifetime. To make sense of this chaos, scientists rely on a rigorous process called a systematic review. This is not a casual glance at the literature; it is a methodical hunt for every single study that answers a specific question, such as whether a certain chemical causes harm or if a new drug works better than an old one. The first, and often most exhausting, step in this hunt is screening. Researchers must read the titles and short summaries of thousands of articles, deciding instantly which ones are worth keeping and which should be thrown away. If a relevant study is missed at this stage, the entire review could be flawed, leading to incorrect conclusions about health and safety. For decades, this has been a job done entirely by human eyes, a slow, expensive, and labor-intensive bottleneck that delays vital answers.

A team of researchers from the Federal Office for Radiation Protection in Germany has now tested a new way to tackle this problem using artificial intelligence. They explored whether large language models—advanced computer programs capable of understanding and generating human language—could act as a first filter, reading these thousands of titles and abstracts to decide which ones deserve a human's attention. The goal was not to replace human reviewers, but to act as a powerful assistant that could do the heavy lifting, removing the vast majority of irrelevant papers while guaranteeing that no important study was ever accidentally thrown away. The researchers treated this like a scientific experiment, building and testing dozens of different ways to ask the computer to do the job, carefully measuring how well each method worked before trusting it with real data.

The team began by creating a controlled testing ground. They generated 100 synthetic summaries of scientific papers, half of which were designed to meet the strict criteria for inclusion and half that were designed to fail. Using this small set, they tried out a wide variety of strategies. Some strategies asked the computer to give a simple yes or no answer. Others asked it to break the decision down, evaluating five specific parts of a study's design separately: the people involved, the exposure or treatment, the comparison group, the outcome measured, and the type of study. They also tested whether giving the computer a few examples of good and bad papers before asking it to judge new ones would help, and whether asking the computer to explain its reasoning for each decision would improve its accuracy. They ran these tests on both open-source models, which can be run on local computers, and proprietary models from major technology companies, comparing their speed, cost, and accuracy.

After narrowing down the best approaches, the researchers moved to a more rigorous phase, testing their top candidates on three real-world systematic reviews that had already been completed by human experts. These reviews focused on the health effects of radiofrequency electromagnetic fields, a topic the research team knows well. Here, the computer's job was to look at the thousands of titles and abstracts from the original searches and decide which ones the humans had ultimately kept. The results were striking. The most successful workflows were able to identify nearly every single study that the humans had included, achieving a sensitivity rate of over 96 percent. This means that if there were 100 relevant studies hidden in a pile of thousands, the computer would find at least 96 of them. At the same time, the system was able to correctly identify and discard the vast majority of irrelevant papers, reducing the number of records that a human would need to read by more than 80 percent.

The study also revealed that the size of the computer model mattered less than the way it was asked to work. The researchers found that the most effective approach was not to use the largest, most expensive models available, but to use smaller, more cost-efficient models that were guided by a specific method. This method involved asking the model to look at the five key parts of a study's design individually and to provide a brief explanation for why it accepted or rejected each part. This step-by-step reasoning, combined with a few examples of what a good or bad study looks like, allowed even the smaller, cheaper models to perform with near-human accuracy. The researchers discovered that breaking the task into five separate questions for the computer to answer in parallel actually made it less accurate, suggesting that the computer works best when it considers the whole picture at once, even if it is analyzing the parts in detail.

When the team took their best-performing workflows and tested them on eight completely different systematic reviews, covering topics from cancer research to arthritis treatment, the results held up. The system consistently found almost all the relevant studies while filtering out the noise. The cost of running this process was remarkably low, amounting to roughly one US dollar for every 1,000 records screened. This suggests that the technology is not just a theoretical possibility but a practical tool that could be used by research teams with limited budgets. The researchers noted that the system is not perfect; it sometimes struggled with papers that had very little information, such as those with only a title and no abstract. In these cases, the system wisely flagged the paper for a human to check, ensuring that nothing important was lost.

The study concludes that artificial intelligence can now serve as a highly reliable partner in the scientific process. By using a carefully designed workflow that asks the computer to think through the criteria step-by-step and explain its choices, researchers can dramatically cut down the time and effort required to review the literature. This does not mean humans are obsolete; rather, it means they can focus their energy on the papers that truly matter, confident that the computer has already done the initial sweep. The work provides a clear, transparent framework for how this technology can be used responsibly, offering a path forward for faster, more efficient evidence synthesis in a world where scientific knowledge is growing faster than ever before.