Zero-To-Few Shot Large Language Model Performance in Systematic Review and Meta-Analysis Automation: A Validation Study Across All Major Steps Of Five Published Systematic Reviews

In the world of medical research, the most trusted way to answer a question is to gather every single study ever written on that topic, read them all, and combine their findings into one clear picture. This process is called a systematic review. It is the gold standard for deciding what treatments work and which do not, forming the backbone of modern clinical guidelines. However, creating these reviews is a monumental task. It requires teams of researchers to spend months or even years searching through millions of articles, reading abstracts to decide which ones matter, extracting specific numbers from the full texts, and checking the quality of the research to ensure it is trustworthy. The process is so labor-intensive that it can cost hundreds of thousands of dollars and delay the arrival of life-saving information to doctors and patients.

Recently, a new kind of computer program known as a large language model has emerged. These are systems trained on vast amounts of human text that can read, understand, and generate language in ways that often mimic human intelligence. Researchers have begun to wonder if these tools could take over the heavy lifting of creating systematic reviews, turning a months-long ordeal into a matter of hours. The hope is that these machines could handle the repetitive, data-heavy parts of the work, freeing human experts to focus on the complex decisions that require deep judgment. But for this to happen, the machines must be accurate enough to be trusted with the lives of patients.

A team of researchers at the Université de Montréal and McGill University set out to test this idea with a very practical approach. They did not try to build a complex, fully automated robot that runs the entire review process from start to finish. Instead, they asked a specific, advanced version of a large language model to perform each major step of a systematic review, one by one, using only simple instructions and the data from five previously published medical reviews. The goal was to see if a standard user, without any special training in artificial intelligence, could get the computer to do the work as well as a human team. The researchers focused on five distinct stages: creating the search terms to find articles, screening the titles and abstracts to pick relevant studies, extracting specific data points from the full texts, assessing the quality and potential bias of the studies, and finally, combining the data to produce the final results.

The results revealed a story of surprising strengths and clear limitations. When it came to pulling specific numbers and facts out of the text of medical articles, the computer model was remarkably good. In fact, it outperformed the original human authors who had written the reviews. When the researchers compared the computer's extracted data against a new, highly rigorous standard created by expert reviewers, the machine was correct about 93 percent of the time. It was also faster, taking only minutes to do what took humans hours. Similarly, when asked to judge the quality of the studies and identify potential flaws or biases, the computer again performed better than the humans, with an accuracy of 83 percent. The researchers noted that when the computer made a mistake in judging quality, it tended to be overly cautious, flagging a study as risky when it might have been safe, which is often a safer error to make than missing a real problem.

However, the machine struggled significantly with the earlier stages of the process, where it had to make judgment calls based on incomplete information. When the computer was asked to generate the search strings—the specific lists of words used to find articles in medical databases—it found about 85 percent of the relevant articles that humans had found. This was a decent result, but it meant it missed a significant portion of the literature. The situation worsened when the computer tried to screen the full text of the articles to decide which ones to include in the final review. In this step, the model missed more than half of the important articles that the human reviewers had correctly identified. While the computer was very good at knowing what not to include, it was poor at finding the right things to keep. This suggests that while the machine can be a powerful tool for reading and organizing data, it is not yet ready to replace humans in the initial, critical phases of deciding what research is relevant.

The final step involved combining the data to calculate the overall results of the review. Here, the computer produced numbers that were very close to the human results, though not always exactly identical. The differences were small and likely due to how the machine interpreted ambiguous data or handled minor calculation details. The researchers found that the computer could generate these results in seconds, a task that would take a human much longer. However, the study highlighted that getting the computer to do this perfectly required very specific, detailed instructions about how to handle missing data and which statistical methods to use. Without this careful guidance, the results could drift slightly from the truth.

The study concludes that the path to automating medical research is not a single leap, but a series of steps where different tools are needed for different jobs. For tasks that involve reading and extracting facts, such as pulling data from tables or judging study quality, these computer models are already superior to humans and can be used immediately to speed up the work. But for the more subjective tasks, like deciding which articles to search for or which full texts to keep, the machines are not yet reliable enough to work alone. The researchers suggest that the most effective approach for non-experts is a hybrid model: let the computer handle the heavy lifting of data extraction and quality checks, while keeping human experts in the loop to guide the search and make the final screening decisions. This partnership could dramatically reduce the time and cost of producing medical evidence without sacrificing the accuracy that patients depend on.