Label-free, low-cost AI ranking for systematic review screening

Every year, doctors and researchers rely on systematic reviews to make sense of the vast ocean of medical studies. These reviews are the bedrock of clinical guidelines, telling practitioners which treatments work and which do not. However, creating one is a grueling, slow process that can take over a year from start to finish. The most exhausting part is not the analysis, but the screening: a team of human experts must read thousands of titles and abstracts, deciding one by one whether each study is relevant enough to read in full. It is a task that demands intense focus and consumes a massive amount of human time, often delaying life-saving information from reaching the people who need it.

For years, scientists have tried to use computers to help with this sorting, hoping to show the most promising studies first so humans can stop reading sooner. But these tools usually require a human to label hundreds of examples before they learn how to work, and they struggle to know exactly when it is safe to stop reading. Recently, powerful new artificial intelligence models have emerged that can understand text without needing any prior training on specific medical topics. The question researchers faced was whether these new tools could act as a cheap, fast, and accurate filter to organize these massive lists of studies before a single human ever touches them.

In a recent study, a team at Indiana University School of Medicine put a new type of artificial intelligence, called Jev, to the test against this very challenge. They did not ask the AI to read the studies and make a final decision; instead, they asked it to simply rank them. The goal was to see if the AI could arrange a list of nearly 170,000 records from 128 different medical reviews so that the most relevant ones appeared at the very top, without the AI ever seeing a single human-labeled example from those specific reviews. The researchers compared this new tool against older, free software and against some of the most advanced, expensive generative AI models available.

The results showed that the new AI was remarkably effective at its job. When the researchers let the AI sort the records, it placed the relevant studies much higher on the list than the best free software could. In fact, it outperformed the free tools in every single review they tested. When compared to the most powerful generative AI models, the new tool came very close, ranking just slightly behind the top performer but doing so at a fraction of the cost. While the most advanced models cost roughly fourteen dollars and sixty-five cents to process a thousand records, this new tool cost only two cents for the same task. It was not just cheaper; it was also faster and more accurate than other mid-tier models that cost significantly more.

The researchers then explored how this ranking could be combined with human work to save time. They found that if humans started reading the list in the order the AI suggested, they would find the relevant studies much faster than if they read in a random order or used older sorting methods. By using the AI's ranking as a starting point and then adding a human's feedback as they went, the team calculated that they could save a substantial amount of reading time while still finding almost all the important studies. However, the study also revealed a crucial limitation: while the AI could sort the list beautifully, it could not reliably tell the team exactly when to stop reading.

Because the AI's scores were not perfectly calibrated to predict the exact chance of a study being relevant, setting a fixed cutoff point to stop reading proved risky. In a few cases, using a simple score threshold caused the team to miss important studies that should have been included. The researchers found that the only way to stop safely was to continue reading until a statistical check confirmed that they had likely found all the relevant studies. This means the AI is an excellent tool for organizing the work and showing humans where to look first, but it cannot yet replace the human judgment needed to decide when the search is truly complete.

Ultimately, this work demonstrates that a low-cost, label-free AI can dramatically improve the efficiency of medical research screening. It offers a way to get a high-quality reading order from the very first record, saving teams from having to read thousands of irrelevant studies before finding the few that matter. While it does not solve the entire problem of when to stop, it provides a powerful, inexpensive starting point that makes the human work that follows much more focused and effective. For the field of evidence-based medicine, this represents a significant step toward making the creation of clinical guidelines faster and less burdensome, ensuring that the best medical knowledge reaches patients sooner.