Large Language Models for Automated AGREE II Quality Appraisal of Sepsis Clinical Practice Guidelines: A Methodological Comparative Study

In the high-stakes world of modern medicine, life-saving treatments are often guided by clinical practice guidelines. These are not casual suggestions but carefully constructed documents that synthesize the best available scientific evidence into clear instructions for doctors. They tell medical teams how to manage complex conditions like sepsis, a dangerous reaction to infection that can shut down organs and claim thousands of lives. However, the quality of these guidelines varies wildly. Some are built on rigorous, transparent methods, while others may lack depth or clarity. To sort the reliable from the unreliable, experts use a specific tool called AGREE II. This instrument acts like a detailed checklist, asking twenty-three specific questions about a guideline's structure, its evidence, and how practical it is for real-world use. Traditionally, filling out this checklist is a slow, expensive process that requires teams of highly trained specialists to read every word of a document and debate its merits. As the number of published guidelines grows faster than the number of experts available to read them, the medical community faces a bottleneck: how to ensure quality without being overwhelmed by volume.

This is where artificial intelligence enters the story, specifically a new generation of computer programs known as large language models. These systems, trained on vast amounts of text, have shown an ability to read, understand, and summarize complex information. Researchers wondered if these digital tools could take on the heavy lifting of grading medical guidelines, acting as a fast, automated first pass before a human expert steps in. A team of scientists set out to test this idea by pitting six different large language models against a panel of human experts. They did not ask the computers to guess the answers; instead, they fed them the full text of eleven real-world guidelines for treating sepsis and asked the models to score them using the same strict AGREE II checklist that the humans had already used. The goal was to see if the machines could think like the experts, or if they would stumble over the nuances of medical judgment.

The results revealed a relationship that is promising but far from perfect. The computer models were able to agree with the human experts to a moderate degree, capturing the general quality of the documents. However, they were not simply mimicking human thought; they were developing their own distinct patterns of error. The most striking finding was a consistent bias in how the machines evaluated the "applicability" of a guideline. This section of the checklist asks whether a document provides enough practical advice for doctors to actually use it in a busy hospital, considering things like cost, equipment, and local resources. The human experts generally gave these sections higher marks, recognizing the practical intent behind the recommendations. The computer models, by contrast, consistently gave these sections much lower scores. It appears the machines were looking for explicit, step-by-step instructions on how to implement a treatment and penalized the guidelines for not providing them, even when the guidelines were otherwise sound. They seemed to miss the subtle, real-world context that human doctors understand instinctively.

Conversely, the models tended to be overly generous when grading the "rigor of development." This part of the checklist examines how the guideline was created, looking for evidence that the authors followed strict scientific methods. The computers were very good at spotting keywords like "systematic review" or "evidence-based," and when they saw these terms, they awarded high scores. Sometimes, they awarded these scores even when the human experts felt the methods were not as strong as the language suggested. The machines were reading the surface of the text very well but occasionally missing the deeper truth of how the work was actually done. This created a strange dynamic where the computers were too harsh on the practical parts of a guideline and too lenient on the theoretical parts.

Beyond the specific scores, the study also looked at the speed and cost of using these different models. The researchers measured how long each computer took to read a guideline and how much digital "fuel" it consumed to produce an answer. One model, developed by a Chinese technology firm, stood out as the most efficient. It produced scores that aligned well with human experts, did so in just fifteen seconds, and used the least amount of digital resources. Another model from a major American tech company was slightly slower and more expensive to run but showed the least amount of bias, meaning its scores were the closest to the human average without leaning too far in one direction or the other. The study found that a larger, more powerful model was not necessarily better at this specific task; in fact, the most efficient model was not the one with the largest size or the most general medical knowledge. This suggests that for this kind of detailed, structured work, the ability to follow a specific set of rules matters more than raw size.

The researchers concluded that these artificial intelligence tools are not ready to replace human experts. If a hospital were to rely solely on a computer to decide which guidelines are good enough to use, they might reject excellent documents simply because the machine was too strict about practical details, or they might accept weak ones because the machine was fooled by fancy language. Instead, the best use for these models is as a triage system. They can quickly scan hundreds of guidelines, flagging the ones that look promising and highlighting the ones that might need a closer look. This would free up human experts to focus their time on the most difficult cases, particularly those guidelines that sit right on the edge of being acceptable. In this way, the technology acts as a powerful assistant, handling the volume and speed while leaving the final, nuanced judgment to the people who understand the reality of patient care. The study confirms that while machines can read the words, they still need humans to understand the meaning.