When Verification Passes the Wrong Calculator: A Human-Adjudicated Study of a Guarded Local LLM Pipeline

In the world of modern medicine, doctors often rely on specialized tools to turn raw patient data into clear answers. These tools, known as clinical calculators, take specific numbers—like a person's weight, age, or blood test results—and apply a fixed set of rules to produce a score or a category that helps guide treatment. For years, these tools have been run by computers that follow strict, unchanging instructions, ensuring the math is always perfect. Recently, a new kind of computer program has emerged: large language models. These are systems trained on vast amounts of text that can read a patient's story, understand the context, and attempt to pick the right calculator and do the math for you. The hope is that these systems can act as intelligent assistants, bridging the gap between a messy medical narrative and a precise numerical result. However, a critical question remains: if a computer program says it has done the math correctly, does that mean it actually chose the right tool for the job in the first place?

A team of researchers at the University of South Florida set out to test this exact question using a specific type of computer system designed to be extra careful. They built a pipeline that acts like a guarded gatekeeper. When a patient case is presented, the system first decides which calculator to use. It then extracts the necessary numbers from the text, runs the calculation using a rigid, error-free computer program, and finally checks its own work by trying to represent the answer in a different way to see if the numbers still match. The researchers wanted to know if this internal check for consistency was enough to guarantee safety. Their study, which involved testing the system on over a thousand records and a separate set of real-world medical case reports, revealed a surprising and potentially dangerous flaw. The system could pass all its internal checks and produce a mathematically consistent answer even when it had selected the completely wrong calculator for the patient's condition.

The researchers evaluated their system, which they called CIBB-Guard, against a standard way of asking the computer questions directly. In a large test involving over one thousand records, most of which were synthetic or computer-generated, the guarded system performed significantly better. It produced the correct answer for about sixty percent of the cases, whereas the direct approach only succeeded in about ten percent. This improvement came from the system's ability to select a rule, extract data, and verify the math. However, the real test came when the researchers looked at cases where no calculator should have been used at all. They took seventy-seven de-identified medical case reports and had human experts classify them as either eligible or ineligible for the calculators being tested. In the cases where the calculators did not apply, the guarded system correctly refused to give an answer in twenty-one out of thirty-three instances. But in the remaining twelve cases, the system confidently provided an answer, and that answer passed its own internal verification checks.

This is where the core discovery lies. In those twelve inappropriate cases, the system had picked a calculator that did not fit the patient's story, yet the math it performed was internally consistent. The system checked its work by representing the numbers in an alternate form, and the two versions agreed. This agreement confirmed that the computer had executed the chosen rule correctly, but it did not confirm that the rule was the right one to use. In fact, in nearly all of these wrong answers, the internal check passed successfully. The study found that the system's own safety net, designed to catch calculation errors, was blind to the error of choosing the wrong tool entirely. When the system was tested on cases where a calculator was actually appropriate, it only managed to produce four correct answers out of forty-four, a result that was not statistically different from the direct approach, highlighting that the system still struggles with the complex task of matching a patient's story to the right rule.

The researchers emphasize that their findings do not mean the system is useless, but rather that it is incomplete. The study used a single, frozen version of a computer model and a specific library of twelve calculators. The results show that while the system can improve the accuracy of math execution and reduce random errors, it cannot yet determine if a calculator is applicable to a specific patient without human oversight. The fact that the system passed its internal checks on wrong answers suggests that relying solely on mathematical consistency is not enough to ensure clinical safety. The authors conclude that before such a system could be used in a hospital, there must be a separate, independent check to verify that the chosen calculator is actually appropriate for the case, rather than just checking that the math adds up. The work serves as a cautionary note that a computer can be perfectly consistent while being completely wrong about the context, and that in medicine, getting the right tool is just as important as doing the math correctly.