Deployment protocol and base model jointly determine residual risk in rule-based clinical LLM supervision

Imagine you've just built a super-smart robot assistant, one that can read medical charts and suggest treatments. You're excited, but you also know that even the smartest robots can sometimes make up facts, give dangerous advice, or get confused by tricky questions. This is the world of Clinical Large Language Models (LLMs): powerful AI tools designed to help doctors, but which carry a real risk of making mistakes that could hurt patients. To keep things safe, scientists have been building "guardrails"—digital safety nets that check the robot's answers before anyone sees them. But here's the big question: Does a safety net work the same way on every robot? And what happens if you take away the human backup plan? This paper dives into exactly that, testing whether a specific set of rule-based safety checks can keep different AI models safe, and how much risk remains if we try to let them run completely alone.

The researchers set up a digital "sentinel"—a strict, rule-following referee made of five different gates—to watch over five different AI models (ranging from smaller open-source ones to massive, top-tier commercial models). They tested these models on two types of medical scenarios: made-up cases designed to trick the AI, and real-life cases pulled from hospital records. They ran the tests in two very different modes. The first mode, "Handoff-On," is like having a human supervisor standing right next to the robot. If the sentinel catches a bad answer, it blocks it and sends it to a human doctor to fix or review. The second mode, "Handoff-Off," is the "autonomous" mode: the robot has to do the job entirely on its own. If the sentinel catches a bad answer here, there's no human to save the day; the original, potentially dangerous answer might still get through, or the system just has to deal with the mess.

The results were a mix of "great news" and "caution, proceed with care." When the human supervisor was present (Handoff-On), the safety net worked incredibly well for all the models. No matter which AI they tested, the rate of dangerous answers actually reaching the user stayed below 6%. It was like having a bouncer at a club who caught almost every troublemaker. However, the cost of this safety varied wildly. Weaker models (like Qwen 7B and Llama 8B) needed to be stopped and sent to a human for help in nearly half of the cases. Stronger models (like Sonnet 4.5 and Gemma 4) barely needed any help at all, letting over 99% of their answers pass through directly.

But when they turned off the human supervisor (Handoff-Off), the story changed completely. The safety of the system suddenly depended entirely on which robot you were using. The strongest models stayed safe, with dangerous answers remaining very low (around 0% to 2%). But the weaker models? Their safety collapsed. Without a human to catch them, the rate of dangerous answers jumped to between 13% and nearly 50%. The paper found that the key to this difference was how well the AI could "self-correct." When the sentinel told a strong model, "Hey, you made a mistake, try again," the strong model usually fixed it. The weaker models, however, often couldn't fix their own mistakes, especially if they messed up the basic format of their answer.

The researchers also tested what happens if you try to "fine-tune" (re-train) a model to make it better at specific tasks. Surprisingly, this sometimes made the model worse at following safety rules, causing it to break its own formatting rules. Yet, even in these cases, the rule-based sentinel was tough enough to catch the errors and keep the final output safe. They even tested a massive 70-billion-parameter model on nearly 3,000 real-world scenarios from three different hospitals, and it confirmed the trend: with a human backup, it was perfectly safe; without one, it was still very safe, but not quite as perfect as the top commercial models.

One final, crucial experiment involved a human doctor reviewing the AI's final answers. The doctor found that while the safety rules caught obvious errors, they missed some subtle but dangerous mistakes in medical reasoning. For example, the AI might follow all the safety rules perfectly but still suggest the wrong drug dosage because it misunderstood a lab result. This proves that while the rule-based sentinel is a fantastic safety net for catching structural errors, it isn't a substitute for a doctor's brain. The paper concludes that if we want to use these AI tools in hospitals, we need to pick the right model for the job and be honest about whether we have a human ready to step in. You can't just assume a safety net works the same way for every robot, and you definitely can't let the weakest robots run the show alone.