Neither the Judge nor the Rules: Instrument-Validity Failure in Safety-Response Classification

In the rapidly evolving world of artificial intelligence, researchers are constantly trying to teach computers to be helpful without being harmful. To ensure these systems do not generate dangerous advice or toxic content, scientists must evaluate their outputs. For years, two main methods have dominated this safety checking. The first involves using one artificial intelligence to grade the work of another, acting as an automated judge. The second relies on hand-written computer rules, which scan text for specific keywords or patterns that suggest harm, functioning like a digital filter. The assumption behind the second method has been that because these rules are fixed and deterministic—meaning they always produce the same result for the same input—they are perfectly reliable and can be treated as the absolute truth. If a computer program follows a strict set of instructions, the logic goes, its verdict on safety must be correct.

A new study challenges this assumption by treating these safety tools not as infallible arbiters, but as instruments that can make mistakes, just like any other measuring device. The researcher, Yangyi Zhang from Georgetown University, decided to audit both the automated judge and the rule-based filter to see if they actually do what they claim to do. The investigation focused on a collection of 550 responses generated by various large language models when asked difficult or potentially harmful questions. To find out the truth, the researcher created a small, independent benchmark by manually reading and labeling 32 of these responses by hand, defining safety based on whether the model provided actionable harmful content. This human-verified set served as the ground truth against which the two automated tools were tested.

The results revealed a significant failure in the reliability of both instruments. The rule-based classifier, which had been widely trusted and used in previous research as the standard for safety, performed poorly. When compared against the human labels, this automated tool was correct on only about half of the cases. More critically, it failed to identify half of the genuinely unsafe responses, missing dangerous content entirely. At the same time, it falsely flagged six harmless responses as unsafe, misinterpreting polite refusals as dangerous deliveries. The automated judge, a smaller language model tasked with grading the responses, performed slightly better in overall accuracy but suffered from a different fatal flaw. It failed to catch any of the four truly unsafe responses in the test set, missing them all completely. Instead, it incorrectly labeled two safe refusals as unsafe.

The study also uncovered a hidden instability in the rule-based system that no one had noticed before. The researcher ran two different versions of the same rule set on the full collection of 550 responses. The first version identified 40 responses as unsafe, while the second, slightly revised version identified only eight. This fivefold difference occurred without any change in the actual content of the responses, proving that the count of unsafe items depends entirely on which version of the rules is loaded. Because the system is deterministic, it reproduces its own errors perfectly every time, making those errors invisible to anyone who only looks at the final numbers. The two automated tools agreed with each other on about 62 percent of the items, a figure that might look like strong consensus at first glance. However, this agreement was largely an illusion caused by the fact that most of the responses were safe. When the researcher adjusted for this imbalance, the actual agreement between the tools was negligible, showing that they were not truly validating each other but rather collapsing into a default prediction of safety.

The core finding is that neither the automated judge nor the rule-based filter is a neutral or valid measure of safety on its own. The rule-based classifier is not a perfect default; it is a measurement tool with a high and unmeasured error rate. The study concludes that the only way to trust these safety figures is to validate every instrument against a separate, human-verified benchmark before publishing any results. Researchers must report not just how often a tool agrees with another, but how well it catches the specific errors that matter, such as missing dangerous content. Without this independent check, the field risks building safety evaluations on foundations that are themselves flawed, mistaking the consistency of a broken tool for the truth.