Establishing AI State of Control: An Error Taxonomy and Assurance Framework for Transformer-Based Probabilistic Systems in Regulated Life Sciences

In the high-stakes world of pharmaceutical manufacturing, where a single error can compromise patient safety, companies rely on strict rules to ensure every step of a drug's creation is perfect. For decades, these rules have been built on the idea of determinism: if you put a specific set of ingredients into a machine, you get the exact same result every time. If something goes wrong, engineers can trace the failure to a broken gear or a misaligned sensor. But a new kind of technology is changing the landscape. Large language models, the same type of artificial intelligence that can write poetry or summarize complex reports, are beginning to enter these regulated environments. Unlike the old machines, these systems are probabilistic. They do not calculate a single correct answer; instead, they predict the most likely next word in a sentence based on patterns they have learned. This means that even when given the exact same instructions, the system might produce a slightly different, and potentially incorrect, output. The challenge for regulators and quality experts is that the old tools for finding and fixing mistakes were designed for machines that never guess, not for systems that constantly estimate.

A researcher named Kayla Britt has proposed a new way to manage this uncertainty, specifically for these AI systems used in life sciences. The core problem she addresses is that current quality systems lack the language to describe what goes wrong when an AI makes a mistake. If a traditional software program fails, it is usually a clear "software error." But when an AI hallucinates a document that looks real but isn't, or ignores a crucial instruction because it was confident it knew better, the old labels don't fit. This confusion makes it impossible to properly investigate the root cause or fix the problem. To solve this, Britt introduces a new classification system, a two-dimensional map that breaks down AI failures into two specific categories: the type of error that occurred and the exact point in the system's architecture where it started.

Imagine a grid where one side lists six different ways an AI can fail, such as making up facts, misinterpreting a context, or being overconfident in its answer. The other side lists six different places where the failure could have originated, such as the data the model was trained on, the system that retrieves information, or the way the AI interacts with human users. By crossing these two lists, the framework creates thirty-six specific "cells" where any given mistake can be logged. This allows a quality engineer to say not just that the AI failed, but precisely that it made a "fabrication error" (inventing a fact) that originated in the "model inference" stage (the moment the AI generated the text). This level of detail transforms a vague complaint into a specific diagnostic tool, enabling teams to understand exactly how the system broke down.

The paper goes further by acknowledging that not all AI errors can be completely eliminated. Because these systems are built on probability, some mistakes are inherent to their design and cannot be engineered away entirely. The framework introduces a concept called an "irreducibility gate" to separate errors that can be fixed with better coding or data from those that must be managed by limiting where and how the system is used. If a failure is deemed "irreducible," the solution is not to try to force the AI to be perfect, but to restrict its scope so that even if it makes a mistake, the consequences are contained. For example, if an AI is prone to inventing medical citations, the system might be designed to only allow it to summarize text that has already been verified by a human, rather than letting it generate new citations on its own.

To test this approach, the author walks through a practical scenario involving a pharmaceutical company using an AI to summarize manufacturing records. In this case, the AI correctly identified a temperature problem in a batch record but then invented a false reason for it, claiming a valve had malfunctioned when the record actually said it was a human error. Using the new taxonomy, the team could pinpoint that this was a "fabrication" error originating from the "model inference" layer, rather than a simple software glitch. This specific diagnosis led to a targeted solution: adding a verification step that checks if the AI's generated text matches the facts in the source document before showing it to a human. This is a more effective fix than a generic "check the software" order.

The proposed framework also aligns these new methods with the existing three-stage process used by the US Food and Drug Administration to validate manufacturing processes. Instead of trying to prove the AI will never fail, the new method focuses on proving that the AI's failures stay within a known, safe boundary. This involves testing the system with many variations of the same input to see how much the output varies, ensuring that the variation is small enough to be safe. If the system stays within these bounds, it can be approved for use, provided that humans are still in the loop to catch the rare errors that slip through.

This work suggests that the future of safe AI in medicine does not lie in trying to make these probabilistic systems behave like deterministic machines, which is likely impossible. Instead, it lies in accepting their nature, mapping their specific weaknesses with precision, and building controls that work with their limitations. By giving quality teams a clear vocabulary and a structured way to categorize failures, the framework provides a path forward for using powerful new tools without compromising the safety of the drugs we rely on. The paper concludes that while the technology is still evolving, having a standardized way to define, log, and manage these errors is the essential first step toward keeping them under control.