Operationalising CAPA for AI: Evidence Requirements and Limits from a Production Case Study
This paper proposes adapting regulated manufacturing quality disciplines—specifically deviation management, root-cause analysis, and CAPA—to govern AI systems by treating operational telemetry as quality records, demonstrating through a UK immigration-law case study that this approach effectively identifies systemic issues and operationalizes accountability, though it requires formal effectiveness verification to fully close the governance loop.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
When a factory makes medicine or a plane part, the rules are strict. If a machine produces a bolt that is slightly too short, or a pill that is mislabeled, that mistake is not just a glitch; it is a "deviation." In these regulated worlds, companies do not simply throw the bad item away and hope for the best. They stop the line, investigate exactly why the mistake happened, fix the machine, and then prove that the fix actually works before they start making things again. This cycle of finding errors, finding their true cause, and verifying the solution is a century-old discipline called quality engineering. It is designed to catch problems before they hurt people. Today, organizations are handing over similar high-stakes work to artificial intelligence. Lawyers use AI to sort legal documents, doctors use it to read scans, and banks use it to approve loans. But when these smart systems make a mistake, the old rules of quality engineering are often missing. We know how to fix a broken robot arm, but we are still learning how to fix a broken algorithm that is quietly filing the wrong documents for thousands of people.
A researcher named Musarat Kabir-Chisty decided to test whether the old, rigorous rules of factory quality control could be applied to a live, working artificial intelligence system. She looked at a real-world example: a document-automation system used by a UK immigration law firm. This system was designed to read thousands of legal papers, figure out what each one was, and file them into the correct digital folders for hundreds of different cases. The system was not perfect. During a specific period of migration and relaunch, it processed over 42,000 actions. The researcher treated the computer logs of this system like a factory quality record. Instead of looking at the code or the math behind the AI, she looked at the actual mistakes the system made and the corrections humans had to make. She wanted to see if the standard steps of quality engineering—finding the error, figuring out why it happened, fixing it, and proving the fix worked—could be followed using only the data the system generated itself.
The study revealed that the system's own logs were indeed powerful enough to act as a quality record. The researcher found that the system made specific types of errors, such as filing the same document twice, misplacing a file into the wrong category, or failing to verify a document's content. By grouping these mistakes, she discovered that just three types of errors accounted for nearly 72 percent of all the corrections needed. However, a simple count of errors told a misleading story. If you just look at the total number of times a mistake happened, you might think the most common error is the most important one to fix. But when the researcher looked at the mistakes from the perspective of the actual cases—how many unique clients were affected—the ranking changed. Some errors that happened frequently were actually minor and affected few people, while other rare errors were so serious that they could mix up two different people's entire legal cases. This showed that counting errors is not enough; you must weigh them by how dangerous they are to the people involved.
The investigation went deeper to find out why these errors occurred. Using a method that maps out causes like a fishbone diagram, the researcher traced the problems back to their roots. Surprisingly, the biggest issues were not because the artificial intelligence model was "dumb" or unable to understand language. Instead, the errors often came from how the data was prepared or how the system was set up. For example, some documents were scanned in a way that left them blank to the computer, or the system was missing a specific code needed to sort a certain type of form. The root cause was often a gap in how knowledge was transferred from the lawyers to the software, rather than a failure of the AI's brain. The team then took corrective actions: they fixed the data, added new rules to catch blank scans, and adjusted the system's confidence settings. They logged every change, just as a factory would log a repair.
However, the study also found a critical missing piece in the loop. While the team successfully found the errors, investigated them, and applied fixes, they could not fully prove that the fixes worked for the long term. Because the data came from a short, intense period of migration and repair, there was no long stretch of time afterward to watch the system and see if the errors stopped happening. In a true quality cycle, you must wait and verify that the problem does not return. The researchers showed that they could do everything up to that final step, but the final step required a different kind of evidence that the logs alone could not provide. They could not say with certainty that lowering the system's confidence threshold was safe because they did not have a way to check the correct answers against the system's guesses after the fix.
This work suggests a new way to govern artificial intelligence. It proves that we can treat AI failures not as mysterious black-box glitches, but as measurable quality deviations that can be tracked, investigated, and fixed using established engineering methods. The logs from the system can tell us what went wrong and help us prioritize which fixes matter most. But it also sets a clear boundary: the data from the system is not enough to close the loop on its own. To truly ensure safety, organizations must add specific checks, like knowing exactly how many cases were affected and having a plan to verify that fixes hold up over time. The study does not claim to have solved all the problems of AI safety, but it provides a concrete, working method for taking responsibility when things go wrong. It shows that by applying the disciplined habits of quality engineering, we can move from simply hoping our AI works to actually knowing how to fix it when it doesn't.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.