From Trustworthy AI Principles to Verifiable System Requirements: A Multi-Stakeholder Engineering Framework for Clinical Decision Support Systems
This paper presents a reproducible, three-phase engineering framework that systematically translates multi-stakeholder needs and regulatory constraints into formal, verifiable system requirements compliant with ISO/IEC/IEEE 29148 and the FUTURE-AI taxonomy, demonstrated through the creation of a publicly available requirements corpus for a high-risk clinical AI system in the AI4HF project.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern hospital, a new kind of tool is taking shape: software that learns from patient data to predict health risks before they become emergencies. These systems, often called artificial intelligence, promise to help doctors make faster, more accurate decisions. But there is a deep divide between the lofty ideals of how such technology should behave and the actual code that runs on a server. We have broad agreements that medical AI must be fair, safe, and understandable, yet these ideas often remain as vague as "be kind" or "be honest." Without a clear bridge, developers might build systems that technically work but fail to meet the strict safety rules required for human care, or worse, they might build systems that ignore the very people they are meant to help. The challenge is not just to say what is right, but to write down exactly how a machine must behave to be considered right, in a way that can be tested and proven.
A team of researchers has built a bridge across this divide. They created a step-by-step method to turn the fuzzy hopes of doctors, patients, and regulators into a rigid, checkable list of instructions for software. This work was tested on a real-world project designed to predict risks for patients with chronic heart failure, a condition where a wrong guess can be fatal. The researchers gathered input from six different groups, including the patients themselves, the doctors who will use the tool, ethicists, and legal experts. They took the concerns raised in these meetings—such as a patient's fear that the computer might treat them differently than someone else, or a doctor's need to see how a prediction was made—and translated them into a formal language that software engineers can follow. The result was a document containing 173 specific rules. Each rule is written in a strict format that leaves no room for ambiguity, stating clearly what the system must do, what it should do, and what it might do. Crucially, every single rule comes with a defined test to prove it works, ensuring that the final product can be verified against the original promises of safety and fairness.
The process began by listening to the people involved. In workshops held across Europe, South America, and Africa, patients shared stories about their journeys, while doctors described their daily workflows. The researchers also looked at the hard constraints of the law, such as European regulations that classify medical AI as high-risk. They combined these bottom-up stories with top-down legal requirements to create a unified list of needs. Before turning these needs into code, the team used a consensus method to decide which were essential and which were optional. They then applied a strict engineering standard to rewrite these needs. Instead of saying "the system should be fair," which is hard to test, they wrote, "the system shall provide a notification about responsible decision-making regardless of whether the clinician relies on the prediction." This shift from vague concepts to concrete actions is the core of their achievement. It transforms a moral principle into a functional requirement that a tester can verify by running a specific check.
Once the list of 173 rules was written, the team checked it against a comprehensive framework for trustworthy AI known as FUTURE-AI. This framework breaks down the concept of trustworthiness into thirty specific sub-principles, covering everything from data privacy to how the system explains its reasoning. The researchers mapped each of their 173 rules to these principles to ensure nothing was missed. They found that their list covered every single sub-principle, proving that their method could capture the full complexity of the problem. The distribution of the rules revealed interesting insights: while many rules focused on how the system works (functional requirements), a surprisingly large number—over forty percent—focused on the quality of the system, such as its security, speed, and ability to handle errors. This highlights that for medical AI, the "non-functional" aspects are just as critical as the calculations themselves.
The researchers also examined how they planned to prove each rule was met. They assigned a specific method of verification to every single requirement. For about half of the rules, the proof would come from real people testing the system, using surveys to see if the explanations made sense or if the interface was easy to use. For others, the proof would come from automated checks that scan the code for security flaws or regulatory compliance. This mix ensures that the system is not only technically sound but also humanly acceptable. The team found that relying solely on automated tests would be insufficient; the human element of trust requires human validation. By pairing every rule with a test, they created a blueprint that leaves no room for guesswork.
This work does not claim to have solved every problem in medical AI, nor does it suggest that writing these rules is easy. The researchers acknowledge that their method was tested on one specific project for heart failure and that applying it to other diseases or different types of artificial intelligence will require further study. They also note that the final step of turning these high-level rules into actual computer code is a separate challenge. However, they have demonstrated that it is possible to take the complex, often conflicting voices of a diverse group of stakeholders and the strict demands of the law, and weave them into a single, coherent set of instructions. By doing so, they have provided a reproducible path for other teams to follow, ensuring that the next generation of medical AI is not just smart, but also safe, fair, and built on a foundation that can be trusted.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.