SENTRY: Deterministic, Intelligent Risk Assessment for IT Change Management
This paper introduces SENTRY, a deterministic machine learning platform that replaces subjective questionnaires with a hybrid XGBoost and RAG pipeline to accurately assess IT change risks in financial institutions, achieving high accuracy and significantly outperforming existing processes while maintaining auditability through explainable AI.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the vast, humming nervous system of a modern financial institution, thousands of small decisions happen every day. A software update is pushed to a trading platform, a server configuration is tweaked to improve speed, or a security patch is applied to close a vulnerability. These are routine maintenance tasks, the daily bread of keeping the digital world running. Yet, in these institutions, a single misstep can trigger a cascade of failures, shutting down services for millions of customers and inviting severe regulatory penalties. To prevent this, banks rely on a process called change management, where every proposed alteration must be reviewed and assessed for risk before it is allowed to proceed. For decades, this assessment has depended on a simple, human-centric tool: a questionnaire. The person proposing the change answers a series of questions about what they plan to do, how they will test it, and what could go wrong. Their answers are scored, and that score determines whether the change is approved, sent for more review, or rejected. The problem is that this system is deeply subjective. It relies on the honesty and self-awareness of the person filling it out, and it often fails to distinguish between a harmless tweak and a dangerous gamble. A skilled requestor can learn to phrase their answers to make a risky change look safe, while a genuine expert might underestimate the complexity of their own work. The result is a process that feels orderly but is fragile, often missing the very changes that later cause major outages.
This is the challenge that a team of researchers at the Royal Bank of Canada set out to solve with a new system they call SENTRY. Their goal was not to replace human judgment, but to give it a sharper, more objective lens. They built a platform that combines two distinct approaches to data analysis. First, they used a proven method for sorting structured information, a type of algorithm that excels at finding patterns in tables of numbers and categories. This part of the system looks at hard facts: how many other computer programs depend on the one being changed, how critical that program is to the bank's daily operations, and whether similar changes have caused problems in the past. This provides a solid, factual foundation. However, the researchers knew that the most important clues often hide in the unstructured text of the change request itself—the free-form descriptions, the implementation plans, and the back-out strategies written by humans. To capture this, they added a second layer that acts like a highly efficient librarian. Instead of trying to read every word of every past request with a complex language model that can be unpredictable, this system searches a massive library of historical records to find changes that are semantically similar to the one being proposed. It looks for the same concepts and technical ideas, even if the words used are different.
The brilliance of the design lies in how these two parts work together. The system takes the results of that library search—the fact that a proposed change looks very much like a past change that caused a major outage—and converts it into a single, simple number. This number is then fed into the main algorithm alongside the hard facts. This approach allows the system to learn from the rich, messy details of human writing without losing the ability to explain its reasoning. In a regulated environment like banking, a computer cannot simply say "this looks risky" because an artificial intelligence told it so; it must be able to point to exactly which factors contributed to that conclusion. By turning the complex search results into a single data point, the system keeps the final decision transparent and auditable. The researchers tested this new platform on real-world data from their institution, comparing its performance against the old questionnaire-based method. The results were striking. The new system correctly identified high-risk changes that would have been missed by the traditional process at a rate more than three times higher. While the old method caught only about one in five of the changes that eventually led to major incidents, the new system caught nearly two-thirds. It did this while maintaining a high level of accuracy for the changes that were safe, ensuring that the review teams were not overwhelmed with false alarms.
The system does not make the final decision to approve or reject a change. Instead, it acts as a powerful decision-support tool for the human reviewers. It provides them with a clear, evidence-based risk score and a breakdown of exactly why that score was given, highlighting whether the risk comes from the critical nature of the application, the history of similar failures, or the specific wording of the plan. This allows the human experts to focus their attention where it is needed most, rather than sifting through hundreds of low-risk requests that look safe on paper but might be dangerous in practice. The researchers found that by combining the reliability of structured data with the nuance of unstructured text, they could create a risk assessment that is both more accurate and more trustworthy than what came before. The system operates with a level of consistency that human reviewers cannot match, never tiring, never being swayed by the way a question is phrased, and always applying the same rigorous standards to every single request. This work demonstrates that in high-stakes environments, the future of risk management is not about replacing humans with machines, but about building tools that help humans see the hidden dangers that routine processes often miss. The success of this platform suggests that with the right architectural choices, it is possible to harness the power of advanced data analysis while preserving the clarity and accountability that regulated industries demand.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.