Runtime Risk Detection and Control for AI Agents and LLM Applications

Imagine a world where computer programs do not just answer questions, but actively go out into the digital world to gather information, make decisions, and take actions on their own. These are known as AI agents. They are like digital employees that can book flights, analyze data, or control smart devices by chaining together many small steps. However, this new ability brings a specific kind of danger. Because these agents can read information from the internet and act on it, a clever trick hidden inside a harmless-looking website could trick the agent into doing something harmful, like stealing data or deleting files. The problem is that by the time a human notices the mistake, the agent might have already caused damage. To stop this, we need a way to watch these agents while they are working, spot when they are about to make a risky move, and pause them for a human to check before they act.

This is the challenge tackled by a research project called AgentGuard AI. The researchers, working from a university in India, built a prototype system designed to act as a watchdog for these autonomous programs. Their goal was not to build a perfect security product for the real world, but to create a working model that could show how to connect three critical ideas: detecting risky behavior, scoring how dangerous that behavior is, and deciding what to do next. They wanted to see if they could build a system that not only spots trouble but also keeps a clear, unchangeable record of every decision, so that humans could later review exactly what happened and why. The result is a detailed look at a software tool that attempts to bring order and human oversight to the chaotic world of AI agents.

The system they built, named AgentGuard AI, operates like a control tower for digital agents. It starts by watching the stream of events as the agent works. It looks for specific patterns that suggest danger, such as an agent trying to access a secret file or following a confusing instruction that might be a trap. When it spots something suspicious, it does not just raise a red flag; it calculates a risk score. This score is not a single number pulled from thin air. Instead, it is a combination of six different factors, including how risky the agent's tools are, how secure the system is, and how the user is behaving. These factors are weighed together to produce a final score that tells the system how serious the situation is.

Once the risk score is calculated, the system moves to the decision phase. It has four possible states it can recommend: it can let the agent continue, keep a close watch, restrict what the agent can do, or block the action entirely. Crucially, this system is designed to be transparent. It keeps a detailed log of every step, from the initial alert to the final recommendation. It allows different people to have different roles; a manager can change the rules, a researcher can run tests, and a reviewer can look at the logs to approve or deny actions. The system also includes privacy features, such as hiding sensitive parts of the conversation in the logs, and it creates a digital "chain" of records that makes it very difficult to tamper with the history of what happened.

To test if this idea actually worked, the researchers ran a specific experiment. They did not use real agents or real hackers. Instead, they used a built-in simulator to generate twenty-five fake events. Some of these events were safe, while others were designed to look like attacks. The system processed these events and produced a complete package of evidence, including the raw data, the risk scores, the alerts, and the final decisions. The researchers then inspected this package carefully, comparing what the system showed on its screen with what was saved in the digital files.

The inspection revealed that the system could indeed connect all the dots. It successfully took an event, scored the risk, made a recommendation, and saved the evidence in a way that could be reviewed later. This proved that the basic workflow was possible. However, the test also uncovered significant flaws that prevent this from being a ready-to-use security tool. The researchers found that the system was not perfectly consistent. For example, the screen showed one version of the risk-scoring rules, but the saved file showed a different version. In another case, the system said it would block a high-risk action, but the saved record said it would simply ask for human review. These mismatches mean that if you tried to replay the experiment later, you might not get the exact same result, which is a major problem for any system that claims to be reliable.

Furthermore, the system lacked some essential details needed for true proof. It did not record the specific random seed used to generate the test events, nor did it record the exact version of the computer code that was running. Without these details, it is impossible for another researcher to recreate the experiment exactly to verify the results. The study also noted that the system was running as a simple simulation on a single computer, not as a complex, high-speed service that could handle real-world traffic. It was a research prototype, not a finished product.

The researchers were very clear about what their work did and did not achieve. They did not prove that their system could stop real hackers or that it was better than other security tools. They did not test it with real people reviewing the alerts to see if the system reduced their workload or improved their decisions. What they did prove was that they could build a framework that links detection, scoring, and human review into a single, inspectable process. They showed that it is possible to create a digital trail that connects a risky event to a human decision, but they also showed that building a system that is consistent, reproducible, and ready for the real world requires much more work.

The value of this study lies in its honesty. Instead of presenting a polished, perfect solution, the researchers presented a working model that they then took apart to show where it was strong and where it was weak. They demonstrated that while the idea of a governance-aware AI watchdog is sound, the details matter immensely. A system that claims to protect AI agents must be able to keep its own records straight, ensure that its rules are applied consistently, and provide enough information for humans to trust its decisions. Until those pieces are fixed, the system remains a powerful tool for research and a blueprint for the future, but not yet a shield for the present.