An Open-Agent Framework for Intelligent Compliance Analysis in Infrastructure as Code Using Large Language Models

In the modern digital world, the infrastructure that powers our applications, stores our data, and runs our services no longer sits in a physical server room with rows of blinking lights. Instead, it lives in the cloud, a vast network of remote computers managed by massive technology companies. For the organizations that use these services, the responsibility for security has shifted. While the cloud providers ensure the hardware works, the customers must ensure their digital configurations are set up correctly. A single mistake in how a storage bucket is opened or how a network permission is granted can expose sensitive information to the entire internet. To manage this complexity, engineers use a method called "Infrastructure as Code." This approach treats the setup of digital resources like a software program, writing out the rules in text files that can be read by machines. This allows for consistency and speed, but it also means that a single error in the code can be copied instantly across hundreds of systems, creating a widespread vulnerability before anyone notices.

To catch these errors, security teams rely on automated tools that scan the code for known mistakes, much like a spellchecker finds typos. However, these traditional tools are often rigid; they can tell you that a rule was broken, but they struggle to explain why it matters or how to fix it in a way that makes sense to a human auditor. On the other hand, a newer technology known as large language models can read code and generate natural language explanations, but they are prone to making things up or providing inconsistent results when used alone. The challenge for researchers has been to combine the strict reliability of the automated scanners with the explanatory power of the intelligent language models to create a system that is both precise and understandable.

A team of researchers at Princess Sumaya University for Technology has built a new framework that successfully merges these two approaches. They created a system that first uses a strict, rule-based engine to scan infrastructure code for violations of specific security standards, and then passes those confirmed violations to an intelligent language model to generate clear, human-readable reports and suggested fixes. The researchers tested this system on a large collection of 231 real-world configuration files. They found that the hybrid system was remarkably accurate, correctly identifying security flaws in nearly 97 percent of the cases where it raised an alert, while also catching more than 90 percent of the actual problems present in the files. This performance was significantly better than existing industrial tools, which tend to flag too many harmless issues, causing unnecessary work for engineers.

The core of this new framework is a five-step process that transforms raw code into a polished audit report. First, the system takes the raw configuration files and translates them into a format that the scanning engine can understand. Next, a deterministic policy engine, which operates like a strict referee, checks the code against a specific set of 16 security rules. These rules are based on the NIST SP 800-53 standard, a widely recognized catalog of security controls used by governments and enterprises. The engine looks for specific patterns, such as a storage drive that lacks encryption or a network port that is left open to the public. Because this engine follows fixed rules, it never guesses; it only reports what it can prove is a violation.

Once the engine identifies a violation, the system does not stop there. It sends the details of the error to a large language model, which acts as a contextual reasoning agent. This model is tasked with three specific jobs: explaining the risk in plain English, mapping the error to the official security standard it violates, and generating a corrected snippet of code to fix the problem. To ensure the model does not make mistakes in the factual parts of the report, the researchers built in a safety mechanism. The model is not allowed to guess the name of the security standard; instead, the system looks up the correct name from a verified list and inserts it automatically. The model is only free to generate the explanation and the code fix. Furthermore, the researchers added strict instructions to the model's prompt to ensure the code it writes is syntactically correct and free of unnecessary text, a step that proved essential for reliability.

The results of the study demonstrate the power of this combined approach. When tested against the 231 configuration files, the system achieved a precision of 96.9 percent, meaning that when it flagged a file as having a problem, it was almost always right. It also achieved a recall of 90.5 percent, successfully finding the vast majority of the actual errors hidden in the code. In comparison, a leading industrial tool called Checkov, which is widely used in the industry, found every single error in the test set but also flagged nearly half of the clean files as problematic. This high rate of false alarms would force security teams to waste time investigating issues that do not exist. The new framework, by contrast, provided a much clearer signal, reducing the noise while maintaining high detection rates.

A closer look at the data revealed a clear pattern in the types of errors found. The most common violation involved the lack of encryption for data at rest, which accounted for more than 64 percent of all detected issues. This finding aligns with previous research suggesting that forgetting to encrypt stored data is the most frequent security oversight in cloud environments. The system also successfully identified problems related to access controls and audit logging, covering a broad spectrum of security requirements. The researchers noted that the system's ability to generate accurate, audit-ready reports was a key advantage, as it bridges the gap between technical detection and the documentation required for compliance.

The study also highlighted the importance of the safeguards built into the system. When the researchers first tested the language model without strict constraints, it produced code fixes that were often syntactically invalid or filled with unnecessary prose. By enforcing strict formatting rules and using a verified lookup table for factual data, the system achieved a 100 percent success rate in generating valid code snippets and accurate control mappings. This suggests that the reliability of such systems comes not from the intelligence of the model alone, but from the careful design of the workflow that surrounds it.

While the system showed strong results, the researchers were careful to note its limitations. The framework currently focuses on a specific set of security controls and cloud resources, and it operates on the code before it is deployed, meaning it cannot detect issues that arise from changes made to the live system after deployment. Additionally, the ground truth used to test the system was derived from a combination of automated heuristics and manual injection, which may not perfectly capture every nuance of a real-world environment. Despite these boundaries, the study provides a compelling proof of concept. It shows that by combining the unyielding precision of rule-based engines with the flexible reasoning of language models, it is possible to create a compliance analysis tool that is both highly accurate and genuinely useful for human operators. The work suggests a path forward where automated security tools do not just find problems, but help teams understand and solve them with confidence.