💻 computer science

Does Fixing Break Security? An Empirical Study of Security Degradation in Iterative LLM-Driven Infrastructure-as-Code Repair

This empirical study analyzes 5,968 iterative LLM-driven Infrastructure-as-Code repair scenarios to reveal that while security regressions occur in up to 13.8% of cases under standard detection, a more conservative strict-mode analysis shows a defensible degradation rate of 3.3%, primarily driven by resource restructuring and mitigated by stopping repairs after the third iteration.

Original authors: Benjamin Agyekum, Fabio Santos

Published 2026-08-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Benjamin Agyekum, Fabio Santos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a massive, intricate castle out of digital blocks. This isn't just any castle; it's the infrastructure that runs the internet, your favorite apps, and cloud services. In the world of software engineering, this is called Infrastructure-as-Code (IaC). Instead of clicking buttons in a menu, engineers write text files (code) that tell the computer exactly how to build these digital castles. Recently, we've started using super-smart AI assistants, known as Large Language Models (LLMs), to write this code for us. It's like hiring a robot architect who can draft blueprints in seconds.

But here's the catch: robots make mistakes. Sometimes, the blueprints they draw have security holes—like leaving the front door wide open or forgetting to lock the treasure chest. To fix this, we use a "feedback loop." We run a security scanner (a digital inspector) that checks the robot's work, points out the mistakes, and sends the bad news back to the robot. The robot then tries to fix the code and sends it back for another check. This cycle repeats, hoping the castle gets safer with every round. The big question is: Does fixing one problem accidentally break something else that was already safe? It's like trying to patch a hole in a boat, only to accidentally punch a hole in the hull while you're at it. This paper dives deep into that exact scenario to see if our AI helpers are actually making things safer or just making a mess.

The Robot Architect's Dilemma: When Fixes Break Fixes

In this study, researchers Benjamin Agyekum and Fabio Santos decided to play detective with a massive dataset of these AI repair attempts. They looked at nearly 6,000 different scenarios where an AI tried to build or fix cloud infrastructure code. They watched what happened over 5 rounds of repairs, tracking 30 specific security checks (like "Is the data encrypted?" or "Is the access locked down?").

Their main discovery is a bit of a twist: Yes, fixing things can break security, but it's not as scary as it first looks.

When they looked at the raw numbers using a "standard" way of counting, they found that 13.8% of the time, the AI managed to break a security rule that was previously working while trying to fix something else. That sounds like a lot, right? But the researchers realized that the way they were counting was a bit tricky. Because the code often involves many different parts (like multiple digital locks on a building), the AI might fix one lock but accidentally mess up the status of another, even if the actual security wasn't truly compromised.

When they switched to a "strict" way of counting—looking only for clear, undeniable cases where security actually got worse—the number dropped dramatically to just 3.3% of scenarios. This suggests that most of the "breaks" were just confusing measurement glitches caused by the complexity of the code, not genuine security disasters.

The "Why" and "How" of the Breakage

So, when the AI does mess up, what's going on? The researchers found that the culprit is almost always resource restructuring. Imagine the robot architect decides to completely rebuild a wall instead of just patching a crack. In doing so, it might forget to put the security camera back on the new wall. This happened in 79% of the cases where security actually regressed.

They also noticed something interesting about the "personality" of the AI models. One model (Mistral) seemed to break things 17 times more often than another (Gemini) when using the standard counting method. However, when they used the strict method, neither model broke anything exclusively. This means the "bad" model wasn't actually creating more dangerous holes; it was just creating more complex code structures that confused the counting method.

The Sweet Spot: When to Stop Fixing

One of the most practical findings is about when to stop. The AI keeps trying to improve the code, but does it ever stop? The study suggests that the 3rd iteration (the third time the AI tries to fix the code) is the sweet spot.

By the 3rd try, the code is about 83.1% secure.
If you keep going to the 4th or 5th try, you barely gain any more security (maybe 0.3% more), but you start increasing the risk of breaking things again.

It's like tuning a radio: after a certain point, turning the dial just makes the static worse without finding a clearer station.

The Silver Lining: Self-Correction

Here is the most hopeful part of the story. The researchers found that when the AI does accidentally break a security rule, it often fixes itself! About 36.6% of the time, the very next round of repairs corrects the mistake the AI just made. It's like the robot architect realizes, "Oops, I took the wrong door off," and puts it back on in the next step.

However, they also found a "tug-of-war" effect. About 28.5% of the time, the security checks would flip back and forth—pass, fail, pass, fail—like a pendulum that can't decide where to stop. This usually happens with complex access controls, suggesting the AI is still figuring out the best way to build those specific parts.

The Bottom Line

This paper tells us that while iterative AI repair is a powerful tool, we need to be careful about how we measure its success.

Don't panic over every glitch: Most apparent security breaks are just confusing measurement artifacts, not real dangers.
Watch out for big rewrites: If the AI starts tearing down whole sections of code to fix a small bug, that's when security is most likely to slip.
Stop at three: Let the AI try three times to fix the code, then call it a day. Going further usually adds more risk than reward.
Use the right tools: If you want to be super safe, use a "strict" way of checking that ignores confusing multi-part errors, but keep an eye on the "standard" alerts just in case.

In short, the AI is a helpful apprentice, but it needs a human supervisor to know when to stop turning the wrench, or it might just end up tightening the bolt so hard it snaps the whole machine.

Technical Summary: Does Fixing Break Security? An Empirical Study of Security Degradation in Iterative LLM-Driven Infrastructure-as-Code Repair

Problem Statement

While iterative feedback loops—where validators like Checkov and terraform validate feed error signals back to Large Language Models (LLMs) for successive repair attempts—have become the dominant paradigm for improving LLM-generated Infrastructure-as-Code (IaC), prior research has largely relied on cumulative-best metrics. These metrics report the highest compliance achieved across all iterations, which are monotonically non-decreasing by construction. Consequently, the raw per-iteration security trajectory has never been examined in the IaC domain.

This study addresses the phenomenon of security regression: instances where a previously passing security check (specifically CIS Benchmark controls) fails after a repair iteration intended to fix other issues. Unlike general-purpose code repair studies (e.g., Shukla et al. [25], Chen et al. [7]) which focus on imperative languages and vulnerability counts, this research investigates the unique challenges of IaC. IaC is declarative, and security checks map to specific configuration properties at the resource level. A critical distinction is that a single Checkov check ID can apply to multiple resources within a single configuration, creating a class of multi-resource ambiguity where a check may pass for some resources and fail for others simultaneously. This ambiguity complicates the definition of a "regression" in IaC compared to general code.

Methodology

The authors conducted a large-scale empirical study analyzing 5,968 scenario timelines from the IaC-Eval benchmark. The study design involved:

Configurations: 15 distinct experimental configurations comprising two model-specific Retrieval-Augmented Generation (RAG) setups (Gemini and Mistral) and nine model-aggregated non-RAG setups (Zero-Shot, Few-Shot, Chain-of-Thought), each run at three temperatures (0.1, 0.4, 0.7).
Process: Each scenario was run through up to 5 repair iterations. The pipeline generated Terraform code, validated it, and fed errors back to the LLM.
Data Volume: The analysis covered 4,440 iteration transitions where Checkov data was available for both the pre- and post-repair states.
Tracking: The study tracked 30 individual Checkov check IDs mapped to CIS AWS Foundations Benchmark controls across six security categories (encryption, access control, logging, networking, data protection, and other).
Detection Modes: To address multi-resource ambiguity, two detection modes were formalized:
- Standard (Inclusive): Counts a regression if a check ID appears in the passed set at iteration $i$ and the failed set at iteration $i+1$ , regardless of whether it also passed for other resources.
- Strict (Exclusive): Counts a regression only if a check ID was exclusively passed (no failures) at iteration $i$ and exclusively failed (no passes) at iteration $i+1$ .
Root Cause Analysis: Code diffs were classified into four mutually exclusive categories: Resource Restructuring, Configuration Drift, Argument Removal, and Unclassified.
Statistical Analysis: The study utilized Chi-squared tests, Mann-Whitney U tests, Fisher's exact test, and effect size measures (Cramér's V, Cohen's d) with Bonferroni correction for multiple comparisons.

Key Contributions

First Large-Scale Empirical Study: This is the first study to analyze the per-iteration security trajectory of iterative LLM-driven IaC repair, moving beyond cumulative-best metrics to detect security regressions.
Regression Taxonomy: The authors derive a taxonomy of regression root causes, identifying Resource Restructuring as the dominant cause (79.0% in standard mode).
Detection Mode Distinction: The paper formalizes the Standard vs. Strict detection distinction, demonstrating that the choice of detection mode qualitatively changes conclusions regarding model performance and risk distribution.
Security-Correctness Trade-off Quantification: The study quantifies the relationship between code changes and security degradation, showing that regression transitions involve significantly higher code churn and check volatility.
Actionable Guidance: The study provides specific guidance on iteration budgets and configuration choices, identifying iteration 3 as the optimal stopping point and highlighting the self-correction capabilities of the feedback loop.

Key Results

RQ1: Frequency of Regressions

Standard Mode: 13.8% of scenarios (24.8% of transitions) exhibited at least one security regression.
Strict Mode: When filtering for unambiguous, exclusive failures, the rate drops to 3.3% of scenarios (5.2% of transitions).
Interpretation: The gap between standard and strict modes indicates that approximately 76% of apparent regressions are multi-resource measurement artifacts rather than genuine exclusive failures.

RQ2: Vulnerable Security Controls

Standard Mode: Access Control dominates (38.5%), driven by IAM-related checks (e.g., CKV_AWS_356, CKV_AWS_111).
Strict Mode: The ranking reverses. Networking (42.6%) and Encryption (21.6%) become the dominant risks, while Access Control drops to 9.2%.
Implication: Access control regressions are largely artifacts of multi-resource ambiguity (e.g., one IAM role passes while another fails), whereas networking and encryption regressions represent genuine exclusive failures.

RQ3: Configuration Factors

Model Gap (RAG Configurations): In standard mode, Mistral showed a regression rate 17× higher than Gemini. However, this gap vanishes entirely in strict mode for RAG configurations, where neither RAG+Gemini nor RAG+Mistral produced a single strict-mode scenario regression (0 of 1,297 and 0 of 929 respectively). This confirms the gap was due to multi-resource ambiguity rather than exclusive check failures.
RAG Effect: RAG appears to have a higher regression rate in standard mode but a lower (zero) rate in strict mode compared to non-RAG strategies. This suggests RAG generates more comprehensive multi-resource structures (increasing standard counts) but provides better grounding for individual security properties (reducing exclusive failures).
Temperature: Temperature had a statistically significant but practically negligible effect (Cramér's V = 0.049).

RQ4: Security-Correctness Trade-off

Code Churn: Regression transitions involve 2.6× more code churn (lines changed) and lower text similarity compared to non-regression transitions.
Check Volatility: This is the strongest predictor of regression. Regression transitions show 4.9× higher strict-mode check volatility (Cohen's d = 1.49).
Syntax Fixes: Transitions immediately following a syntax fix carry a 21% higher regression risk in standard mode.
Clean Timelines: Counter-intuitively, scenarios that maintained valid syntax throughout ("clean" timelines) had a higher regression rate (20.8%) than those that struggled with syntax errors (12.1%).

Additional Findings

Self-Correction: 36.6% of standard-mode regressions self-corrected within an average of 1.2 iterations, suggesting the feedback loop often catches its own mistakes.
Oscillation: 28.5% of scenarios exhibited oscillating checks (passing, failing, then passing again), particularly in IAM configurations.
Optimal Stopping: Iteration 3 is identified as the optimal stopping point, capturing 83.1% of the pass rate (within 0.3 percentage points of the peak) while keeping regression risk manageable.

Significance and Claims

The paper claims to provide the first empirical evidence that iterative IaC repair introduces security regressions, though the severity depends heavily on the detection methodology. The authors assert that while the "conservative, defensible rate" of genuine exclusive regressions is approximately 3.3%, the phenomenon is non-negligible and is entirely masked by the cumulative-best metrics used in prior work.

The study emphasizes that resource restructuring is the primary driver of degradation, not the prompting strategy or temperature alone. The findings motivate the design of security-aware feedback loops that:

Distinguish between multi-resource artifacts and exclusive failures.
Implement "security anchoring" to prevent the model from regenerating entire resource blocks when only minor fixes are needed.
Adopt an iteration budget of approximately 3 iterations to balance compliance gains against regression risks.
Utilize check volatility as a real-time signal to halt or flag repair loops.

The authors conclude that while iterative repair improves overall compliance, it introduces a specific class of security degradation that requires new detection methodologies and operational safeguards, particularly given the declarative and multi-resource nature of IaC.