Technical Summary: Can Developers Rely on LLMs for Secure IaC Development?

Problem Statement

Infrastructure as Code (IaC) is essential for automating infrastructure management but is frequently plagued by "security smells"—recurring patterns indicating poor security practices such as hard-coded secrets, weak cryptography, and invalid IP bindings. While specialized tools (e.g., SLIC, SLAC) exist to detect these issues, their adoption remains low. Concurrently, developers are increasingly adopting Large Language Models (LLMs) for coding assistance. This study addresses a critical gap: the lack of empirical evidence regarding the ability of proprietary LLMs (specifically GPT-4o and Gemini 2.0 Flash) to effectively detect security smells in existing IaC scripts and to generate secure IaC from scratch. The authors highlight a specific risk where LLMs may generate functional code that inadvertently includes severe vulnerabilities (e.g., hard-coded passwords, HTTP without TLS) without issuing warnings, potentially giving developers a false sense of security.

Methodology

The research employs a two-phase empirical approach involving two distinct datasets: Stack Overflow (SO) code snippets and real-world GitHub repositories.

1. Datasets and Scope

• Tools: The study focuses on Ansible (imperative) and Puppet (declarative), the most widely used IaC tools.
• Security Smells: The evaluation targets nine specific security smells (S1–S9), including hard-coded secrets, empty passwords, weak cryptography (SHA1/MD5), admin-by-default configurations, HTTP without TLS, missing integrity checks, invalid IP binding (0.0.0.0), suspicious comments, and missing default case statements.
• Stack Overflow Study:
  • Data: 2,569 SO posts tagged with Ansible or Puppet were filtered via keyword search (e.g., "password," "HTTP," "TODO").
  • Ground Truth: Two researchers manually reviewed snippets, identifying 169 insecure examples. Inter-rater reliability was high (Cohen's kappa = 0.93).
• GitHub Study:
  • Data: A total of 21,757 files were collected from state-of-the-art literature replication packages and direct GitHub searches.
  • Sampling: A representative subset of 430 files (302 from literature, 128 from search) was manually categorized, revealing 341 instances with security smells.
  • Synthetic Scenarios: Based on the 341 insecure patterns, 89 unique synthetic scenarios were designed to test code generation capabilities.

2. Experimental Design

The study evaluates LLMs under two prompting conditions:

1. General Prompt: A standard request asking the model to analyze code from a security perspective (e.g., "As a security expert, please analyze the code...").
2. Guided Prompt: A structured, step-by-step instruction requiring the model to identify tools, list smells, review code/comments precisely, and output results in a specific CSV format. This was designed to reduce verbosity and hallucination.

Evaluation Tasks:

• Detection: Assessing the models' ability to identify existing security smells in the provided code snippets.
• Fix Suggestion: Analyzing whether models provide actionable code fixes or general recommendations.
• Generation: Prompting models with the 89 synthetic scenarios to generate IaC scripts, measuring the rate of secure output and the frequency of warnings issued for insecure patterns.

Key Results

Security Smell Detection

• Stack Overflow (Small Snippets):
  • General Prompt: GPT-4o detected 80% of smells; Gemini detected 71%.
  • Guided Prompt: Performance improved to 88% (GPT-4o) and 78% (Gemini).
• GitHub (Real-world Projects):
  • General Prompt: Performance dropped significantly. GPT-4o detected only 42% and Gemini 51% of smells.
  • Guided Prompt: GPT-4o's F1 score rose from 58% to 89%; Gemini's F1 score increased from 66% to 74%.
  • Observation: Detection rates were consistently higher for Puppet (declarative) than Ansible (imperative/YAML-based), likely due to the rigid structure of Puppet aiding pattern recognition.
• Specific Smells: Detection of "Suspicious Comments" (S8) was particularly challenging, even with guided prompts, while "HTTP without TLS" (S5) was detected more reliably.

Secure Code Generation

• Baseline Performance: When prompted to generate code for scenarios prone to vulnerabilities without explicit security constraints, only 7% of the generated scripts were fully secure.
  • 75% of GPT-4o outputs and 56% of Gemini outputs contained security smells with no warnings.
• Explicit Security Instruction: Even when explicitly prompted to "generate secure code":
  • Only 17% of GPT-4o outputs and 8% of Gemini outputs were fully secure.
  • A significant portion (44% for GPT-4o, 34% for Gemini) still contained security smells without any accompanying warnings.
• Fix Suggestions: While models frequently offered general recommendations (93–96%), they rarely provided explicit code fixes (GPT-4o: 15%, Gemini: 5%).

Significance and Claims

The paper claims that while LLMs show promise as assistants for detecting security smells in IaC, particularly when guided by structured prompts, they currently cannot be relied upon as standalone tools for secure IaC development.

• Prompt Sensitivity: The effectiveness of LLMs is highly dependent on prompt engineering. General prompts often fail to detect more than half of the security issues in complex, real-world repositories.
• Generation Risks: The most critical finding is the high failure rate in secure code generation. The models frequently produce insecure code without flagging the vulnerabilities, even when explicitly asked to be secure. This suggests a risk of developers receiving a "false sense of security."
• Contextual Limitations: LLMs struggle more with the flexibility of Ansible compared to the declarative rigidity of Puppet, and they perform better on simplified snippets (SO) than on full-scale project scripts (GitHub).

The authors conclude that further research is required to improve LLM capabilities in this domain. They emphasize that developers should not rely solely on LLMs for security assurance and that the research community must develop better prompting strategies and tools to mitigate these risks. The authors have released a replication package, including the 89 synthetic scenarios, to facilitate future studies.