Technical Summary: Cross-Cutting Security Analysis of LLM-Generated Code via Metamorphic Testing and Association Rule Mining

Problem Statement

Large Language Models (LLMs) frequently generate code containing security vulnerabilities. While prior research has established that LLM-generated code is often insecure, existing evaluations typically treat vulnerabilities as isolated defects (e.g., analyzing SQL injection separately from buffer overflows). This approach overlooks the cross-cutting nature of software security, where failures in authentication, credential management, input validation, and memory safety often co-occur within the same artifact. Consequently, current methods fail to diagnose how these vulnerabilities cluster together or to identify which prompt characteristics drive broader, systemic security risks.

Methodology

The authors propose a four-phase framework that integrates Metamorphic Testing (MT) with Association Rule (AR) mining to detect, diagnose, and explain security failures in LLM-generated code.

1. Data Generation (Phase A): The study utilizes the LLMSecEval benchmark, comprising 148 unique natural-language prompts covering 18 Common Weakness Enumeration (CWE) categories. These prompts were executed across five open-source models (Qwen3-Coder, Qwen2.5-Coder, DeepSeek-Coder, CodeGemma, and Gemma4:e4b) with five runs each, generating 3,700 code snippets (Python and C).
2. Metamorphic Judgment (Phase B): The authors define a catalog of nine security-oriented Metamorphic Relations (MRs) covering major CWE categories, including SQL injection (CWE-89), XSS (CWE-79), command injection (CWE-78), path traversal (CWE-22), authentication bypass (CWE-862), hard-coded credentials (CWE-798), weak cryptography (CWE-327), buffer overflow (CWE-120), and integer overflow (CWE-190).
   • An LLM-based judge (Claude Sonnet 4.6) evaluates each snippet against these MRs.
   • For each MR, the judge determines applicability, provides a verdict (violated/pass/N/A), and offers evidence.
   • This process constructs a binary violation matrix where rows represent snippets and columns represent MRs.
3. Association Rule Mining (Phase C): The violation matrix is processed using the Apriori algorithm to discover frequent co-violation patterns. Rules are evaluated based on support, confidence, and lift (where lift > 1 indicates positive association beyond statistical independence). This step identifies "cross-cutting" clusters of vulnerabilities.
4. Prompt-Level Risk Analysis (Phase D): The authors correlate the identified violation patterns with prompt features, including structural metrics, topic keywords (e.g., database, authentication, memory), security awareness phrasing, and multi-topic complexity. They also measure cross-model consistency to determine if vulnerabilities are prompt-inherent or model-dependent.

Baselines: The approach is compared against four static analysis tools (CodeQL, Bandit, Flawfinder, Semgrep) and the secure code samples provided in the LLMSecEval benchmark.

Key Contributions

1. Security-Oriented MR Catalog: Definition and application of nine metamorphic relations spanning major CWE categories specifically tailored for LLM-generated code verification.
2. Cross-Cutting Structural Diagnosis: The integration of AR mining with MR outcomes to reveal that security failures form structured clusters rather than isolated defects.
3. Prompt-Level Risk Analysis: A novel analysis linking co-violation clusters to specific prompt characteristics, identifying which topics drive broad insecurity.
4. Large-Scale Empirical Evaluation: Comprehensive evaluation on 3,700 snippets from five diverse open models, providing evidence on the prevalence and nature of cross-cutting vulnerabilities.

Key Results

Vulnerability Prevalence

• 68.8% of all generated snippets violated at least one MR.
• Hard-coded credentials (MR6) and Command Injection (MR3) were the most prevalent failures among applicable snippets (79.1% and 74.4%, respectively).
• SQL Injection (MR1) had the lowest violation rate (11.5%), suggesting LLMs have partially internalized parameterized query patterns.
• Model Performance: DeepSeek-Coder (6.7B) produced the most vulnerable code (73.8% violation rate), while Gemma4:e4b (4.5B) was the safest (65.1%). Notably, model scale did not correlate linearly with safety; the largest model (Qwen3-Coder, 30B) was not the safest.

Detection Effectiveness

• The MR-based approach detected 68.8% of vulnerable snippets.
• In contrast, the union of four standard SAST tools (CodeQL, Bandit, Semgrep, Flawfinder) detected only 34.2%.
• SAST tools failed to detect semantic violations such as Authentication Bypass (0% detection) and Hard-coded Credentials (only 44 of 651 detected).

Cross-Cutting Co-Violation Patterns (RQ2)

AR mining identified 44 association rules, revealing two primary vulnerability clusters:

1. Authentication–Credential–Cryptography Cluster: A tightly coupled group involving Authentication Bypass (MR5), Hard-coded Credentials (MR6), and Weak Cryptography (MR7).
   • Key Finding: The rule XSS ∧ WeakCrypto ⇒ HardCred has a confidence of 82.5% and a lift of 3.23.
   • Hard-coded credentials and missing authorization checks were the most frequent co-violation pair (226 snippets).
2. Input Handling–Memory Safety Cluster: Linking XSS, Path Traversal, and Buffer Overflow.
   • Key Finding: XSS ∧ PathTrav ⇒ BuffOvf (Conf: 63.0%, Lift: 2.04).

• Inter-Cluster Bridge: The rule AuthByp ∧ BuffOvf ⇒ XSS (Conf: 33.3%) bridges the two clusters, indicating that some prompts trigger failures spanning both authentication and memory safety domains simultaneously.

Prompt-Level Drivers (RQ3)

• Predictors: Database-related prompts were the strongest predictor of overall vulnerability (r = 0.52), followed by authentication topics (r = 0.43).
• Cluster Specificity: Database and authentication keywords strongly predicted Cluster 1 but not Cluster 2. Conversely, File I/O keywords were the sole predictor for Cluster 2 (r = 0.31), a relationship missed by total-count analysis.
• Security Awareness: Explicitly requesting secure code (e.g., using keywords like "secure," "sanitize") showed no correlation with safer output (r = -0.04).
• Cross-Model Consistency: In 65.5% of prompts, all five models agreed on the violation status. High-risk prompts (those triggering both clusters) were 14.2 times more likely to contain database keywords and 6.8 times more likely to contain authentication keywords than low-risk prompts.

Significance and Claims

The paper argues that insecure code generation by LLMs is not merely a collection of independent defects but a structured, prompt-conditioned phenomenon.

• Holistic Security Failure: The discovery of strong cross-cutting clusters (e.g., the triad of authentication bypass, hard-coded credentials, and weak cryptography) suggests that LLMs lack a holistic model of security. They may learn to parameterize SQL queries (low SQLi rate) while simultaneously failing to avoid hard-coding the credentials required for those queries (high HardCred rate).
• Prompt-Driven Risk: The high cross-model consistency (65.5% agreement) indicates that vulnerability is predominantly driven by the prompt content rather than the specific model architecture. This implies that mitigation strategies focusing on prompt engineering or training data may be more effective than simply switching between models of comparable size.
• Practical Implications:
  • Cluster-Aware Verification: Developers should adopt a "cluster-based" review strategy; detecting one vulnerability (e.g., hard-coded credentials) should trigger immediate checks for associated vulnerabilities in the same cluster (e.g., weak cryptography).
  • Targeted Intervention: Automated tools can flag prompts based on specific topic keywords (e.g., database + file I/O) to predict specific vulnerability clusters, rather than just estimating overall risk.
  • Limitations of Surface Instructions: The lack of correlation between security-aware phrasing and secure output reinforces that current LLMs do not respond to surface-level security instructions, making post-generation verification essential.

The authors conclude that their framework enables a shift from simple detection to structural diagnosis and prompt-level explanation, providing a foundation for safer LLM-assisted programming through data-driven insights into the cross-cutting nature of security failures.