Technical Summary: Tool-Guided Retrieval-Augmented Repair for Securing LLM-Generated C Code

Problem Statement

Large Language Models (LLMs) are increasingly utilized for low-level programming tasks, including the generation of C code for firmware and embedded systems. However, outputs from these models frequently contain security vulnerabilities and compilation errors. Prior research indicates that AI-assisted developers may produce less secure code than those without such tools, and modern code-generation models still exhibit nontrivial vulnerability rates on secure coding benchmarks.

These risks are particularly acute in embedded systems, where resource constraints, limited memory, and architecture-specific requirements make memory-safety violations, unchecked inputs, or low-level API misuse potentially catastrophic. Existing approaches to mitigate these issues generally fall into two categories:

1. Retrieval-Augmented Generation (RAG): Uses external security knowledge or examples to guide generation. However, naive retrieval from security documents can introduce noise and weaken guidance.
2. Iterative Repair: Uses tool feedback (e.g., static analysis) to refine code. While effective for higher-level languages like Python, these methods often fail to address the combined requirements of embedded C, where security, compilation correctness, and low-level behavior must be considered simultaneously.

Current workflows often rely on single-pass generation followed by manual inspection, which is insufficient for safety-critical environments. There is a gap in approaches that coordinate multiple verification signals (compilation, static analysis, symbolic execution) with retrieval to support an incremental repair process grounded in prior outcomes.

Methodology

The authors propose a Tool-Guided Retrieval-Augmented Repair workflow designed to improve the reliability of LLM-generated C code. The approach operates in four distinct stages:

1. Task Specification and Initial Generation

The system accepts a natural-language description of a C function or small program (covering tasks like mathematical computation, string processing, and sorting). The model prompt includes the task description and a short security checklist. The model generates an initial candidate implementation without human intervention.

2. Compilation and Static Analysis

The candidate code undergoes a two-step analysis:

• Compilation: The code is compiled using GCC. If compilation fails, diagnostics (e.g., missing headers, undeclared identifiers) are collected.
• Static Security Analysis: If compilation succeeds, the code is analyzed using CodeQL. Queries target common C vulnerability classes, including buffer overflows, out-of-bounds memory accesses, unsafe library functions (e.g., gets, strcpy, unbounded sprintf), unchecked return values, and integer overflow/signedness errors.

3. Retrieval-Augmented Repair Patterns

A central component is an execution-time repository of repair patterns that grows incrementally across tasks.

• Repository Structure: Each entry stores the task description, generated code, and metadata including compilation status, CodeQL findings, KLEE analyzability, and a composite quality score.
• Quality Scoring: Scores range from 0.0 (non-compiling) to 1.0 (compiling, CodeQL-clean, and KLEE-clean). This prioritizes security cleanliness while distinguishing partial successes.
• Retrieval Strategy: For a new task, the system retrieves semantically similar prior entries. It uses a contrastive design:
  • Positive Evidence: Up to three high-quality entries (score > 0.60) are used to extract security practices (e.g., checking scanf return values).
  • Negative Evidence: Up to two insecure but compilable entries are used to provide avoidance hints, but only if sufficient positive examples exist.
• Guidance Distillation: Instead of inserting raw code as few-shot examples, the system distills retrieved entries into brief textual guidance (practices and warnings). This reduces the risk of models copying code without adaptation.
• Iterative Repair: The system performs up to three repair iterations per task. Each iteration's prompt includes the task, current code, diagnostics, and retrieval-derived guidance. The best version is selected based on the composite quality score. The repository is capped at 500 items, with deduplication applied to maintain quality.

4. Final Symbolic Validation

After the repair loop, the final candidate undergoes symbolic execution using KLEE. This stage acts as a one-shot semantic check outside the repair loop to detect errors missed by static analysis, such as assertion failures, null dereferences, or memory-safety violations reachable via specific input paths. The KLEE result updates the repository metadata to favor patterns that are both security-clean and symbolically robust.

Key Contributions

• Integrated Workflow: The paper proposes a pipeline that coordinates compilation feedback, static security analysis (CodeQL), and symbolic validation (KLEE) with retrieval, rather than relying on a single feedback source or retrieval alone.
• Prioritized Repair Logic: The approach prioritizes repairs by impact, addressing security issues before compilation failures, reflecting how developers reason about low-level code.
• Dynamic Repository: Unlike static knowledge bases, the system utilizes an execution-time repository of verified repair outcomes that evolves during the process, enabling the reuse of validated guidance across tasks.
• Distilled Guidance: The method converts retrieved examples into concise textual instructions rather than raw code, mitigating the risk of overfitting or inappropriate copying by smaller models.

Experimental Results

The approach was evaluated on a dataset of 5,000 C programming tasks using three open-source models: DeepSeek Coder 1.3B, CodeLlama 7B, and Qwen2.5. The evaluation measured compilation failure rates, static defect rates (CodeQL), and KLEE analyzability.

Baseline Performance:

• Baseline models showed substantial reliability gaps. Compilation failure rates ranged from 27.18% (Qwen2.5) to 45.56% (CodeLlama 7B).
• Security defect rates were high, with CodeLlama 7B showing a 48.50% defect rate and DeepSeek Coder 1.3B at 34.62%.

Performance with Tool-Guided Repair:

• CodeLlama 7B: Security defect rates decreased from 49% to 19%. Total CodeQL errors dropped from 15,088 to 2,463 (an 83.7% reduction). Compilation failures were reduced from 45.56% to 27.50%.
• DeepSeek Coder 1.3B: Compilation failures were reduced from 42% to 22%, and security defects fell from 35% to 15%. KLEE analyzability improved from 56.80% to 77.44%.
• Security-Clean Rates: When conditioned on successful compilation, the rate of security-clean code for CodeLlama rose from 10.9% to 73.4%, and for DeepSeek Coder from 65.4% to 80.6%.

Error Analysis:
Remaining security issues in repaired code aligned closely with NIST software flaw categories. The most persistent issues were related to Exceptional Condition Handling (e.g., missing checks on scanf return values) and Buffer Overflows (unbounded writes).

Significance and Claims

The authors claim that this work demonstrates the feasibility of a structured, tool-guided workflow that improves the safety of LLM-generated C code for embedded development without requiring larger models or retraining.

• Modest Scope: The paper acknowledges that the current evaluation uses general-purpose C tasks as a proxy for embedded reliability, as the dataset is not embedded-specific. Direct embedded benchmarks are reserved for future work.
• Component Contribution: The paper notes that while the workflow shows consistent gains, the specific contributions of retrieval versus multi-tool feedback require further ablation studies to fully isolate.
• Future Directions: The authors plan to extend the evaluation to embedded-focused benchmarks, replace aggregate error reporting with taxonomy-based vulnerability analysis, and conduct ablation studies to better understand component contributions and real-world applicability.

In conclusion, the paper posits that integrating lightweight analysis tools into the generation loop significantly reduces both compilation failures and security defects, offering a practical path toward more reliable AI-assisted low-level programming.