From LLMs to Agents in Programming: The Impact of Providing an LLM with a Compiler

Technical Summary: From LLMs to Agents in Programming: The Impact of Providing an LLM with a Compiler

Problem Statement
Large Language Models (LLMs) have demonstrated significant capabilities in generating code from natural language prompts. However, a critical limitation remains: the generation process is unidirectional, relying solely on neural network probabilities without an oracle to verify correctness. Consequently, generated code often fails to compile or lacks structural and functional quality. While current mitigation strategies involve increasing model size, fine-tuning on code, or using Retrieval-Augmented Generation (RAG), these approaches demand substantial computational resources and do not inherently solve the lack of quality assurance. The paper posits that transforming LLMs into "agents" capable of interacting with external software development tools, specifically compilers, could bridge the gap between generation and executable correctness without necessitating massive model scaling.

Methodology
The authors conducted a computational experiment to evaluate the impact of integrating a compiler into the code generation loop.

• Dataset: The study utilized a subset of 699 tasks from the RosettaCode dataset, specifically those with solutions in C. Tasks ranged from simple functions to complex algorithms.
• Models: Sixteen LLMs were evaluated, spanning a parameter range from small (135 million) to large (70 billion).
• Experimental Setup:
  • Baseline: Models were prompted to generate C code in a single attempt.
  • Agent Framework: Models acted as agents in an iterative loop. If the generated code failed to compile via a gcc compiler, the error message was aggregated with the original task and the faulty code, then fed back to the model for a correction attempt. This process continued for a maximum of five iterations.
• Evaluation Metrics:
  • Compilation Success: The percentage of generated programs that compiled without errors.
  • Textual Similarity: Measured using BLEU and ROUGE-1 against ground-truth solutions.
  • Semantic Similarity: Measured using CodeBERTScore (precision and recall) on C-tuned embeddings.
  • Error Analysis: Generated outputs were categorized by error type (e.g., Syntax Error, Undefined Reference, Language Mismatch, Missing Code) to determine which errors were most effectively resolved by the agent framework.

Key Contributions and Results
The study provides empirical evidence that access to a compiler significantly enhances the capability of LLMs to produce executable code, often allowing smaller models to outperform larger ones.

1. Compilation Success Rates: Access to a compiler increased the rate of successfully compiled programs by 5.3 to 79.4 percentage points across the evaluated models.

   • The most dramatic improvement was observed in Qwen 3 (4 billion parameters), which increased its success rate from 18.0% (baseline) to 97.4% (agent), surpassing many larger models.
   • Llama 3.3 (70 billion parameters) achieved a 99.9% success rate with the compiler, up from 92.0%.
   • Crucially, the study found that smaller models with compiler access could outperform larger models without it, suggesting that tool integration can reduce the need for massive parameter counts.

2. Preservation of Semantics: The iterative repair process focused on fixing compilation errors without altering the program's core logic.

   • Textual similarity (BLEU/ROUGE) and semantic similarity (CodeBERTScore) remained largely consistent between baselines and agents.
   • Manual evaluation of three specific tasks (Easy, Medium, Hard) confirmed that while the number of incorrect solutions decreased for agents, the functional correctness of the successful solutions did not significantly diverge from the baseline.

3. Error Resolution Patterns: The agent framework was most effective at resolving specific error categories where compiler feedback was unambiguous:

   • Syntax Errors: Reduced by 75%.
   • Undefined References: Reduced by 87%.
   • Missing Code: Reduced by 66.5%.
   • Conversely, errors with ambiguous feedback, such as Language Mismatch (e.g., generating Python instead of C) and Markdown Errors (failure to isolate code blocks), saw less significant improvements (35–43% reduction), indicating that the clarity of the compiler's error message is a limiting factor.

4. Task Characteristics: The study found no significant correlation between task categories, description length, or solution complexity and the degree of improvement provided by the compiler. The benefit was consistent across the diverse range of programming tasks.

Significance and Claims
The paper concludes that integrating software engineering tools, specifically compilers, is essential for enhancing LLM performance in program synthesis. The primary significance lies in the demonstration that access to feedback loops can compensate for model size. By enabling smaller models to iteratively refine their output based on compiler feedback, the need for computationally expensive, large-scale models can be reduced, thereby lowering the energy footprint of software engineering tasks.

The authors modestly claim that while their results are robust within the context of C and the RosettaCode dataset, the findings highlight a pathway toward more efficient, tool-augmented AI agents that prioritize iterative validation over brute-force model scaling. They note that the effectiveness of this approach is contingent on the model's ability to interpret compiler error messages accurately.