Evaluating LLM-Generated Code: A Benchmark and Developer Study

Problem Statement
Current evaluation methods for Large Language Models (LLMs) in code generation (NL2Code) predominantly rely on benchmarks that measure solution correctness through unit tests on isolated, function-level tasks (e.g., HumanEval, MBPP). While effective for establishing baseline capabilities, these benchmarks fail to reflect real-world programming scenarios. Real-world coding often involves creating multi-component projects with dependencies, requiring sequential prompting, and necessitating code that is not only correct but also readable, maintainable, and production-ready. Furthermore, there is a scarcity of research involving human experts to evaluate the quality and usability of LLM-generated code, as most evaluations rely solely on automated metrics or "LLM-as-a-Judge" approaches, which may not fully capture developer-centric concerns like structure and documentation.

Methodology
The paper proposes a custom three-fold evaluation methodology designed to bridge the gap between isolated function testing and holistic project evaluation. This framework integrates automated correctness testing, automated code quality verification, and structured human evaluation.

1. Coding Assignment: The study utilizes a complex, multi-component computer science project titled "Building the Tree of Life from Scratch," adapted from the NIFTY Assignments list. The task involves bioinformatics (phylogenetic trees) and requires implementing dynamic programming, recursion, graphs, and tree structures. The LLMs are prompted via a fixed set of 14 sequential prompts to generate the solution from scratch, without starter code, mimicking a real developer workflow.
2. Correctness Benchmark: A custom benchmark was developed to verify the solution at multiple checkpoints rather than just the final output. This includes:
   • Verifying Needleman-Wunsch scores.
   • Validating Newick format syntax (for tree structure and node distances).
   • Checking phylogenetic tree structure.
   • Validating clustering results based on thresholds.
   • Confirming the generation of a dendrogram PNG file.
   • Technical Verification: A manual step was included where researchers corrected minor syntax, compilation, and dependency errors to ensure the code could run, documenting these fixes to assess "fixability."
3. Automated Quality Verification: The generated code was analyzed using SonarQube to measure reliability, maintainability, and security, and to count code smells and technical debt.
4. Human-Based Quality Assessment (Developer Study): A structured survey was designed to gather code reviews from volunteer developers. The survey utilized a 5-point Likert scale across 14 metrics categorized into:
   • Structure of the code base (naming, formatting, separation of concerns).
   • Quality of comments and explanations (usefulness, placement, README quality).
   • Ease of usage (modifiability, testability).
   • An open-ended field for additional comments and a final question asking which code the developer would choose to work with.

Experimental Setup
The methodology was applied to three general-purpose LLMs: GPT-4.1, DeepSeek-V3-0324, and Claude Opus 4. Each model was run three times with fixed parameters (temperature 0.2, max tokens 4096) to account for non-determinism. For the human survey, one solution per model (selected based on the highest correctness pass rate) was reviewed by 10 volunteer developers recruited via LinkedIn.

Key Results

• Correctness: DeepSeek-V3-0324 achieved the highest correctness score, passing 5 out of 7 test cases in its best run. GPT-4.1 was the only model to generate solutions that ran without manual intervention in two of its three attempts, though it passed fewer test cases overall (3-4/7). Claude Opus 4 performed the worst in terms of correctness, with one attempt passing zero tests.
• Automated Quality: All models received 'A' ratings for security, reliability, and maintainability in SonarQube. However, DeepSeek solutions had the fewest code smells and the lowest estimated time to fix issues. GPT-4.1 solutions had the highest technical debt ratio and longest fix times.
• Human Survey: Despite DeepSeek having the best automated metrics, the developer survey revealed a divergence in preference. Developers rated GPT-4.1 the lowest, citing poor comments, chaotic structure, and hard-coded arguments. While DeepSeek was praised for parameterization, Claude Opus 4 received the most positive feedback regarding ease of modification and testing. Notably, when asked which code they would choose to work with, 6 out of 10 developers selected the Claude solution, while only 4 chose DeepSeek.
• Alignment: The study found that automated correctness metrics and human quality assessments complement each other but do not always align. A model with higher correctness (DeepSeek) was not necessarily preferred by developers over a model with slightly lower correctness but better structure and readability (Claude).

Significance and Contributions
The paper claims to make four primary contributions to the field:

1. A three-fold evaluation framework that combines automated testing, code quality verification, and developer surveys to capture both correctness and usability.
2. A project-level correctness benchmark based on a complex, multi-file assignment, moving beyond isolated function-level tasks.
3. A structured survey design dedicated to gathering code reviews from developers in a standardized manner, addressing the lack of human-centric evaluation in LLM research.
4. A demonstration of the framework through the evaluation of three LLMs, showing that developer reviews can yield critical findings—particularly regarding production-readiness and maintainability—that standard correctness benchmarks miss.

The authors conclude that while automated benchmarks are useful, a comprehensive evaluation of LLMs for code generation requires integrating human feedback to understand the practical utility and developer experience of the generated solutions. The study highlights that "correct" code is not always "good" code in a production context.