Technical Summary: Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs

Problem Statement

Current evaluations of Large Language Models (LLMs) for code generation frequently rely on compact, single-language benchmarks and aggregate pass rates (e.g., pass@1). The authors argue that these summaries obscure critical dimensions of model performance, including variability across programming languages, problem families, difficulty levels, and specific failure modes. Furthermore, standard benchmarks often fail to distinguish between functional correctness and code maintainability (static quality), and they do not adequately capture the nature of failures (e.g., syntactic vs. semantic errors). This study addresses the gap by proposing a large-scale, execution-grounded evaluation that preserves the full generation-to-execution pipeline to reveal tradeoffs hidden by single-metric leaderboards.

Methodology

Corpus Construction

The authors constructed a benchmark using 2,707 free (non-premium) LeetCode problems to ensure reproducibility. The corpus spans 12 programming languages: C, C++, C#, Go, Java, JavaScript, PHP, Python, Ruby, Scala, Swift, and TypeScript.

• Scale: The study generated 325,343 problem–model–language jobs.
• Models: Nine openly accessible or open-weight instruction-tuned code LLMs were evaluated, including Yi-Coder-9B-Chat, Qwen2.5-Coder-14B-Instruct, Gemma-2-27B-IT, DeepSeek-Coder-V2-Lite-Instruct, Codestral-22B-v0.1, Phi-4, Meta-Llama-3.1-8B-Instruct, Granite-3.1-8B-Instruct, and Stable-Code-Instruct-3B.
• Data Preservation: Each job retains the prompt, raw model response, extracted code, official LeetCode execution outcome, and static-analysis signals.

Execution Pipeline

1. Prompting: Language-specific prompts were generated for each problem–language pair.
2. Inference: A distributed server-backed job queue managed inference across local models (hosted on LM Studio) to handle the scale of 300k+ jobs. Operational errors (network, timeouts) were distinguished from model failures and handled via retries.
3. Code Extraction: Post-processing extracted code blocks from model responses (specifically triple-quoted sections) to serve as candidate submissions.
4. Evaluation: A Python automation script using Selenium submitted extracted code to the official LeetCode judge. Results were retrieved via GraphQL, capturing statuses such as Accepted, Wrong Answer, Compile Error, Runtime Error, Time Limit Exceeded, and Memory Limit Exceeded.
5. Metric Selection: For each job, the "best" submission was selected based on highest correctness ratio, then accepted status, then timestamp.
6. Static Analysis: The study performed static analysis (linting) on accepted code artifacts using ast-grep and Pylint-style tools to measure code quality, complexity, and adherence to conventions.

Key Contributions

1. Large-Scale Archive: The construction of a reproducible execution archive covering 2,707 problems, 12 languages, and 9 models, containing over 325,000 execution jobs.
2. Multidimensional Evaluation: An evaluation framework that moves beyond aggregate scores to analyze performance across mean correctness, accepted-problem coverage, difficulty slices, language slices, topic categories, pairwise head-to-head outcomes, and failure composition.
3. Joint Functional and Static Analysis: A simultaneous analysis of functional correctness and static quality, demonstrating that execution success and maintainability-oriented signals can diverge significantly.

Experimental Results

Model Performance and Rankings

• Mean Correctness: Yi-Coder-9B-Chat achieved the highest mean correctness (23.64%), followed by Qwen2.5-Coder-14B-Instruct and Gemma-2-27B-IT. However, all models remain far below the human acceptance baseline of 57.2%.
• Problem Coverage: Qwen2.5-Coder-14B-Instruct demonstrated the broadest accepted-problem coverage (77.1%), indicating it can solve a wider variety of distinct problems even if its average job-level correctness is slightly lower than Yi-Coder.
• Difficulty Slices: Rankings are slice-dependent. Yi-Coder leads on easy and medium problems, while Qwen2.5-Coder-14B-Instruct performs best on hard problems.
• Language Slices: Model quality is not language-invariant. Yi-Coder is strongest in conventional compiled/scripting targets (C++, Java, Python), Qwen is competitive in C#, JavaScript, Ruby, Swift, and TypeScript, and Gemma is strongest in Go and Scala.
• Topic Slices: Rankings vary by topic. For instance, Yi-Coder is consistently strong across high-frequency topics like arrays and strings, while Qwen leads in specific algorithmic categories.

Failure Analysis

• Dominant Failure Mode: Compile errors account for 63.25% of non-accepted best submissions. This indicates that the majority of failures occur before semantic correctness can be tested.
• Failure Shift: Stronger models tend to reduce the share of compile errors, exposing relatively more runtime or wrong-answer cases.
• Partial Correctness: The archive shows that wrong-answer cases are concentrated in the zero-percent passed-testcase bin, with limited evidence of intermediate "near-miss" success.

Static Quality vs. Functional Correctness

• Divergence: Functional correctness and static cleanliness measure different properties. Stable-Code-Instruct-3B achieved the strongest structural quality score despite having the weakest functional performance.
• Lint Pass Rates: Gemma-2-27B-IT achieved the highest all-language lint pass rate. Conversely, Qwen, despite strong execution performance, passed lint rules less often.
• Issue Types: The static analysis burden is dominated by warnings (141,376 findings) and conventions (45,537 findings) rather than hard errors. Common convention issues include using-namespace-std (48% of convention findings) and wildcard imports. Refactor findings were rare (402 total).

Significance and Claims

The paper claims that its significance lies in demonstrating that multilingual, artifact-preserving evaluation reveals tradeoffs hidden by single-language or single-metric leaderboards.

• No Single Best Model: The "best" model depends entirely on the downstream objective. If the goal is average correctness, Yi-Coder-9B-Chat is preferred; if the goal is coverage of distinct problems or success on hard tasks, Qwen2.5-Coder-14B-Instruct is superior; if the goal is minimizing review burden via clean code, Gemma-2-27B-IT or Stable-Code-Instruct-3B may be preferable.
• Reliability Gaps: The study highlights that current open models are far from human-level reliability, with compile errors being the primary barrier to execution.
• Evaluation Rigor: The authors argue that code-model evaluation must preserve execution, language coverage, failure modes, and quality signals. Relying on a single score collapses performance dimensions that are critical for practical deployment, such as whether a model produces maintainable code or fails due to syntax rather than logic.

The paper concludes that evaluating code-generating LLMs requires a shift from identifying a single "winner" to understanding how and why models succeed or fail across diverse programming contexts.