Technical Summary: Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks

Problem Statement

While Large Language Models (LLMs) have achieved significant success in code generation, the reliance on benchmarks and leaderboards often obscures the specific tasks where models consistently fail. Current evaluation frameworks provide quantitative rankings but offer limited insight into the nature of these failures. Understanding the specific tasks that challenge LLMs and the underlying causes of these failures is critical for guiding both benchmark design and the development of more capable models. Prior work has analyzed errors in specific benchmarks like HumanEval but lacks a systematic, cross-benchmark analysis of failure patterns and their relationship to solution complexity.

Methodology

The authors conducted an empirical study across four widely used function-level code generation benchmarks: HumanEval, MBPP (Most Basic Python Problems), LiveCodeBench, and BigCodeBench (specifically the BCB-Hard subset).

• Models Evaluated: Six representative advanced LLMs were tested: Claude Sonnet-4, DeepSeek-V3, Qwen3-Coder, GPT-4o, Llama-3.3-70B, and Mistral-3.2-24B.
• Experimental Setup: The study utilized the latest versions of the benchmarks to minimize data contamination. A total of 865 tasks were evaluated. Each model generated a single solution per task (PASS@1), and success was determined by whether the solution passed all executable test cases.
• Complexity Analysis: The authors developed a tool to measure the static complexity of ground-truth solution code using six dimensions: Cyclomatic Complexity, Data Structures usage, Function Calls, Code Length, Nesting Depth, and Recursions. They employed Spearman correlation and linear regression to analyze the relationship between these complexity metrics and model failure rates.
• Failure Inspection: A fine-grained, task-by-task manual inspection was performed on 114 tasks that all six models consistently failed. This qualitative analysis aimed to identify recurring failure patterns and benchmark design issues.

Key Results

1. Task Difficulty and Failure Distribution

The study categorized tasks based on the number of models that failed them (0 to 6).

• HumanEval and MBPP: These benchmarks showed high solvability, with the majority of tasks solved by all models. Only a small fraction of tasks resulted in universal failure.
• LiveCodeBench and BCB-Hard: These benchmarks presented significantly higher difficulty. In BCB-Hard, 76 out of 148 tasks (approx. 51%) were failed by all models, with failure rates for individual models ranging from 68% to 77%. LiveCodeBench also showed a high number of tasks (35) that no model could solve.

2. Correlation with Code Complexity

The analysis of solution code complexity yielded nuanced results:

• LiveCodeBench: Showed a clear, statistically significant positive correlation between code complexity metrics (Cyclomatic Complexity, Function Calls, Code Length) and failure rates.
• Other Benchmarks: For HumanEval, MBPP, and BCB-Hard, the correlation between solution complexity and failure rates was weak and statistically non-significant. Regression analysis indicated that complexity metrics alone could not systematically explain failure rates in these benchmarks.

3. Identified Failure Patterns

The inspection of the 114 consistently failed tasks revealed four recurring patterns of LLM weakness:

1. Wrong Problem Mapping: Models misinterpret the task as a familiar problem class (e.g., treating a specific subsequence problem as a standard "balanced brackets" problem) and apply incorrect heuristics.
2. Flawed or Incomplete Algorithm Design: Models identify the correct general approach but fail to implement necessary steps or handle specific constraints (e.g., failing to account for non-monotonic trends in regression tasks).
3. Edge Case Mishandling: Code fails to handle boundary conditions or uncommon scenarios, such as failing to recursively traverse nested subdirectories when required.
4. Formatting Mistakes: The underlying logic is correct, but the output fails strict input/output requirements (e.g., returning integers instead of string literals).

Additionally, the authors noted that benchmark design issues, specifically ambiguous prompts and overspecified test cases, contribute to failures. In some instances, models that adhered strictly to literal prompt instructions (like Llama-3.3-70B) succeeded where more "capable" models failed because the latter applied reasonable but unrequested real-world assumptions (e.g., skipping network/broadcast IPs) that violated the benchmark's hidden test expectations.

Contributions

The paper makes three primary contributions:

1. Detailed Experimental Data: Provides task-level performance results for four major benchmarks across six leading LLMs, highlighting specific tasks where models consistently fail.
2. Complexity Correlation Method: Introduces a method for measuring solution code complexity and demonstrates that while complexity correlates with failure in some benchmarks (LiveCodeBench), it is not a universal predictor of failure across all benchmarks.
3. Fine-Grained Failure Analysis: Identifies four specific failure patterns and highlights the role of benchmark ambiguity (underspecified prompts vs. overspecified tests) in causing model failures.

Significance

The study concludes that code complexity alone is insufficient to explain LLM failures in code generation. The findings suggest that semantic task properties and benchmark-specific factors (such as prompt ambiguity and strict test constraints) play a more significant role. The identification of recurring failure patterns—particularly wrong problem mapping and edge case mishandling—provides a roadmap for future research. The authors argue that these insights are essential for designing better training strategies and creating benchmarks that more effectively discriminate between model capabilities, moving beyond simple pass/fail metrics to understand the nature of model limitations.