Do AI Models Dream of Faster Code? An Empirical Study on LLM-Proposed Performance Improvements in Real-World Software

1. Problem Statement

While Large Language Models (LLMs) have demonstrated proficiency in generating syntactically correct code and solving algorithmic puzzles, their ability to optimize complex, real-world software systems for performance remains unproven.

• The Gap: Existing benchmarks (e.g., HumanEval, CoderEval) focus on isolated, function-level algorithmic tasks. They fail to capture the complexity of real-world performance engineering, which involves concurrency, memory allocation, cross-file dependencies, and system-specific bottlenecks.
• The Concern: Current evaluations suggest LLMs are ready for "vibe coding" (generating features based on high-level prompts), but there is a lack of rigorous evidence regarding their ability to act as "Performance Engineers." Furthermore, previous studies suggest that code efficiency does not necessarily improve with larger foundation models, unlike correctness.
• The Question: Can LLMs autonomously identify performance hotspots and synthesize optimal performance improvements in production-grade Java code, or do they merely generate plausible but suboptimal or even regressive code?

2. Methodology

The authors conducted a rigorous empirical study using a custom dataset and an automated evaluation pipeline.

A. Dataset Construction: PerfOpt

• Source: 65 real-world performance improvement tasks mined from four high-performance open-source Java projects: Apache Kafka, Netty, Presto, and RoaringBitmap.
• Selection Criteria: Tasks were selected based on strict criteria:
  1. The commit fixed a demonstrable performance problem.
  2. The commit included developer-written JMH (Java Microbenchmark Harness) benchmarks.
  3. The benchmarks showed a statistically significant performance improvement (Wilcoxon signed-rank test, $p < 0.05$) with a medium-to-large effect size ($\hat{A}_{12} \ge 0.64$).
  4. Changes were concise (affecting $\le$ 3 files) to exclude massive refactoring while retaining multi-file complexity.
• Artifacts: Each task includes the original code, the developer's optimized solution, unit tests, JMH benchmarks, and a natural language description of the issue.

B. Experimental Setup

• Models Evaluated:
  • Commercial: OpenAI o4-mini, Gemini 2.5 Pro.
  • Open-Weight/Reasoning: DeepSeek-V3.2, DeepSeek-R1-0528 (with reasoning mode).
• Prompting Strategies: Four distinct contexts were tested to isolate the impact of information:
  1. No Hint: Code + generic request to improve performance.
  2. Benchmark: Code + JMH benchmark source (structural pointers).
  3. Problem: Code + natural language issue description (semantic clues).
  4. Both: Code + Benchmark + Problem description.
• Automated Repair Loop: An iterative process where implausible patches (compilation errors or failing unit tests) were fed back to the model with error messages for self-correction (max 2-3 attempts).
• Evaluation Metric: Patch Performance Score (PSS). A ratio comparing the average benchmark score of the LLM patch against the original code ($PSS > 1$ indicates improvement). Statistical significance was verified using the Wilcoxon signed-rank test and Vargha-Delaney $\hat{A}_{12}$ effect sizes.

3. Key Contributions

1. PerfOpt Dataset: The first dataset of 65 real-world, multi-file performance optimization tasks with rigorous JMH benchmarks, moving beyond synthetic algorithmic puzzles.
2. Empirical Evidence of Volatility: Demonstrated that while LLMs can improve code, their performance is highly volatile. They frequently cause regressions or fail to match human-level optimizations, contradicting the optimistic view derived from algorithmic benchmarks.
3. Diagnosis of Limitations: Identified two primary failure modes:
   • Inability to Profile: LLMs struggle to autonomously pinpoint performance hotspots without explicit guidance.
   • Sub-optimal Synthesis: Even when guided, LLMs often synthesize solutions with higher structural complexity (Cyclomatic Complexity) and fail to replicate the specific, often minimal, structural changes humans make.
4. Strategic Taxonomy: Introduced a classification system (Match, Alignment, Divergence) to analyze how closely LLM strategies mimic human developers, revealing that "Alignment" (finding the right bottleneck but using a different, often worse, algorithm) is the most common outcome.

4. Key Results

RQ1: How effectively can LLMs optimize real-world software performance?

• Human Lag: LLMs generally lag behind human developers. In 3 out of 4 projects, human solutions significantly outperformed all LLM configurations.
• Volatility: The landscape is characterized by extreme variance.
  • Regressions: LLMs frequently introduced performance regressions (e.g., in Netty and Presto tasks), sometimes degrading performance by 60%+ due to incorrect assumptions (e.g., adding unnecessary method overhead or breaking JIT inlining).
  • Novelty: In rare cases (e.g., specific Kafka tasks), LLMs discovered novel strategies (like avoiding locks entirely) that outperformed human developers.
• Prompt Sensitivity:
  • Problem Description is Key: Providing a natural language description of the problem was the single most effective driver for optimization.
  • Benchmark Code: Providing JMH code alone was insufficient; it helped slightly but did not replace the need for semantic context.
  • Combined Context: The "Both" strategy yielded the best results but still fell short of human baselines on average.

RQ2: Do LLM-proposed optimization strategies align with human solutions?

• Structural Differences: LLM patches are more localized (fewer files changed) but introduce higher Cyclomatic Complexity. Humans tend to simplify structure; LLMs tend to add elaborate logic.
• Strategy Classification:
  • Strategy Match (Identical logic): Rare (13–32% depending on model).
  • Strategy Alignment (Same bottleneck, different logic): Most common (45–59%). These solutions usually yield modest improvements (median PSS ~1.11).
  • Strategy Divergence (Wrong bottleneck): High in "No Hint" scenarios (59%), dropping to negligible levels with explicit problem descriptions.
• Model Behavior:
  • Gemini 2.5 Pro: Showed the highest rate of "Strategy Match" (32%) and best overall repair capabilities.
  • DeepSeek R1: Showed strong reasoning but low "Match" rates (16%), often diverging into complex, sub-optimal solutions.
  • OpenAI o4-mini: Had the lowest repair success rate and struggled to utilize feedback loops.

5. Significance and Implications

• Re-evaluating Benchmarks: The paper argues that current algorithmic benchmarks provide an overly optimistic assessment of LLM capabilities. Success in isolated functions does not translate to real-world performance engineering.
• Role of LLMs: LLMs should not be viewed as autonomous performance engineers. Their optimal role is as collaborative assistants:
  • Humans must provide the diagnosis (identifying the bottleneck) and the benchmark.
  • LLMs can then synthesize implementation patches.
• Future Directions: The authors advocate for Agent-Based Systems. Future tools must integrate LLMs with runtime profiling, tracing, and benchmarking tools to allow the AI to "observe" performance behavior rather than relying solely on static code analysis.
• Energy Efficiency: Since performance is linked to energy consumption, improving the reliability of AI-driven optimization is critical for reducing the carbon footprint of future software.

Conclusion

The study concludes that while LLMs possess the raw capability to generate performance improvements, they currently lack the diagnostic intuition and structural simplicity of human experts. They act as "high-variance search engines" rather than reliable engineers. Bridging this gap requires moving beyond static code generation toward agent-based systems that incorporate runtime profiling and human-in-the-loop guidance.