Technical Summary: FasterPy: An LLM-based Code Execution Efficiency Optimization Framework

Problem Statement

Code execution efficiency is a critical factor in software performance and user satisfaction. While traditional rule-based methods and algorithmic approaches (e.g., genetic algorithms, reinforcement learning) have addressed specific performance bugs like redundant loops, they suffer from high maintenance costs, limited adaptability to unseen scenarios, and heavy dependence on specific program representations or meticulously crafted datasets. Recent Machine Learning (ML) and Deep Learning (DL) methods offer promise but often require specific intermediate representations (e.g., LLVM IR, ASTs) and large, expensive training datasets, limiting their scalability. Furthermore, existing Large Language Model (LLM) approaches for code optimization often face challenges such as high computational costs, reliance on specialized hardware, or suboptimal efficiency due to a lack of targeted adaptation for performance-critical tasks.

Methodology: The FasterPy Framework

FasterPy is a low-cost, efficient framework designed to optimize the execution efficiency of Python code (and preliminarily C++) by adapting LLMs without requiring new architectures or massive retraining. It integrates Retrieval-Augmented Generation (RAG) with Low-Rank Adaptation (LoRA) fine-tuning. The framework operates through four primary steps:

1. Code Preprocessing:

   • The system extracts target functions and their essential context (global variables, class definitions, invoked functions) from the original source code using syntactic parsing (Tree-sitter) and Abstract Syntax Tree (AST) analysis.
   • Irrelevant code is removed to reduce input noise and length, preventing LLM hallucinations while preserving execution semantics.
   • The cleaned code is formatted using the Black formatter.

2. Code Embedding:

   • The cleaned code is converted into a semantic vector representation using UniXcoder (an Encoder-Only multimodal code model).
   • UniXcoder is chosen for its ability to incorporate raw code text, comments, and AST structure simultaneously, providing a robust semantic representation normalized via L2 normalization.

3. Knowledge Base Retrieval (RAG):

   • A specialized Code Efficiency Optimization Knowledge Base is constructed from the PIE (Performance Improving Code Edits) and Mercury datasets.
   • This database stores pairs of slow code and their corresponding optimization suggestions (natural language summaries) along with performance improvement metrics ($rate$).
   • Upon receiving an input, the system retrieves the top-$m$ semantically similar slow code snippets.
   • A dual-criteria filtration process selects the top-$k$ suggestions: it retains the most semantically similar record and filters the rest based on a similarity tolerance threshold, then ranks the remaining candidates by their historical performance improvement ($rate$). This ensures both relevance and practical effectiveness.

4. Code Generation (LoRA):

   • The cleaned input code and the filtered optimization suggestions (natural language summaries only, excluding the slow code examples to prevent contamination) are combined into a prompt.
   • This prompt is fed into an LLM that has been fine-tuned using LoRA. LoRA allows for parameter-efficient adaptation by training low-rank matrices while keeping the original model weights frozen.
   • The model generates an optimized version of the code ($C_{optimized}$) that maintains semantic correctness while improving execution time.
   • Post-processing extracts the code block from the model's output.

Key Contributions

The paper presents two primary contributions:

1. A Code Efficiency Optimization Knowledge Base:

   • Constructed from the PIE and Mercury benchmarks, this repository contains over 60,000 instances of low-efficiency code paired with high-efficiency counterparts.
   • Crucially, the authors used an LLM (Qwen-Max) to generate natural language summaries describing the optimization strategies and transformation intents for each pair.
   • The knowledge base is structured to support semantic retrieval, enabling the system to find relevant optimization patterns based on code semantics rather than exact string matching.

2. The FasterPy Framework:

   • A practical, modular framework that combines RAG and LoRA to optimize code execution efficiency under standard hardware settings.
   • It avoids the high costs of full model retraining and specialized hardware dependencies.
   • The framework is designed to be extensible to other programming languages, with preliminary evidence provided for C++.

Experimental Results

The framework was evaluated on the PIE benchmark (Python split) using a refined benchmarking program that reduces process-spawning overhead and improves timing accuracy. The study compared FasterPy-enhanced models against base models and the PIE baseline across five LLMs (Deepseek-Coder-6.7B, Qwen2.5-Coder-7B, CodeLLaMA-7B, Qwen-Max-0125, and GPT-5.1-Codex).

• Performance Gains: Models integrated with FasterPy consistently outperformed their base versions. For example, Deepseek-Coder-6.7B-Instruct-FasterPy achieved a Pass@1 of 0.7779 (up from 0.4029) and an @Speedup of 1.7111.
• Efficiency vs. Correctness: The framework significantly improved the %OPT (proportion of correctly optimized samples that are at least 10% faster). Qwen2.5-Coder-7B-FasterPy reached a %OPT of 0.3856, compared to 0.3072 for the base model.
• Comparison with Baselines: FasterPy outperformed the PIE framework (which uses retrieval-based few-shot prompting) across all metrics, including lower runtime, higher Pass@1, and higher speedup.
• Ablation Studies:
  • Component Ablation: Both RAG and LoRA contributed positively, with the full framework yielding the best results, indicating a complementary relationship where RAG provides context and LoRA enables the model to utilize it effectively.
  • Context Representation: Using natural language summaries only (Summary-only) in the prompt outperformed using code snippets or a mix of both, suggesting that summaries provide sufficient guidance without introducing noise.
  • Dataset Composition: Combining PIE and Mercury datasets yielded better results than using either alone, demonstrating the value of diverse optimization patterns.
• Cost-Effectiveness: While token consumption increased slightly (approx. 10-20%) due to retrieval context, the significant improvement in optimization effectiveness resulted in a favorable cost-performance ratio. The initial cost of dataset construction was amortized over the workflow.

Significance and Claims

The authors claim that FasterPy offers a practical and scalable solution for code execution efficiency optimization. Unlike traditional rule-based methods that are labor-intensive and rigid, or complex ML approaches that require specific representations and massive resources, FasterPy leverages the generative capabilities of LLMs enhanced by historical optimization data.

The paper emphasizes that:

• LLMs possess latent capacity for performance-aware code generation, which can be elicited through well-designed prompts incorporating retrieved optimization suggestions.
• The framework is low-cost and does not rely on expensive retraining or specialized hardware, making it suitable for industrial deployment.
• It demonstrates strong generalization, improving optimization capabilities across models of varying sizes (from 6.7B to >200B parameters) and showing preliminary success in C++.
• The approach moves beyond fixed heuristics by learning from historical optimization records, allowing for the discovery of diverse and potentially novel optimization strategies.

The authors remain modest regarding the current scope, acknowledging that the evaluation is primarily on Python and competitive programming datasets, and that future work is needed to validate the framework on broader industrial codebases and additional languages.