A Reference Framework for Empirical Studies on LLM-Based Code Generation: A Review-Grounded Synthesis

Technical Summary: A Reference Framework for Empirical Studies on LLM-Based Code Generation

Problem Statement
Large Language Models (LLMs) are increasingly utilized for code generation and related software engineering tasks. However, the empirical evaluation of these systems remains fragmented, creating significant barriers to reproducibility, comparability, and the accumulation of cumulative evidence. Existing studies vary widely in their tasks, problem sources, application contexts, quality attributes, metrics, empirical methods, model configurations, environments, and generated artifacts. This heterogeneity makes it difficult to synthesize findings across studies or identify common methodological practices. Furthermore, LLM-based software engineering introduces unique methodological challenges, including non-deterministic outputs, evolving model versions, opaque model behavior, prompt sensitivity, and the need to evaluate artifacts beyond functional correctness. Current guidelines for empirical software engineering are often insufficient for these specific challenges, necessitating tailored methodological guidance.

Methodology
To address this gap, the authors conducted a structured review of empirical studies involving LLMs in code generation and related tasks. The research followed a bottom-up synthesis process rather than proposing a framework a priori.

• Search Strategy: A structured search was performed in the ACM Digital Library using a Boolean query targeting LLMs, code generation/programming, and empirical evaluation. The search covered papers published between 2023 and 2025.
• Selection: The initial search yielded 75 papers. After applying inclusion and exclusion criteria (focusing on empirical evaluations of code generation, adaptation, repair, testing, etc., while excluding non-empirical vision papers), 32 papers were retained. Three additional papers were identified through snowballing, resulting in a final corpus of 35 studies.
• Data Extraction: For each study, the authors extracted and standardized data regarding coding tasks, application contexts, problem sources, quality attributes, evaluation metrics, empirical methods, LLM configurations, generated outputs, and comparative setups.
• Synthesis: A bottom-up coding process was used to identify recurring concepts within the extraction fields. These concepts were grouped into lower-level categories and then abstracted into higher-level dimensions that recurred across the reviewed studies.

Key Contributions
The paper makes five primary contributions:

1. Synthesis of Empirical Practices: A comprehensive summary of empirical practices found in 35 selected studies, including controlled experiments, benchmark evaluations, repository mining, user studies, and mixed-methods designs.
2. Identification of Recurring Dimensions: The identification of nine major methodological dimensions: tasks, problem sources, application contexts, quality attributes, metrics, empirical methods, model configurations, environments, and generated outputs.
3. Reference Framework Proposal: The derivation of a bottom-up reference framework consisting of six core components to support the design, reporting, and comparison of empirical studies on LLM-based code generation.
4. Traceability Dataset: The provision of a coded traceability dataset linking the reviewed studies to the framework dimensions to support transparency and future evidence synthesis.
5. Illustrative Application: Demonstrations of how the framework can be used both retrospectively to map existing studies and prospectively to derive new empirical study designs.

Results and Framework Components
The analysis of the 35 studies revealed substantial methodological variability. The most frequent empirical practices were controlled experiments/evaluations (23 studies) and prompt strategy comparisons (22 studies). The evaluation scope extends beyond functional correctness to include efficiency, reliability, security, fairness, license compliance, explanation quality, and human-centered concerns.

Based on the synthesis, the authors propose a reference framework organized into six core components:

1. Coding Task: Captures the activity being evaluated, including task type (e.g., general generation, repair, explanation), application context (e.g., competitive programming, security, data science), and problem source (e.g., benchmarks, repositories, documentation).
2. Quality and Metrics Evaluation: Defines the quality attributes (e.g., correctness, security, fairness) and the specific metrics used to operationalize them (e.g., pass@k, runtime, human ratings, vulnerability rates).
3. Empirical Research Design: Encompasses the methodological structure, including the study method (e.g., controlled experiment, user study), experimental design (hypotheses, variables), comparative setup (e.g., LLM vs. Human, cross-model), and data analysis techniques.
4. Environment: Captures the computational and execution settings, including infrastructure (cloud, local), execution/test environments (Docker, online judges), and human/organizational settings (classrooms, user studies).
5. LLM Configuration: Covers model selection (proprietary vs. open-source), prompting strategies (e.g., chain-of-thought, retrieval augmentation), parameter tuning (temperature, sampling), and workflow strategies (fine-tuning, agent-based systems).
6. Generated Output: Describes the artifacts produced, ranging from code snippets and repository-level functions to tests, patches, explanations, and interaction traces.

Significance and Claims
The paper claims that this framework serves as a structured vocabulary for describing, comparing, and designing empirical studies. It is not intended to prescribe a single experimental methodology but rather to make the design dimensions of empirical studies explicit. By organizing the empirical design space, the framework aims to:

• Improve consistency, comparability, and reproducibility in future evaluations.
• Help researchers identify methodological gaps and reason about compatible study configurations.
• Support the planning of better-grounded empirical investigations by treating studies as configurations of interdependent choices rather than isolated model evaluations.

The authors emphasize that the framework is an evolving reference structure. It reflects the current state of the literature (2023–2025) and acknowledges that future refinements may be necessary to address emerging models, prompting workflows, and evaluation concerns. The framework is designed to be extensible, with future work planned to model compatibility constraints among components and develop tool support for empirical study design.