LLM-Assisted Empirical Software Engineering: Systematic Literature Review and Research Agenda

1. Problem Statement

Empirical Software Engineering (ESE) faces increasing challenges regarding data scale, methodological complexity, and reproducibility. While Large Language Models (LLMs) have emerged as powerful tools for software engineering tasks (e.g., code generation), their application as methodological instruments to support the research process itself (e.g., data collection, analysis, synthesis) remains fragmented.

Current literature lacks a systematic synthesis of:

• Which specific ESE research tasks and lifecycle phases LLMs support.
• How LLMs are integrated into empirical workflows (automation vs. augmentation).
• The reported benefits, limitations, and risks (e.g., hallucinations, bias).
• The extent to which studies report necessary details for reproducibility (prompts, configurations, validation).

Without this synthesis, researchers cannot critically assess the reliability of LLM-assisted workflows or establish guidelines for responsible adoption.

2. Methodology

The authors conducted a Systematic Literature Review (SLR) following established guidelines for secondary studies in software engineering.

• Scope & Timeframe: Peer-reviewed papers published between 2020 and 2025.
• Venues: 12 leading software engineering venues (CORE A*/A ranked), including conferences (ICSE, FSE, MSR, ESEM, etc.) and journals (TSE, EMSE, IST, etc.).
• Search Strategy: Due to the difficulty of capturing methodological LLM usage via title/abstract keywords alone, the authors:
  1. Downloaded full proceedings of the 12 venues (8,641 papers).
  2. Applied full-text keyword filtering (e.g., "LLM," "Generative AI," "ChatGPT," "Prompt").
  3. Manually screened 1,882 papers against inclusion/exclusion criteria.
• Selection Criteria:
  • Inclusion: Studies where LLMs support/automate ESE research tasks (screening, coding, synthesis) OR studies empirically examining LLMs within an ESE context.
  • Exclusion: Studies using LLMs solely for general software development (e.g., code generation tools) without an empirical research component.
• Final Dataset: 50 primary studies were selected for in-depth analysis.
• Data Analysis:
  • Qualitative: Descriptive coding of tasks, thematic analysis of benefits/limitations, and mapping to research lifecycle phases.
  • Quantitative: Frequency analysis of tasks, models, and reporting practices.
  • Reproducibility Assessment: Evaluated against 9 transparency guidelines (e.g., reporting model versions, prompts, human validation).

3. Key Contributions

1. Taxonomy of LLM-Assisted Tasks: Identification of 69 distinct tasks supported by LLMs, abstracted into functional roles (e.g., Semantic Classifier, Criteria-Based Screener, Evaluator).
2. Lifecycle Mapping: Analysis of how LLMs are distributed across the ESE research lifecycle (Planning, Data Collection, Processing, Analysis, Reporting).
3. Integration Patterns: Characterization of LLM involvement modes: Automation, Augmentation, Decision Support, and Evaluation.
4. Benefits & Limitations Synthesis: A thematic map of reported advantages (efficiency, scalability) and risks (hallucinations, prompt sensitivity, reproducibility issues).
5. Reproducibility Audit: An assessment of reporting practices, revealing significant gaps in transparency (e.g., missing prompts or configuration details).
6. Research Agenda: A roadmap for future work focusing on human-AI collaboration, agentic systems, and standardized reporting.

4. Key Results

A. Tasks and Roles (RQ1)

• Dominant Roles: LLMs are primarily used for Classification (17 tasks), Screening/Filtering (14 tasks), and Evaluation (11 tasks).
• Underutilized Roles: Complex reasoning, cross-domain abstraction, and integrative analysis are rarely explored.
• Context: Tasks are mostly structured and well-defined (e.g., classifying commit messages, screening paper titles).

B. Research Phases (RQ2)

• Concentration: LLM usage is heavily concentrated in Data Processing and Analysis & Synthesis phases.
• Methods: Most prevalent in Mining Software Repositories (MSR) and Controlled Experiments/Case Studies.
• Gaps: Limited adoption in Planning & Design and Reporting phases.

C. Integration Modes (RQ3)

• Automation Dominance: 60.9% of tasks are fully automated (LLM performs the task with minimal human intervention).
• Limited Collaboration: Only 18.8% are Augmentation (LLM assists human) and 1.5% are Decision Support (LLM suggests options for human choice).
• Implication: LLMs are currently used to replace decision-making steps rather than to support human reasoning.

D. Benefits and Limitations (RQ4)

• Benefits: Primarily cited as Efficiency Gains (reduced manual effort) and Scalability (handling large datasets). Analytical performance (semantic understanding) is also highly valued.
• Limitations:
  • Cognitive/Knowledge: Hallucinations, classification errors, and lack of domain understanding.
  • Input Sensitivity: High dependency on prompt engineering; small changes in prompts yield different results.
  • Reproducibility: Inconsistency in outputs and lack of stable formatting.
  • Reporting Gap: Many studies (29/50) did not report limitations; benefits are often assumed rather than empirically validated within the study.

E. Reproducibility (RQ5)

• Reporting Deficiencies: While model versions are often reported, critical details are missing:
  • Prompts: Often incomplete or missing.
  • Configurations: Temperature, token limits, and rationale for settings are frequently omitted.
  • Validation: Human validation procedures are rarely described in detail.
• Conclusion: Current reporting practices are insufficient to reproduce LLM-assisted empirical analyses.

5. Significance and Research Agenda

Significance:
This paper establishes the first comprehensive view of LLMs as research instruments in ESE. It highlights a critical disconnect: while LLMs offer transformative potential for scalability, current adoption is automation-centric, task-specific, and methodologically opaque. The lack of transparency threatens the validity and reproducibility of empirical findings.

Proposed Research Agenda:
The authors outline a shift from "automation" to "collaboration" and "discovery":

1. Human-AI Collaboration: Move beyond full automation to interactive, human-in-the-loop workflows where LLMs provide suggestions rather than final decisions.
2. Agentic AI: Explore agentic systems capable of planning multi-step workflows, tool use, and iterative reasoning.
3. Discovery Support: Investigate LLMs for hypothesis generation, exploratory analysis, and uncovering novel patterns.
4. Lifecycle Expansion: Extend LLM support to Planning (study design) and Reporting phases.
5. Standardization: Establish and enforce strict reporting guidelines (prompts, configs, validation) to ensure reproducibility.
6. Diversification: Reduce reliance on a single model family (e.g., ChatGPT) to improve external validity.

In summary, the paper argues that for LLM-assisted ESE to mature, the community must move beyond treating LLMs as simple automation tools and instead develop rigorous, transparent, and collaborative frameworks that integrate them responsibly into the scientific method of software engineering.