💻 computer science

Guidelines for Empirical Studies in Software Engineering involving Large Language Models

This paper presents a collaborative framework of seven study types and eight mandatory or recommended guidelines to enhance the reproducibility and rigor of empirical software engineering studies involving Large Language Models, supported by an applicability matrix, reporting checklist, and a living online resource.

Original authors: Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón, Chunyang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Junda He, Christoph Treud

Published 2026-03-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón, Chunyang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Junda He, Christoph Treude, Marcos Kalinowski, Stefano Lambiase, Daniel Russo, Mircea Lungu, Cristina Martinez Montes, Lutz Prechelt, Paul Ralph, Rijnard van Tonder, Stefan Wagner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of software engineering research as a massive, bustling kitchen where chefs (researchers) are trying to invent new recipes (tools and theories) to help cooks (software engineers) make better meals (software).

For a long time, these chefs followed strict rules to ensure that if someone else tried to make the same dish, it would taste exactly the same. But recently, a new, magical ingredient has entered the kitchen: Large Language Models (LLMs). Think of LLMs as a super-smart, but slightly unpredictable, sous-chef.

This new sous-chef is amazing. It can chop vegetables (code), write recipes (documentation), and even taste-test dishes (review code) incredibly fast. However, it has three weird quirks:

It's a bit moody: If you ask it to make the same dish twice, it might add a pinch more salt the second time, even if you gave it the exact same instructions.
It forgets its history: We don't always know exactly what ingredients it learned from in the past, and the "recipe book" it uses changes constantly without us knowing.
It's a black box: Sometimes, we can't see how it decided to add that extra salt.

Because of these quirks, if a researcher says, "I made a great cake using this magical sous-chef," other chefs can't replicate the cake. They don't know exactly which version of the sous-chef was used, what the exact instructions were, or if the cake would taste different tomorrow.

This paper is a "New Kitchen Handbook" written by a team of 22 expert chefs. Their goal is to teach everyone how to use this magical sous-chef without ruining the science of cooking. They've organized the kitchen into 7 different ways the sous-chef is used (like "The Taster," "The Recipe Writer," or "The Fake Customer") and created 8 Golden Rules to follow.

Here are the 8 rules, explained with simple analogies:

1. 📢 The "Honest Chef" Rule (Declare Usage)

The Rule: You must admit you used the sous-chef.
The Analogy: If you put a secret spice in your soup, you have to tell the food critics. Don't just say "I made this soup." Say, "I used the Magic Sous-Chef v2.0 to chop the onions." If you hide it, people can't judge if the soup is actually good or just lucky.

2. 📝 The "Exact Receipt" Rule (Report Version & Settings)

The Rule: Write down the exact model name, the date you used it, and the settings (like "temperature" or "creativity level").
The Analogy: Imagine a recipe that just says "add some flour." That's useless! You need to say "2 cups of King Arthur Flour, measured on a Tuesday." Since the magical sous-chef changes its mind often, you need to record the exact moment and settings so someone else can try to copy your result later.

3. 🏗️ The "Blueprint" Rule (Report Architecture)

The Rule: Don't just talk about the sous-chef; explain the whole kitchen setup around it.
The Analogy: The sous-chef doesn't work in a vacuum. It's connected to a fridge, a timer, and a list of instructions. If you built a robot chef, you need to draw the blueprints of the robot, not just say "it uses a brain." How does the robot talk to the oven? What happens if the internet cuts out? Show the whole machine.

4. 🗣️ The "Script" Rule (Report Prompts & Logs)

The Rule: Share the exact questions you asked the sous-chef and its answers.
The Analogy: If you ask a genie, "I wish for a million dollars," and it gives you Monopoly money, the problem might be how you asked. You need to show the exact script you gave the genie. Also, keep a diary of the conversation. If the genie changes its mind next week, you need to know what it said today.

5. 👨‍🍳 The "Human Taste-Test" Rule (Human Validation)

The Rule: Don't trust the robot blindly; have a human taste the food.
The Analogy: The sous-chef might think a dish is perfect because it looks pretty, but a human might say, "This tastes like cardboard." You need to compare the robot's work with a human expert's work to make sure the robot isn't just hallucinating. If the robot says "This code is safe," a human needs to double-check it.

6. 🆓 The "Open Source" Rule (Use an Open Baseline)

The Rule: If you use a fancy, expensive, closed-door sous-chef (like a secret company's AI), also test a free, open-source one.
The Analogy: If you say, "My secret sauce is the best!" but you won't let anyone see the recipe, we don't believe you. Show us that your secret sauce is better than a simple, homemade sauce we can all make ourselves. This proves your results aren't just because of a magic trick hidden behind a paywall.

7. 📏 The "Fair Ruler" Rule (Use Good Metrics)

The Rule: Use the right tools to measure success, and explain why.
The Analogy: You can't measure the "tastiness" of a soup with a ruler. You need a taste-test score. Don't just say "It worked!" Explain how you measured it. Did you count how many bugs were fixed? Did you ask users if they liked it? And remember, sometimes the ruler itself is broken (bad benchmarks), so admit that too.

8. 🚧 The "Honest Mistakes" Rule (Report Limitations)

The Rule: Admit where your experiment might have gone wrong.
The Analogy: No chef is perfect. Maybe your kitchen was too hot, or maybe the ingredients were old. Be honest: "The robot worked great, but only on Tuesdays," or "We only tested this on pizza, not on sushi." By admitting the flaws, you help others avoid them.

The Big Picture

This paper is essentially a quality control manual for the future of software research. It says: "The magical sous-chef is here to stay, and it's powerful. But if we don't write down exactly how we use it, we'll never know if the results are real or just a fluke."

By following these rules, researchers can ensure that their discoveries are solid, reproducible, and actually helpful to the world, rather than just a one-time magic trick that disappears when the model updates.

1. Problem Statement

The rapid integration of Large Language Models (LLMs) into Software Engineering (SE) research and practice has introduced significant threats to the reproducibility and replicability of empirical studies. Unlike traditional software tools, LLMs possess three critical characteristics that undermine scientific rigor:

Non-determinism: Identical prompts can yield different outputs due to stochastic decoding, floating-point arithmetic variations, and batching effects.
Opacity: Commercial models often lack transparent training data, and their internal configurations evolve silently over time (e.g., API updates), making "versioning" difficult.
Rapid Evolution: Model performance can degrade or shift significantly within months, rendering longitudinal comparisons obsolete.

Existing empirical guidelines (e.g., for controlled experiments or case studies) do not address these LLM-specific challenges. Consequently, many SE studies involving LLMs lack the necessary transparency to be verified, replicated, or trusted by the community.

2. Methodology

The authors employed a collaborative, expert-driven methodology similar to the development of clinical trial guidelines (CONSORT) and medical prediction guidelines (TRIPOD-LLM).

Collaboration: The guidelines were co-developed by 22 researchers from the International Software Engineering Research Network (ISERN) and other institutions.
Process:
1. Initiation: Discussions began at the 2024 ISERN meeting and were refined through a position paper presented at CHASEAI 2024.
2. Iterative Refinement: The team held bi-weekly meetings to define a taxonomy of study types and draft guidelines.
3. Peer Review: Content was assigned to co-author pairs for refinement, followed by multiple rounds of internal peer review and external expert review.
4. Validation: The guidelines were self-assessed against the SIGSOFT Empirical Standards for quality dimensions (comprehensiveness, usefulness, argumentation, and integration).
Scope: The guidelines focus on text-based LLM use (natural language and programming languages) in direct research support (data collection, analysis, synthesis) and engineering practice (AI4SE tools), excluding peripheral tasks like spell-checking.

3. Key Contributions

A. Taxonomy of Seven Study Types

The paper introduces a taxonomy categorizing how LLMs are involved in SE research. This contextualizes the guidelines, as different study types have different requirements:

LLMs as Annotators (S1): Automating the coding/labeling of qualitative data.
LLMs as Judges (S2): Evaluating software artifacts (e.g., code quality, requirement adherence).
LLMs for Synthesis (S3): Integrating information to generate insights, theories, or synthetic datasets.
LLMs as Subjects (S4): Simulating human participants in surveys or experiments.
Studying LLM Usage (S5): Observational studies on how engineers use LLM tools in practice.
LLMs for New Tools (S6): Developing and evaluating new SE tools powered by LLMs (including agentic systems).
Benchmarking LLMs (S7): Evaluating model performance on standardized tasks.

B. Eight Core Guidelines

The paper proposes eight guidelines, distinguishing between "Must" (requirements) and "Should" (recommended practices).

ID	Guideline	Core Requirement (Must/Should)
G1	Declare LLM Usage and Role	Must: Explicitly disclose LLM use, specific model, role, and research stage.
G2	Report Model Version, Config, & Customizations	Must: Report exact model version, date, parameters (temp, seed), and fine-tuning details.
G3	Report Tool Architecture Beyond Models	Must: Describe system architecture, agent roles, retrieval mechanisms, and infrastructure.
G4	Report Prompts, Development, and Logs	Must: Publish all prompts/templates and interaction logs (where privacy allows).
G5	Use Human Validation for LLM Outputs	Should: Validate subjective outputs against human judges with inter-rater reliability metrics.
G6	Use an Open LLM as a Baseline	Should: Include an open-weight model baseline to ensure replicability.
G7	Use Suitable Baselines, Benchmarks, and Metrics	Must: Justify metric choices, report descriptive statistics for non-determinism, and use inferential statistics.
G8	Report Limitations and Mitigations	Must: Transparently report threats to validity (internal, external, construct) and mitigation strategies.

C. Supporting Artifacts

Applicability Matrix: A table mapping each guideline to the seven study types, indicating which are mandatory ("Must") and which are recommended ("Should") for each type.
Reporting Checklist: A CONSORT-inspired checklist organized by paper sections (Introduction, Methods, Results, etc.) to assist authors and reviewers.
Living Resource: The guidelines are maintained online at llm-guidelines.org to allow community updates as the technology evolves.

4. Results and Findings

The primary "result" of this work is the establishment of a community-agreed standard for LLM-based empirical research. Key findings from the development process include:

Reproducibility is achievable but requires effort: While exact bit-for-bit reproducibility is often impossible with SaaS models, "reasonable efforts" (e.g., reporting fingerprints, seeds, and interaction logs) can make studies auditable.
Human-in-the-loop is critical: Automated metrics alone are insufficient for validating subjective constructs (e.g., code readability, requirement quality). Human validation with established inter-rater reliability (e.g., Krippendorff's $\alpha$ ) is essential.
Context matters: The "best" guideline depends on the study type. For example, using an open baseline (G6) is mandatory for benchmarking but often infeasible for observational studies of proprietary tool usage.
Prompt engineering is a scientific variable: Prompts must be treated as experimental artifacts, requiring version control, changelogs, and full disclosure, not just as "instructions."

5. Significance

This paper is a foundational contribution to the field of Empirical Software Engineering for several reasons:

Restoring Scientific Rigor: It addresses the "reproducibility crisis" in LLM research by providing concrete, actionable steps to mitigate non-determinism and opacity.
Standardization: It moves the community away from ad-hoc reporting toward a unified framework, enabling better cross-study comparisons and meta-analyses.
Reviewer Empowerment: It equips reviewers with specific criteria to assess the validity of LLM-based studies, reducing the acceptance of opaque or irreproducible research.
Ethical and Practical Awareness: It highlights critical issues like data contamination, bias in synthetic data, and the environmental cost of LLM experiments, urging researchers to consider these in study design.

In summary, the paper provides the definitive roadmap for conducting, reporting, and reviewing empirical software engineering research in the era of Large Language Models, ensuring that the field advances with transparency and scientific integrity.