How Many Interviews Are Enough in a Software Engineering Study? Preliminary Findings on Sample Size and Saturation

Technical Summary: How Many Interviews Are Enough in a Software Engineering Study?

Problem Statement
Interview-based studies are a cornerstone of empirical software engineering (ESE) for investigating human, organizational, and socio-technical phenomena. However, the literature reports interview sample adequacy and saturation inconsistently. While methodological guidelines emphasize transparency and saturation, they generally avoid prescribing exact sample sizes. This lack of standardized reporting creates practical challenges for researchers during peer review, where interview-based work is sometimes dismissed as "too soft" or lacking methodological legitimacy. Reviewers frequently question the number of interviewees, particularly in studies involving marginalized populations, senior professionals, or specialized industrial contexts where recruitment is difficult. Although broader qualitative methodology literature offers numerical ranges and stopping criteria (e.g., saturation thresholds), there is limited empirical evidence regarding how software engineering studies actually operationalize and report these decisions in practice.

Methodology
The authors conducted a large-scale review of 429 software engineering papers published between 2016 and 2025. The study employed a manual search strategy across five major venues: ESEM, EASE, ICSE, FSE, and the Empirical Software Engineering (EMSE) journal. The selection process involved:

1. Automatic Filtering: A script identified 608 candidate papers based on titles, keywords, and abstracts containing terms associated with qualitative methods (e.g., case studies, grounded theory).
2. Manual Verification: Two authors manually screened the full text of candidates to confirm the actual use of interviews, resulting in a final corpus of 429 papers.
3. Data Analysis: The authors utilized descriptive statistics to characterize interview sample size distributions. They categorized sample sizes into ranges derived from qualitative research literature (e.g., <5, 5–12, 13–24, 25–40, etc.).
4. Preliminary Qualitative Analysis: To understand reporting practices, the authors performed an exploratory analysis on a stratified subset of 14 papers. This subset was selected to capture variation across sample sizes, including the paper with the highest count and a random selection from each size range. This analysis focused on how authors justified sample sizes and discussed saturation.

Key Results

• Sample Size Distribution: The 429 studies reported a total of 7,444 interviews, with an average of 17.35 participants per study. The distribution was not uniform; the most frequent range was 13 to 24 interviewees (n=152), followed closely by 5 to 12 interviewees (n=144). Studies with fewer than 5 participants were common (n=46), often associated with specialized or elite contexts, while studies with more than 60 participants were rare (n=9).
• Justification Practices: Justifications for sample sizes were heterogeneous. Smaller studies typically justified their sample size based on the specificity of the context, the uniqueness of participants, or limited accessibility (e.g., release managers, security specialists). Larger studies often justified their counts through broader organizational coverage, multisite designs, or heterogeneous stakeholder groups. Explicit methodological justification for the specific number of interviews was often limited, with many authors relying on contextual or practical reasoning rather than formal qualitative sampling guidelines.
• Saturation Reporting: Explicit discussion of saturation was infrequent, appearing in only four of the 14 papers analyzed in the preliminary qualitative subset. When saturation was discussed, it was primarily associated with grounded theory investigations involving iterative recruitment and theoretical sampling. Outside of grounded theory, saturation was rarely treated as a systematic stopping criterion; in some cases, it was used to adapt interview guides rather than to determine recruitment sufficiency. One study explicitly acknowledged that theoretical saturation was not achieved due to the rapidly evolving nature of the security context.

Key Contributions
This paper provides initial empirical insights into the methodological reporting practices of interview-based software engineering research. Its primary contributions include:

• Characterization of Current Practices: It establishes that moderate-sized samples (13–24 interviewees) are the most common configuration in ESE, challenging the notion that a single dominant sample size exists.
• Evidence of Reporting Gaps: It highlights that while sample sizes vary widely, the justification for these sizes and the discussion of saturation are often implicit or contextual rather than explicit and methodological.
• Contextual Validation: It supports the legitimacy of small-sample studies (<12 participants) when they are associated with specialized industrial contexts or constrained access, arguing against dismissing such work solely based on participant count.

Significance and Claims
The authors modestly frame these findings as "preliminary" and "exploratory," intended to characterize current practices rather than to prescribe rigid numerical thresholds. The paper claims that interview sample adequacy in software engineering is highly context-dependent, determined by research objectives, participant accessibility, and methodological design rather than fixed numerical expectations.

The significance of the work lies in its potential to improve transparency and rigor in qualitative reporting. The authors argue that the field should move away from rigid numerical expectations that may not fit the constraints of industrial research. Instead, they advocate for clearer articulation of how adequacy decisions are made. They suggest that while 13 to 24 interviewees represent the most empirically common range in software engineering and could serve as a practical starting point for planning, the central challenge is not finding a "universally correct" number, but rather improving the transparency of the decision-making process regarding sample size and saturation. The paper concludes that small samples should not be grounds for rejection if authors provide clear, context-sensitive justification. Future work is planned to extend the qualitative analysis to the full corpus of 429 papers to further refine these insights.