Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis

1. Problem Statement

Empirical Software Engineering (SE) and Mining Software Repositories (MSR) research face a "replication crisis" driven by the Garden of Forking Paths. This concept, introduced by Gelman and Loken, describes the vast, often unacknowledged flexibility researchers have in data processing, operationalization, and statistical modeling.

• The Issue: In MSR studies, particularly those using observational data (like Regression Discontinuity in Time, or RDiT), researchers must make numerous methodological decisions. Even when individual decisions are justifiable, the sheer number of combinations can lead to divergent, contradictory, or non-reproducible conclusions from the same dataset.
• The Gap: While sensitivity to methodological choices has been studied in psychology and epidemiology via Multiverse Analysis, it has not been systematically applied to SE. Existing SE literature acknowledges heterogeneity in methods but lacks empirical evidence quantifying how much these choices alter study outcomes.

2. Methodology

The authors conducted a Multiverse Analysis on a previously published MSR study: "The Silent Helper: The Impact of Continuous Integration on Code Reviews" (Cassee et al., 2023). This study used an RDiT design to analyze the impact of Continuous Integration (CI) adoption on code review communication and changes.

A. Experimental Design

The authors systematically re-ran the original analysis across 3,072 distinct analytical pipelines (universes). Each universe represented a unique combination of nine pivotal methodological decisions.

B. The Nine Decision Points

The decisions were categorized into data processing and statistical modeling:

1. Time Series Creation:
   • Number of Periods: {12, 18, 24, 36}
   • Period Length: {7, 15, 30, 45} days
2. Instability Exclusion: Days excluded before/after the intervention to account for transition noise.
   • Values: {(0,0), (3.5, 3.5), (15, 15), (0, 7), (0, 15)}
3. Data Aggregation: How data is summarized within a period.
   • Scaling: {Original, Natural Log, Log10}
   • Averaging: {Mean, Median}
4. Rounding Precision: Number of decimal places for data instantiation.
   • Values: {Unmodified, 10, 5}
5. Statistical Model Fitting:
   • Collinearity Threshold (VIF): {2.5, 5}
   • Fitting Algorithm: {REML (True/False)}

C. Outcome Classification

For each of the 3,072 universes, the authors fitted four models (one for each original hypothesis) and classified the results into four buckets relative to the original study:

1. Full Replication: All significance and signs match the original.
2. Unconfirmed Results: At least one claim cannot be confirmed (e.g., p-value > 0.05).
3. Opposite Results: At least one claim is reversed (e.g., significant negative trend vs. original positive trend).
4. Model Fit Failure: The model could not be fitted due to data issues.

3. Key Results

The analysis revealed extreme fragility in the original study's conclusions:

• Replication Rate: Only 6 out of 3,072 universes (0.2%) reproduced the original study's findings across all four hypotheses.
• Divergence:
  • ~33% of universes produced Unconfirmed Results (null findings).
  • ~33% produced Opposite Results (contradictory conclusions).
  • ~33% resulted in Model Fit Failures.
• Decision Sensitivity:
  • Time Modeling is Critical: The choice of Period Length was the most impactful decision. Changing this single variable altered the outcome in 86.3% of universes. Shorter periods (7 or 15 days) almost never replicated the original findings.
  • Hidden Decisions Matter: Even seemingly trivial choices, such as rounding precision, changed outcomes in ~2% of cases.
  • No Stable Decisions: No single decision point was "stable"; every decision had the power to flip the result.
• Specification Curves: Visual analysis showed that specific combinations of decisions (e.g., 45-day periods with 15-day exclusions) were required to replicate the original, while other reasonable combinations led to null or opposite results.

4. Key Contributions

The paper makes three primary contributions to the field of Empirical SE:

1. Empirical Evidence of Fragility: It provides the first quantitative proof that MSR studies using RDiT are highly sensitive to methodological choices. It demonstrates that the "richness" of MSR data is a double-edged sword, enabling high researcher freedom that threatens validity.
2. The Justification Ladder of Analytical Choices (JLAC): The authors propose a new framework (Table 5) to classify the strength of justifications for methodological decisions:
   • Level 5 (Causal): Based on causal models/theories.
   • Level 4 (Empirical): Based on data properties.
   • Level 3 (Heuristic): General reasoning.
   • Level 2 (Conventional): Common practice.
   • Level 1 (Implicit): Software defaults/unknown choices.
   • Application: The authors applied JLAC to the original study, finding that most decisions (e.g., period length) were Level 1 (Implicit) or Level 2 (Conventional), explaining the fragility.
3. Multiverse Analysis as a Tool: They advocate for Multiverse Analysis as a practical "smoke test" for SE researchers. It helps identify which decisions drive results and forces researchers to either justify their choices rigorously or acknowledge the exploratory nature of their findings.

5. Significance and Implications

• Reproducibility Crisis: The study confirms that the "Garden of Forking Paths" is a major threat to SE research. A single dataset can support conflicting narratives depending on arbitrary analytical choices.
• Methodological Rigor: The paper argues that standard "threats to validity" sections are insufficient. Researchers must move up the JLAC, grounding decisions in causal theory or empirical validation rather than defaults or convention.
• Actionable Recommendations:
  • Researchers should perform Multiverse Analyses or robustness checks before publishing.
  • Authors must explicitly justify every analytical decision, moving away from "implicit" defaults.
  • For RDiT specifically, domain-specific guidelines for time modeling (period length, exclusion windows) need to be established based on theory or simulation, not just convention.
• Future Work: The authors call for applying multiverse analysis to other MSR studies and research designs to map the sensitivity of the broader field.

In conclusion, the paper serves as a cautionary tale: in empirical software engineering, methodological choices exert a more profound influence on outcomes than previously acknowledged, and without rigorous justification and transparency, scientific conclusions may be artifacts of the analytical path chosen rather than the data itself.