The State of Open Science in Software Engineering Research: A Case Study of ICSE Artifacts

1. Problem Statement

While open science practices (sharing code, data, and tools) are increasingly mandated and encouraged in Software Engineering (SE) research, there is a critical gap between the availability of replication packages and their practical usability.

• The Gap: Although artifact sharing rates at top venues like ICSE have risen significantly (nearly 80% of papers from 2015–2024 included links), it is unclear if these artifacts are actually executable or reproducible.
• The Issue: Many shared artifacts suffer from undocumented environmental dependencies, missing files, broken instructions, or configuration leakage, rendering them unusable for validation or reuse.
• The Goal: The authors aim to empirically assess the state of open science by evaluating the executability, effort required, modification types, and reproducibility of replication packages from the past decade.

2. Methodology

The study employed a rigorous, hands-on empirical approach involving the analysis of 100 replication packages from ICSE research-track papers published between 2015 and 2024.

Dataset Construction

• Source: 1,372 ICSE papers were collected.
• Filtering: Papers were screened for artifact links. 1,085 had links; 941 were accessible; 796 contained executable components.
• Sampling: A stratified random sampling strategy was used to select 100 artifacts, ensuring temporal representativeness across the 10-year span.

Execution Protocol

Two experienced software engineers (with 12+ and 15+ years of experience) executed the artifacts in isolated environments to prevent host-system contamination.

• Environment Setup: Used Docker containers, Conda environments, and specific hardware configurations (including GPUs) to match original requirements. When specifications were missing, they applied a "two-year backward buffer" heuristic for OS versions.
• Process:
  1. Retrieval: Cloned repos or downloaded archives.
  2. Setup: Installed dependencies based on documentation or inferred from code (e.g., requirements.txt, pom.xml).
  3. Execution: Attempted to run the artifact following provided instructions.
  4. Troubleshooting: If execution failed, evaluators debugged issues (fixing paths, dependencies, code) for up to 4 hours per artifact.
  5. Labeling: Artifacts were classified as Executable, Partially Executable, or Not Executable.
• Reproducibility Check: For executable artifacts, outputs were compared against the original paper's results to determine if findings could be reproduced.
• Total Effort: Approximately 650 person-hours were invested in the study.

3. Key Contributions

The paper addresses four specific Research Questions (RQs):

1. RQ1 (Executability & Effort): What is the status of executability, and how much effort/modification is required?
2. RQ2 (Modification Types): What specific types of modifications are needed to make artifacts run?
3. RQ3 (Execution Barriers): What challenges prevent successful execution?
4. RQ4 (Reproducibility): Do executable artifacts actually reproduce the original findings?

Additionally, the authors propose three actionable guidelines to improve future artifact preparation and review.

4. Key Results & Findings

RQ1: Executability and Effort

• Overall Executability: Only 40% (40/100) of the artifacts were fully executable.
  • 24% were partially executable.
  • 36% were not executable.
• Effort Distribution:
  • Among the 40 executable artifacts, only 32.5% ran without modification.
  • 17.5% required low effort; 82.5% required moderate to high effort.
  • Even "executable" artifacts often required significant manual intervention (e.g., fixing environment variables, updating code).
• Trend: There is no consistent improvement in executability over time; 2024 saw only 31.8% full executability, while 2021 peaked at 72.7%.

RQ2: Types of Modifications

Five categories of modifications were identified, with Environment Setup being the most prevalent (57% of artifacts):

1. Environment Setup: Installing missing dependencies (43.86%) and resolving version conflicts (45.61%).
2. Instruction Modification: Fixing syntax, flags, or paths in documentation (e.g., missing --gpus flags).
3. Source Code Modification: Fixing runtime errors due to poor documentation (22% of artifacts).
4. File Organization: Reorganizing folders or creating missing directories.
5. Configuration Adjustment: Updating config files (e.g., config.json) to match the local environment.

RQ3: Execution Barriers

Three main themes caused execution failures:

1. Environment Setup Issues (91.67% of failures):
   • Missing OS/hardware specs.
   • Undocumented dependencies (60%).
   • Missing version information for languages/libraries (54.55%).
   • Impact: Artifacts facing all three issues had an 8% executability rate vs. 78.8% when none were present.
2. Inadequate Documentation (70% of failures):
   • Ambiguous steps (57.1%) or missing instructions (45.2%).
   • Artifacts with no documentation had a 7.1% success rate.
3. Inaccessible/Outdated Components: Missing datasets, scripts, or deprecated configurations.

RQ4: Reproducibility

• Reproducibility Rate: Among the 64 artifacts that were at least partially executable, only 14% were fully reproducible without major effort.
• Specifics:
  • Only 35% of the 40 fully executable artifacts reproduced the original results.
  • 30% produced divergent outputs.
  • 5% produced no output.
• Validation Gap: A major barrier was the lack of validation instructions. Only 6.3% of executable artifacts provided sufficient scripts to verify results against the paper.

5. Significance and Recommendations

The study highlights that sharing artifacts is insufficient for open science; they must be runnable, verifiable, and reproducible. The authors propose three guidelines to close this gap:

1. G1: Provide Comprehensive and Structured Documentation:

   • Must include: Project Metadata, System Specifications (OS, Hardware, Versions), Installation/Environment Setup, Execution Steps, and Validation Steps.
   • Impact: Artifacts with all five components achieved 85.7% executability and 83.3% reproducibility.

2. G2: Avoid Configuration Leakage:

   • Authors must test artifacts in a clean, isolated environment (e.g., fresh Docker container) before submission to ensure they do not rely on implicit local settings, cached tools, or source-built packages not available to others.

3. G3: Combine Containerized Environments with Accessible Source Code:

   • Containers (Docker/VM) ensure portability and reduce setup complexity (67.5% executability for containerized vs. 31% for source-only).
   • Source Code ensures transparency and long-term adaptability.
   • Recommendation: Provide both to balance ease of use with extensibility.

Conclusion

The paper concludes that while the SE community has made progress in sharing artifacts, the usability of these artifacts remains critically low. Without addressing environmental specification, documentation quality, and configuration leakage, the goals of open science—transparency, validation, and reuse—will remain unmet. The authors' guidelines offer a concrete path forward for researchers, conference organizers, and reviewers to improve the rigor and sustainability of SE research.