Technical Summary: ReproScore: Separating Readiness from Outcome in Research Software Reproducibility Assessment

1. Problem Statement

Digital libraries (e.g., Zenodo, Software Heritage) curate millions of research software artifacts but lack scalable infrastructure to assess whether these artifacts are actually executable. Existing automated assessment tools suffer from a critical flaw: they conflate readiness (static repository completeness, such as the presence of a README or requirements.txt) with outcome (actual execution success).

This "readiness–outcome conflation" leads to a false assumption that a repository with complete static documentation is likely to run. In reality, a repository may possess perfect static signals yet fail due to transitive dependency conflicts, inaccessible data, or runtime environment issues that static analysis cannot detect. Current tools either rely on static analysis (ignoring execution) or require expensive, expert-driven sandboxed execution (lacking scalability). There is a need for a framework that explicitly distinguishes between the potential for reproducibility (readiness) and the actual result of execution (outcome), providing curators with diagnostic information rather than a binary pass/fail prediction.

2. Methodology: The ReproScore Architecture

ReproScore is a two-tier framework designed to separate reproducibility readiness from execution outcome, combining them into a coverage-adaptive composite score.

Tier 1: Reproducibility Readiness Score (RRS)

RRS is a static, always-available metric computed via deterministic, rule-based analysis of a locally cloned repository. It does not require code execution.

• Structure: RRS comprises 26 atomic sub-metrics grouped into five categories:
  1. Environment (E, weight 0.30): Specification of dependencies, containerization, and runtime versions.
  2. Data (A, weight 0.25): Accessibility, pointers, and acquisition methods for input data.
  3. Documentation (D, weight 0.20): Presence of execution instructions, entry points, and usage examples.
  4. Code Portability (C, weight 0.15): Absence of absolute paths, hardcoded credentials, and unresolvable imports.
  5. Reproducibility Signals (S, weight 0.10): Determinism signals like seed management and execution order.
• Scoring Logic:
  • Gate Function: A power-law gate function $g(x, \tau, k)$ penalizes sub-threshold scores non-linearly. Scores below a threshold $\tau$ are compressed, reflecting a curation policy that severe deficiencies (e.g., missing environment spec) are disproportionately damaging.
  • Penalties: Hard penalties are applied for the complete absence of environment specifications or data artifacts ($P_{hard}$), and for stochastic operations without seed management ($P_{seed}$).
  • Formula: $RRS = 100 \cdot \sum (w_i \cdot g(x_i)) - \text{penalties}$.

Tier 2: Reproducibility Outcome Score (ROS)

ROS is an optional, execution-based metric computed when a sandboxed environment (e.g., repo2docker, nbconvert) is available.

• Components: It aggregates up to six probes: Install success ($I$), Execution success ($X$), Output determinism ($\Delta$), Notebook execution rate ($N$), Import success rate ($E'$), and Test pass rate ($T$).
• Normalization: ROS is a weighted sum of available components, allowing the score to be computed even if only partial execution data is available.

Composite Score (RCS)

The ReproScore Composite (RCS) blends RRS and ROS based on the amount of execution evidence collected.

• Coverage Weight ($\alpha$): A parameter that scales from 0 (no execution) to a maximum of 0.70. It ensures that static readiness is never entirely dominated by execution outcomes, acknowledging that a successful run does not validate documentation or portability.
• Formula: $RCS = (1 - \alpha) \cdot RRS + \alpha \cdot ROS$. If no execution data exists, $RCS = RRS$.

Community Rubric Mechanism

ReproScore externalizes assessment priorities via a versioned YAML profile. This allows different communities (e.g., bioinformatics vs. general software) to override default weights and gate thresholds without modifying the source code. This makes curation policies explicit, auditable, and portable across institutions.

3. Key Contributions

1. Conceptual Separation: A formal two-tier architecture (RRS/ROS/RCS) that decouples static readiness from execution outcome, addressing the readiness–outcome conflation in digital library workflows.
2. Readiness Model: A 5-category, 26-sub-metric model (RRS) with expert-informed weights, non-linear gating, and file-level evidence provenance.
3. Governance Mechanism: A community rubric system that allows assessment priorities to be defined as citable, versioned artifacts.
4. Empirical Validation: An evaluation on 423 GitHub repositories (Python/Jupyter) demonstrating that static readiness strongly discriminates between failure modes but exhibits near-zero correlation with binary execution success.

4. Evaluation Results

The framework was evaluated on a stratified sample of 423 repositories from a ground-truth corpus, categorized into five failure modes: Success, Install Dependency Errors, Missing Modules, Missing Data, and Code Errors.

• Failure Mode Discrimination: The Environment (E) category showed the strongest ability to discriminate between failure modes (Kruskal-Wallis $H = 96.89, p < 0.001$).
  • The Detection Paradox: Repositories with install dependency errors (install_dep) scored highest on Environment (mean 22.8) because they explicitly specified environments, yet they failed due to version conflicts. Conversely, successful repositories scored lower (mean 10.3) because they often lacked explicit specifications. This confirms that RRS characterizes what kind of problem exists, not if it will succeed.
• Binary Correlation: The correlation between RRS and binary execution success was near-zero ($r_{pb} = -0.014$). This empirically quantifies the readiness–outcome gap: a high static score does not predict a successful run.
• Robustness:
  • Weight Stability: Repository rankings remained stable ($\tau \ge 0.911$) even when weights were perturbed by $\pm 50\%$.
  • Gate Insensitivity: The gate function had negligible impact on binary prediction (AUC range $\approx 0.008$), confirming its role as a normative policy tool rather than a predictive mechanism.
  • Leave-One-Category-Out: Removing the Environment or Data categories actually improved binary AUC, further proving that these categories are designed to characterize failure types, not predict binary success.
• Composite Validation: In partial ROS validation, the RCS correctly re-ranked repositories. For instance, install_dep repositories had a moderate RRS (26.8) but a near-zero ROS (13.5), resulting in a low RCS (19.8), accurately reflecting their execution failure despite static completeness.

5. Significance and Claims

The paper positions ReproScore as a diagnostic instrument for digital library curators rather than a binary predictor of success.

• Architectural Necessity: The study validates that separating readiness from outcome is both necessary and non-trivial. Static signals capture meaningful structural differences (e.g., distinguishing a missing module from a version conflict) that binary success metrics obscure.
• Curatorial Utility: By providing a category-level profile, ReproScore enables targeted remediation. A low Environment score suggests dependency pinning is needed; a low Data score suggests missing pointers. This is more actionable than a single "reproducibility" score.
• Scalability: The static-first approach (RRS) allows libraries to triage millions of deposits without the computational cost of sandboxed execution, reserving execution resources for artifacts that show promise or require specific validation.
• Policy Transparency: The community rubric mechanism transforms scoring heuristics from opaque code into auditable, citable policies, facilitating cross-institutional comparison and domain-specific adaptation (e.g., prioritizing data accessibility in bioinformatics).

The authors conclude that a framework which explicitly names the gap between readiness and outcome is more informative for curation workflows than one that collapses them. ReproScore provides the infrastructure to manage this gap, supporting the OAIS Ingest and Archival Storage entities with computable, scalable metrics.