Technical Summary: ARVO – Atlas of Reproducible Vulnerabilities for Open-Source Software

1. Problem Statement

Achieving a balance between reproducibility, quantity, and diversity in vulnerability datasets has historically been viewed as a three-way trade-off, with reproducibility often being the most neglected dimension. Existing large-scale datasets, such as Google's OSS-Fuzz and its derivative OSS-Fuzz-OSV, suffer from significant reproducibility failures. In comparative evaluations, 63% of historical vulnerable versions failed to build due to dependency drift, build system bit-rot, and missing resources. Consequently, recent techniques based on these sources can reproduce only ~37% of vulnerabilities.

Furthermore, existing datasets lack critical metadata for security research, such as triggering inputs, reproducible environments, and ground-truth remediation data. This hampers the evaluation of dynamic analysis techniques, automated patch generation, and the verification of candidate patches. Manual efforts to reproduce vulnerabilities (e.g., Mu et al.) have achieved limited scale (368 vulnerabilities) due to the prohibitive cost of human labor, while synthetic datasets often lack real-world overlap.

2. Methodology: The ARVO Framework

ARVO (Atlas of Reproducible Vulnerabilities in Open-source software) is a framework designed to generate a large-scale, reproducible, and diverse vulnerability dataset by addressing the specific obstacles to rebuilding historical software versions. The system ingests metadata from upstream sources (primarily OSS-Fuzz) and augments it with source code, build steps, and binaries.

Core Components

1. The Reproducer:

   • Goal: Rebuild historical vulnerable and fixed versions from source and verify that the provided Proof of Concept (PoC) triggers the intended crash in the vulnerable version but not in the fixed version.
   • Strategies for Reproducibility:
     • Minimally Intrusive Build Instrumentation: Unlike OSS-Fuzz, which swaps source code outside the container after the build environment is prepared (ignoring resource initialization), ARVO instruments the build process to separate resource-download actions from compilation. It hooks only resource-fetching commands, preserving the natural build flow.
     • Comprehensive Revision Control: ARVO enforces precise version control not just for the main project but for all dependencies. It uses commit timestamps to select the most recent compatible dependency versions available at the time of each main component commit, ensuring the build environment remains consistent across the bisection process.
     • Broken Resource Fixing: ARVO maintains a detection-fixing loop for missing resources (e.g., URLs that have migrated from SVN to Git). It categorizes resources into "core" (required for compilation) and "non-core" (e.g., fuzzing corpora). Non-core missing resources are skipped or modified to prevent build crashes, while core resources are manually fixed (requiring only a few hours of maintenance per season) and stored as reusable rules.

2. The Vulnerability Patch Locator:

   • Goal: Automatically identify the precise developer commit that fixes a vulnerability.
   • Mechanism: ARVO employs commit bisection across the range of commits between the vulnerability discovery and verification.
   • Dependency Handling: During bisection, if a chosen commit fails to build due to dependency mismatches, ARVO steps to a neighboring commit to continue. It relies on commit timestamps to dynamically select compatible dependency versions for each step, allowing it to bisect through thousands of commits even in complex, multi-component projects.

3. Dataset Construction:

   • ARVO outputs a dataset where each vulnerability is packaged with its build environment as a Docker image.
   • It supports backporting: The system can revert fixing commits to reintroduce vulnerabilities into older or different versions of software, provided the patch applies cleanly.

3. Key Contributions

1. A New Dataset Paradigm: ARVO introduces reproducibility to security datasets at scale, combining quantity, diversity, and the ability to rebuild and trigger vulnerabilities from source.
2. Solutions to Reproduction Challenges: The paper identifies and solves key challenges in historical software reproduction, specifically incompatible dependencies, missing resources, and fragile build processes, through a scalable, automated design.
3. The ARVO Dataset: A comprehensive dataset of 6,138 real-world vulnerabilities across 311 open-source C/C++ projects. It includes:
   • Interactive Docker images for both vulnerable and fixed versions.
   • Precise patch commits identified via automated bisection.
   • Triggering PoC inputs for each vulnerability.
4. Open Source Infrastructure: The authors have open-sourced the framework, evaluation infrastructure, and 12,276 Docker images (vul and fix versions for each reproduction) to enable downstream research and re-compilation with arbitrary modifications.

4. Results and Evaluation

• Reproducibility Success: ARVO successfully reproduces 81% of attempted vulnerabilities (6,138 out of 8,921 initial candidates). This represents a 44 percentage-point improvement over the state-of-the-art (OSS-Fuzz-OSV, which achieves ~37%).
• Patch Accuracy: In a manual evaluation of 100 random cases across different agreement groups, ARVO achieved an overall 89.4% accuracy in locating the correct fixing patch. This outperforms OSS-Fuzz-OSV's 82.9% accuracy.
• Complexity Handling: ARVO effectively handles complex, multi-component projects where OSS-Fuzz fails. In a comparison of 100 random issues, ARVO reproduced 81% of cases, while OSS-Fuzz reproduced only 37%, with OSS-Fuzz failing almost entirely on projects with more than one component.
• Upstream Impact: By cross-comparing with upstream data, ARVO identified over 300 cases where vulnerabilities were incorrectly reported as fixed (remaining unpatched) and 1,518 cases of incorrect vulnerability data (false positives) in the upstream OSS-Fuzz dataset.

5. Significance and Claims

The paper claims that ARVO transforms static vulnerability reports into interactive, verifiable artifacts. Its primary significance lies in:

• Enabling Downstream Research: By providing reproducible environments, ARVO supports the evaluation of automated patch generation, program repair, and dynamic analysis techniques that were previously hindered by the lack of ground-truth data.
• Improving Upstream Quality: The framework acts as a feedback loop, identifying and correcting false positives and mislabeled patches in upstream datasets like OSS-Fuzz.
• Scalability and Automation: Unlike manually crafted datasets (e.g., Magma) which are limited in size and diversity, ARVO demonstrates that high-quality reproducibility can be achieved automatically across hundreds of diverse projects.

The authors note that while ARVO currently focuses on C/C++ projects manifesting as crashes on PoC inputs, the methodology is general and can extend to other languages and bug types. They also acknowledge limitations, such as the resource intensity of rebuilding datasets and the exclusion of race conditions that are difficult to trigger deterministically. The ultimate goal is for upstream sources to natively adopt reproducibility, at which point ARVO's role as a separate reconstruction layer may become unnecessary, though the methodology remains a critical contribution to the field.