Technical Summary: NICE – A Framework for Declarative and Machine-Checkable Vulnerability Reproduction

1. Problem Statement

Reproducing software vulnerabilities is a fundamental activity for security researchers, maintainers, and educators, yet it remains fragile, time-consuming, and often infeasible in practice. Current methods suffer from several critical issues:

• Fragility and Obsolescence: Vulnerability reports often lack precise build configurations, dependency lists, and execution environment details. As software ecosystems evolve, reproducing historical environments becomes increasingly difficult.
• Manual Effort: Empirical studies indicate that despite significant manual effort, a large percentage of vulnerabilities cannot be successfully reproduced due to missing or ambiguous information.
• Lack of Verification: Existing tools often rely on static descriptions or manual validation, making it difficult to independently verify exploitability claims or ensure that a reported vulnerability is truly present and exploitable.
• Limitations of Existing Tools: Current solutions often rely on containerization (which fails for kernel vulnerabilities), specific software ecosystems (e.g., only WordPress), or manual maintenance of base images, leading to a lack of long-term reproducibility and versatility.

2. Methodology: The NICE Framework

The authors introduce NICE (NIx CvE reproduction framework), a system that leverages Functional Package Management (FPM), specifically Nix and NixOS, to create declarative, executable, and machine-checkable vulnerability reproductions.

Core Components

A NICE vulnerability "recipe" consists of two primary declarative components:

1. Virtual Machine (VM) Configurations:

   • Defined using NixOS configuration files, these specify the exact software environment required for the vulnerability (kernel version, installed packages, system services, network settings, user accounts).
   • Nix's pinning mechanism allows authors to lock the environment to specific historical revisions of the nixpkgs repository, ensuring that the environment can be rebuilt identically years later, regardless of ecosystem drift.
   • Recipes can define single VMs or clusters (e.g., client/server/attacker setups) to support complex attack vectors.

2. Exploitation Scenario (Test Script):

   • Written in Python using the NixOS test framework API, this component orchestrates the execution of the VMs.
   • It automates the exploitation process: booting systems, waiting for services, executing exploit scripts (PoCs), and interacting with the system (including GUI interactions via optical character recognition).
   • Assertion Blocks: The framework uses reusable, machine-checkable assertions to validate the outcome. Instead of relying on human observation, the test explicitly checks for unauthorized system states (e.g., checking if a file exists, if a process has root privileges, or if specific log messages appear).

Workflow

The NICE workflow follows three stages:

1. Report Analysis: Extracting affected software versions, attack vectors, and preconditions from public advisories.
2. Recipe Specification: Defining the VM configurations and selecting a pinning strategy (Snapshot pinning for full historical fidelity or Component pinning for modern test framework compatibility).
3. Exploit Validation: Executing the test script. The framework boots the VM cluster, runs the exploit, and evaluates assertions. The result is a binary pass/fail code, providing factual evidence of exploitability.

Build Outputs

NICE produces two types of artifacts:

• Debuggable Environment: A running VM cluster for interactive analysis and teaching.
• Executable Proof: A self-contained, reproducible build that can be run as a single command to verify the vulnerability without manual intervention.

3. Key Contributions

• Declarative Vulnerability Reproduction: NICE is the first framework to combine functional package management with automated vulnerability validation, allowing vulnerability reports to be expressed as executable, reviewable code.
• Machine-Checkable Evidence: By replacing manual verification with automated assertions, NICE provides concrete, reproducible proof of exploitability, reducing ambiguity in vulnerability reporting and triage.
• Long-Term Reproducibility: Through Nix's pinning capabilities, the framework ensures that vulnerability environments remain verifiable over time, addressing the "bit-rot" problem common in security research.
• Versatility: The framework is designed to handle diverse vulnerability classes (CWE categories), attack vectors (remote, local, GUI, network), and target types (userland, system software, kernel, graphical applications).

4. Experimental Results

The authors evaluated NICE on a corpus of 19 real-world CVEs spanning from 2014 to 2025.

• Coverage: The selected CVEs covered four target categories: Headless applications (13), System software (3), Kernel (2), and Graphical applications (1). They represented various CWE pillars, including improper access control, memory corruption, and resource exhaustion.
• Performance:
  • All 19 vulnerabilities were successfully reproduced and validated.
  • Test runtimes ranged from approximately 1 minute to 12 minutes (with a hot Nix cache).
  • The code size (Lines of Code) for recipes and tests was generally concise (e.g., Heartbleed required 76 lines for VM config and 63 for tests).
• Limitations Observed:
  • Exploit Availability: The framework relies on the existence of reliable Proof-of-Concept (PoC) exploits. Cases where PoCs were platform-specific or required complex binary manipulation (e.g., XZ backdoor) were difficult to automate.
  • Temporal Boundaries: Very old vulnerabilities (pre-2013) posed challenges due to the historical separation of NixOS and Nixpkgs repositories and the evolution of the NixOS test framework itself.
  • Scope: The framework is limited to software vulnerabilities in Linux/NixOS contexts and does not address hardware vulnerabilities or those requiring physical interaction.

5. Significance and Claims

The paper claims that NICE fundamentally shifts vulnerability reproduction from a fragile, manual process to a robust, automated, and verifiable engineering practice.

• For Security Reporting: NICE enables reporters to submit "executable specifications" alongside vulnerability reports, allowing maintainers to verify exploits with a single command, thereby increasing trust and reducing triage time.
• For Education and Training: It allows for the creation of "cyber ranges" and security labs that are lightweight, shareable, and guaranteed to work over time, solving the maintenance burden of traditional VM images.
• For Research: It provides a standardized, machine-readable corpus of vulnerabilities that can be used for benchmarking security tools and training AI models, addressing the current lack of high-quality, reproducible datasets.

The authors emphasize that while NICE does not automate the generation of exploits (it relies on existing PoCs or manual script writing), it successfully automates the validation and reproduction of the environment, ensuring that the conditions for exploitation are met and the results are objectively verifiable.