Evidence Quality in Assurance-Oriented Benchmark Construction

Technical Summary: Evidence Quality in Assurance-Oriented Benchmark Construction

Problem Statement
Assurance-oriented benchmarks constructed from public software incidents (vulnerability advisories, repository issues, incident reports) inherit a fundamental evidence-quality problem. While a patch or corrected state may be visible, the semantic claims required for downstream assurance analysis—such as reconstruction of the affected state, deployment context, replay decisions, oracle feasibility, and release conditions—are often incomplete or ambiguous. Existing work (e.g., Vul4J, ReposVul) has established the necessity of filtering public vulnerabilities for reproducibility and intrinsic data quality (accuracy, consistency, completeness). However, a narrower gap remains: the task-relative evidence quality required to construct a reconstructed system representation for software assurance. Specifically, even when an affected/fixed pair is available, analysts must author source/consequence boundaries, context fields, and enforcement activities that public records do not explicitly state. The evidentiary status of these "authored semantics" is distinct from simple source availability, and public release may be constrained even when technical reconstruction is possible.

Methodology
The study employs an exploratory empirical case-study approach following software-engineering transparency principles. The unit of analysis is an incident-derived affected/fixed case.

1. AIRR-40 Registry Construction: The author constructed a frozen 40-case frame of public tool-using-agent and Model Context Protocol (MCP) incidents. This registry, AIRR-40, contains 49 fields covering public evidence pointers, timestamps, source strata, patch availability, deployment reconstructability, replay-triage status, oracle feasibility, releaseability, and paired-control availability.
2. Qualification Protocol: Cases were screened against a "strict" definition (requiring exact-upstream fidelity and specific release conditions) and a "lenient" definition (coinciding with replay-decidability). Blockers were recorded as non-exclusive categories (e.g., reconstruction, releaseability, pairing).
3. Deep Evidence-Bounded Representation Audit: A subset of 10 cases (8 strict primary, 2 lenient-only stress cases) underwent a granular audit. This involved:
   • Independent reacquisition of affected/fixed artifacts.
   • Construction of a representation $R = (G, \Gamma, P, U)$, where $G$ is the source-to-consequence graph, $\Gamma$ represents analyzer-relative semantics (capabilities, context, bindings, transformations, activity), $P$ maps elements to nominal provenance categories, and $U$ records unresolved alternatives.
   • Classification of 190 representation rows into nominal provenance categories (e.g., Direct Anchor Supported, Synthesis Explicit).
   • Verification that all 52 analyzer-relevant semantic attributes were source-informed rather than independent ground truth.
4. Analyzer Role: Two analyzers (A0 and A1) were used to check path/capability mediation and context binding. The study focuses on how unresolved assumptions (specifically in Case C4) alter static assurance conclusions under different admissible assignments (fail-open vs. fail-closed).

Key Results

• Qualification Attrition: Out of 40 candidates, 39/40 exposed a patch or corrected state. However, only 20/40 met the strict paired criteria. The gap between "patch available" and "strict inclusion" highlights that technical fixes do not guarantee assurance-ready evidence.
• Blocker Decomposition: Among the 15 prefix survivors that failed strict inclusion, the most frequent non-exclusive blockers were:
  • Reconstruction (14 cases)
  • Releaseability (9 cases)
  • Replay/Source Resolution (4 cases)
  • Pairing (3 cases)
  • Oracle Feasibility (1 case)
  • Note: Case C38 was technically reconstructable (exact-upstream) but excluded solely due to Conditional releaseability, demonstrating that release constraints can be independent of technical evidence quality.
• Provenance Audit: In the deep audit of 190 rows, all 52 analyzer-relevant semantic attributes were source-informed rather than independent ground truth. No direct or documented-only consequence boundary existed in the deep subset; every declared consequence sink was inferred.
• Conclusion-Relevant Uncertainty (Case C4): Case C4 demonstrated that an unresolved enforcement-activity field yields different fixed-state conclusions depending on the assignment (fail-open vs. fail-closed). Under one assignment, the result is Witness; under the other, it is No-Witness-under-Abstraction. This proves that unresolved alternatives ($U$) are not merely documentation overhead but can fundamentally alter assurance conclusions.

Key Contributions

1. Empirical Characterization of Evidence Quality: The study quantifies the gap between public disclosure and strict assurance-ready benchmarks (39/40 vs. 20/40) and decomposes the specific blockers (reconstruction, releaseability, etc.) that prevent inclusion.
2. Element-Level Provenance Audit: It introduces a framework for separating analyzer-relative semantics ($\Gamma$), nominal provenance ($P$), and unresolved alternatives ($U$), explicitly marking that source-pair recovery does not automatically provide independent ground truth for semantic fields.
3. Identification of Uncertainty Boundaries: Through Case C4, the study demonstrates that unresolved representation assumptions can change static assurance conclusions, arguing that unresolved alternatives should remain first-class entities rather than being forced into negative findings.
4. AIRR-40 Resource: The release of a reusable, frozen 40-case, 49-field registry with a 20-case exact-upstream strict manifest, public evidence pointers, and reproducibility artifacts.

Significance and Claims
The paper explicitly does not claim to:

• Estimate population readiness or detector accuracy.
• Propose a new ISO/IEC 25012 data quality model or a universal ordinal evidence-strength scale.
• Serve as a training corpus for detectors.
• Claim novelty in the general filtering of public vulnerabilities (acknowledging prior work like Vul4J and Croft et al.).

Instead, the significance lies in characterizing evidence quality for assurance-oriented benchmark construction. The study argues that for tool-using agents and assurance studies, the "fitness" of evidence is determined not just by the existence of a patch, but by the reconstructability of the system representation, the feasibility of release, and the explicit handling of unresolved semantic alternatives. The AIRR-40 resource allows researchers to inspect these task-relative quality conditions, recompute results, and apply alternative admissibility policies without accepting the study's strict criteria as universal rules. The findings serve as a warning that $R \neq S$ (Representation does not equal System) and that assurance questions may be ill-posed if the underlying evidence lacks specific semantic provenance.