Technical Summary: From Verdict to Diagnosis: Attributable Security Review of Pull Requests

1. Problem Definition: The Verdict–Diagnosis (VD) Gap

Current automated code reviewers are primarily evaluated on their ability to issue a "blocking" verdict for malicious Pull Requests (PRs). However, the paper identifies a critical flaw in this evaluation paradigm: a reviewer may correctly block a PR for the wrong reason. A block might be triggered by an unrelated issue (e.g., a formatting error or a non-critical warning) rather than the specific vulnerability that renders the PR unsafe.

This discrepancy is termed the Verdict–Diagnosis (VD) gap.

• Verdict: The decision to approve or block a PR.
• Diagnosis: The specific identification of the vulnerability and the evidence supporting it.
• The Gap: A correct blocking verdict paired with an incorrect or unsubstantiated diagnosis. In such cases, remediation efforts are misdirected, leaving the actual vulnerability unaddressed.

The paper argues that existing benchmarks and evaluation metrics fail to distinguish between a system that merely "blocks" and one that correctly "diagnoses" the underlying security defect. Furthermore, many vulnerabilities (particularly "absence-type" defects where a required guard is missing) require evidence from unchanged parts of the repository, which standard diff-based analysis often misses.

2. Methodology

2.1 MALPR-BENCH: A Mechanism-Grounded Benchmark

To measure the VD gap, the authors introduce MALPR-BENCH, a benchmark designed to evaluate three distinct dimensions separately:

1. Verdict Correctness (V): Did the system block the PR?
2. Target Vulnerability Identification (I): Did the system correctly identify the specific vulnerability mechanism?
3. Evidence Validation (E): Did the system ground its diagnosis in concrete, auditable repository facts (code locations, unchanged files, etc.)?

Construction:

• Scale: 89 malicious PRs and 50 benign controls across 44 repositories and eight language families.
• Sources:
  • Mined History: Recovering incomplete fixes from project histories.
  • Advisory-Derived: Constructing malicious states from public security advisories (Pool A: incomplete fix; Pool B: reversed enforcement).
  • Real-World Discovery: Previously undisclosed vulnerabilities found by the authors' tool.
• Ground Truth: Each case includes a "frozen rubric" specifying the target vulnerability, the required evidence chain, and accepted descriptions. This allows for precise grading of whether a review is "attributable" (i.e., $A = V \land I \land E$).
• Defect Classification: Cases are categorized as Present-type (unsafe behavior is visible in the diff) or Absence-type (required security enforcement is missing). Evidence locations are classified from L0 (diff only) to L2b (semantic correspondence in unrelated files).

2.2 PRGUARD: An Attributable Security Reviewer

To address the VD gap, the authors propose PRGUARD, a system that separates vulnerability identification from evidence validation. Unlike end-to-end models that jump from diff to verdict, PRGUARD operates through a staged pipeline:

• Stage 0 (Structural Collection): Deterministically collects structural context (callers, callees, imports) around the changed code before any model reasoning occurs.
• Stage 1 (Change Characterization): The model describes the security-relevant behavior of the change without proposing a specific vulnerability yet.
• Stage 2 & 2.5 (Evidence Acquisition):
  • Path 1 (Knowledge-Directed): Uses a Mechanism Knowledge Base (KB) derived from development cases to retrieve specific repository evidence via typed relations (e.g., SIBLING-ENDPOINT).
  • Path 2 (Code-Directed): Builds a worklist of repository paths based on the changed code's structure, independent of the KB.
• Stage 3 (Candidate Construction): Formulates concrete candidate vulnerabilities based on the collected evidence.
• Stage 4 (Evidence Validation): A separate model invocation tests the candidates against the repository evidence. It verifies security-critical premises (attacker control, reachability, missing guards). Candidates are marked as VALIDATED, DOWNGRADED, or REJECTED.
• Stage 5 (Review Synthesis): A deterministic policy maps validation outcomes to a verdict (Block, Comment, Approve) and synthesizes a review that explains the validated findings with specific code locations.

3. Key Contributions

1. Formulation of the VD Gap: The paper defines and characterizes the discrepancy between a correct blocking verdict and a correct diagnosis, arguing that current evaluation metrics obscure this failure mode.
2. MALPR-BENCH: A systematic evaluation framework and benchmark that separates verdict correctness from vulnerability identification and evidence validation, using pre-committed rubrics for ground truth.
3. PRGUARD: An attributable PR security reviewer architecture that decouples hypothesis generation from evidence validation and retrieves context beyond the diff.
4. Empirical Validation: Demonstration that separating identification from validation improves the attribution of security findings, particularly for absence-type defects.

4. Results

4.1 Performance on the Common-Coverage Challenge Set

Evaluated on 31 held-out malicious PRs (19 self-generalization + 12 discovery cases) against CodeRabbit (a widely deployed commercial AI reviewer):

• Blocking Performance: Both systems achieved similar blocking rates (CodeRabbit: 24/31; PRGUARD/DeepSeek: 22/31).
• Vulnerability Identification (I): PRGUARD/DeepSeek identified 1.38× more target vulnerabilities than CodeRabbit (22 vs. 16).
• Absence-Type Defects: On 14 cases where a required guard was missing, both systems blocked 9 PRs. However, PRGUARD/DeepSeek identified the target vulnerability in 9/14 cases, while CodeRabbit identified it in only 3/14 (a 3× difference).
• Attributable Blocks (A): PRGUARD/DeepSeek achieved 19/31 attributable blocks, compared to 16/31 for CodeRabbit.
• Evidence Location: CodeRabbit failed to identify targets in 0/7 cases requiring evidence outside the touched files (L2a/L2b), whereas PRGUARD succeeded in most.

4.2 Full Pipeline Evaluation

On 63 held-out malicious cases (excluding the discovery tier to avoid bias):

• Pool B (Reversed Enforcement): Both backends (GPT-5.5 and DeepSeek) identified all 37 target vulnerabilities (I=37/37). However, evidence validation (E) varied (26/37 for GPT-5.5, 34/37 for DeepSeek), highlighting that identification does not guarantee valid evidence grounding.
• Benign Controls: PRGUARD showed low false-positive rates (4–5 blocks on 50 benign controls), comparable to CodeRabbit (0 blocks on a subset of 6 controls).

4.3 Real-World Discovery

Applied to production repositories, PRGUARD uncovered 12 previously undisclosed, proof-of-concept-backed vulnerabilities across five widely used projects.

• Independent runs of PRGUARD and CodeRabbit both blocked 10/12 PRs in this discovery tier.
• However, PRGUARD produced 10/12 attributable blocks, while CodeRabbit produced only 4/12, demonstrating that identical verdict totals can mask a 2.5× difference in diagnostic quality.

5. Significance and Claims

The paper claims that the Verdict–Diagnosis gap is a fundamental limitation in current automated security review. A "successful" block is insufficient if it does not correctly identify and substantiate the vulnerability, as this leads to ineffective remediation.

• Attributability is Key: The authors argue that security reviews must be attributable—the verdict must be grounded in specific repository evidence that validates the identified mechanism.
• Separation of Concerns: The results suggest that separating the tasks of identifying a candidate vulnerability and validating it against evidence improves the reliability of the diagnosis, particularly for complex defects requiring cross-file context.
• Limitations: The paper acknowledges that PRGUARD is not a silver bullet. It treats prompt injection and judgment errors as residual attack surfaces. The discovery of real-world vulnerabilities demonstrates capability but does not claim to estimate the recall rate on arbitrary PRs, as the candidate stream was filtered for manual validation.

In summary, the work shifts the evaluation focus from "did it block?" to "did it block for the right reason, with proof?", introducing a methodology and toolset to measure and mitigate the risks of misdiagnosed security reviews.