Technical Summary: AI Slop and Hallucinations in Vulnerability Assessment

1. Problem Statement

The integration of Large Language Models (LLMs) into cybersecurity vulnerability assessment has introduced a "trustworthiness crisis" driven by the proliferation of "AI slop." Unlike traditional false positives arising from conservative rule-based Static Application Security Testing (SAST), AI slop consists of security artifacts (reports, proofs-of-concept, patches) that exhibit high linguistic fluency and structural plausibility but lack semantic validity or executable grounding.

These artifacts include:

• Hallucinated vulnerabilities: Fabricated Common Vulnerabilities and Exposures (CVEs), nonexistent APIs, and disjointed execution paths.
• Incorrect patch synthesis: Fixes that address surface symptoms while ignoring root causes, or patches that introduce new regressions.
• Semantic repackaging: The transformation of existing knowledge or "silent" (undisclosed) fixes into seemingly novel, authoritative reports.

This phenomenon creates a cognitive burden on human triage pipelines that mirrors a denial-of-service (DoS) attack. The core issue is a fundamental reasoning gap: human security experts rely on causal, multi-step deductive reasoning grounded in system constraints, whereas autoregressive LLMs operate via probabilistic token generation based on statistical correlations. When information exceeds an LLM's reasoning horizon, it defaults to statistically plausible continuations rather than verified facts.

2. Methodology

The paper employs a systematic literature review and conceptual analysis to survey empirical evidence and formalize the problem.

• Literature Search: The authors queried IEEE Xplore, ACM Digital Library, and arXiv using LLM and security-related keywords. Two independent reviewers screened results, retaining studies with empirical evidence of LLM failure modes in security contexts.
• Taxonomy Construction: Failures were categorized into three primary branches based on the nature of the reasoning failure.
• Gap Analysis: The paper contrasts human cognitive models (deductive, invariant-preserving) with LLM architectural constraints (probabilistic, context-window limited).
• Metric Proposal: The authors operationalize the reasoning gap through a measurable proxy, the Deductive Coverage Score (DCS), and propose new evaluation instruments (CVE-Bench, Slop-Score).
• Architectural Review: Existing mitigation strategies (passive detection, watermarking) are reviewed and critiqued, followed by a proposal for active neuro-symbolic verification architectures.

3. Key Contributions

A. Formalization and Taxonomy of AI Slop

The paper defines AI slop in cybersecurity and establishes a taxonomy comprising three distinct failure modes:

1. Hallucinated Vulnerabilities: The model invents flaws (e.g., fabricated CVEs, nonexistent APIs) to satisfy prompt expectations. This includes sycophantic hallucination (finding bugs in safe code) and token-bias confounding (judgment shifting based on superficial syntax changes).
2. Incorrect Patch Synthesis: Models generate patches that compile and pass superficial checks but fail to resolve the underlying issue or introduce new security regressions (e.g., trading one CWE for another).
3. Semantic Repackaging: Models reshape existing information (including silent fixes from open-source commits) into "novel" reports. This exploits the lack of public ground truth for undisclosed fixes, making the repackaged claims undetectable by lookup-based validation.

B. Analysis of the Reasoning Gap

The paper identifies the root cause as the divergence between human deductive reasoning and LLM probabilistic generation:

• Human Experts: Build mental models, trace data flow across procedural boundaries, and verify constraints. They treat findings with skepticism until proven.
• LLMs: Estimate token probabilities based on training distributions. They stitch together familiar patterns (API calls, variable names) without verifying a single execution path.
• Limitations of Current Mitigations: The authors argue that Chain-of-Thought (CoT) prompting and tool-using agents narrow but do not close this gap. CoT often results in longer but equally incorrect reasoning chains, and tool use often leads to "tool-use confabulation" where agents misinterpret tool outputs.

C. Critique of Passive Detection and Watermarking

The paper argues that current mitigation strategies targeting provenance (determining if text was written by a machine) are misaligned with the need for correctness.

• Statistical Detection: Fails because security reports are inherently formulaic (low lexical diversity), causing high false positives for human-written reports.
• Watermarking: Incompatible with security logic; forcing watermarked token selection in constrained code domains often breaks the logic or creates invalid payloads.

D. Proposed Solutions and Evaluation Instruments

• Neuro-Symbolic Verification: The authors advocate for an architecture where LLMs act as hypothesis generators, and their outputs are subjected to deterministic verification (Static Analysis, Dynamic Fuzzing, Symbolic Execution) before reaching human analysts.
• Deductive Coverage Score (DCS): A metric quantifying the degree to which a vulnerability claim is grounded in explicit, verifiable evidence (e.g., specific code references, execution traces) versus statistical interpolation.
• CVE-Bench: A benchmark designed to test triage systems' ability to separate grounded reports from fluent, phantom exploit paths using execution-validated ground truth.
• Slop-Score: A continuous metric quantifying the gap between linguistic fluency and verifiable substance, penalizing artifacts that are fluent but lack evidence density or constraint satisfaction.

4. Results and Findings

• Empirical Evidence: The survey highlights that roughly 20% of reports in some bug bounty programs (e.g., HackerOne by mid-2025) were identified as low-quality AI slop, leading to the discontinuation of certain programs (e.g., cURL's bounty program in 2026) due to the inability to distinguish valid reports from AI-generated noise.
• Grounding Deficit: A review of representative works (Table 1) reveals that no study provides execution-level validation of LLM outputs. The strongest grounding found is compile checking or static analysis, which operate far below the requirements of rigorous security reasoning.
• Failure of Partial Bridges: While tools like ReAct and Retrieval-Augmented Generation (RAG) reduce factual hallucinations (e.g., inventing non-existent CVEs), they fail to address logical hallucinations (drawing causally incorrect inferences from accurate facts).

5. Significance and Claims

The paper positions itself as a survey, conceptual analysis, and roadmap. Its primary significance lies in shifting the evaluation paradigm from linguistic fluency to mathematical verifiability.

• Structural Failure: The authors assert that AI slop is not merely a performance issue but a structural failure where probabilistic generation substitutes for causal verification.
• Trustworthiness: Trust in AI-driven triage cannot be achieved through better prompting or provenance tracking alone. It requires active neuro-symbolic verification that maps each pipeline component to prior systems with documented limits (e.g., the path explosion limits of symbolic execution).
• Future Direction: The paper concludes that trustworthy AI in cybersecurity will emerge from systems that treat generation as the starting point of a verification pipeline, ensuring that human judgment is exercised only on claims that have survived deterministic tests. The proposed metrics (DCS, Slop-Score) and benchmarks (CVE-Bench) are intended to provide the necessary infrastructure to engineer this structural failure out of the system.