Technical Summary: CODE-AUGUR: Agentic Vulnerability Detection via Specification Inference

1. Problem Statement

The emergence of autonomous Large Language Model (LLM) agents for vulnerability detection has revealed critical, long-dormant bugs in foundational software. However, the reasoning behind these discoveries remains opaque. Current agentic systems operate by implicitly inferring security assumptions and reasoning over code paths to flag suspects. This approach suffers from two primary limitations:

1. Opacity: The agent's reasoning is an ambiguous natural-language trajectory that is difficult to understand, validate, or reuse in future audits.
2. Unvalidated Assumptions: Agents often deem code secure based on intuitive, path-specific reasoning that fails to account for the combinatorial space of all possible inputs. Crucially, while agents validate suspected bugs via dynamic execution, they rarely validate the specifications (assumptions) that led them to deem other code secure. This leads to false negatives where vulnerabilities are missed because the underlying security assumptions were incorrect.

Existing approaches either rely on purely static agentic reasoning (prone to hallucinations and missed corner cases) or brute-force fuzzing (which lacks semantic understanding and struggles to reach deep program states). There is a need for a system that makes an agent's tacit security judgments explicit, falsifiable, and continuously aligned with the program's actual behavior.

2. Methodology: CODE-AUGUR

The authors propose a security-specification-first paradigm realized in CODE-AUGUR, a novel harness that bridges agentic semantic analysis and guided fuzzing. The system operates on a "reason-falsify-refine" loop.

Core Workflow

1. Threat Model Construction: CODE-AUGUR first distills a threat model from the codebase, documentation, and build context. This defines the attack surface, trust boundaries, and security-relevant state variables.
2. Invariant Analysis (Static Reasoning):
   • The agent analyzes code components against the threat model.
   • If a component is deemed insecure, it is flagged as a bug candidate.
   • If a component is deemed secure, the agent explicitly commits its underlying assumptions as local invariants (security specifications) in the form of in-source assertions (e.g., assert(fmt.channels == cs.channels)).
   • These invariants serve as "reasoning anchors" and "directional landmarks" for subsequent exploration.
3. Invariant Falsification (Dynamic Validation):
   • A Guided Fuzzer (using grey-box fuzzing tools like libFuzzer or Jazzer) is tasked with falsifying the committed invariants.
   • The fuzzer is steered not just by code coverage, but by the distance to violating the specific invariant. This makes the search more tractable by targeting shallow semantic inconsistencies rather than deep crash symptoms.
   • Outcomes of Falsification:
     • Genuine Vulnerability: If the fuzzer triggers an invariant violation that corresponds to a real security defect, it is reported as a bug.
     • Flawed Specification: If the violation is benign (the code is actually secure, but the agent's assumption was too strict or incorrect), the agent refines the invariant.
4. Bug Validation: All candidates are validated by constructing a Proof-of-Vulnerability (PoV) that reaches the defect via a trusted entry point, ensuring the bug is reachable by an attacker.

Key Technical Mechanisms

• Executable Specifications: Invariants are written directly into the source code in the project's native language (e.g., C assertions, Java checks), making them durable artifacts.
• Feedback Steering: The fuzzer receives feedback on how close an execution is to violating an invariant (e.g., via bucketed distance functions), guiding it toward inconsistent states that static reasoning missed.
• Iterative Refinement: When a "benign" violation occurs, the agent reassesses the code region and refines the specification before the fuzzer attempts falsification again.

3. Key Contributions

1. Security-Specification-First Paradigm: A novel approach that externalizes an agent's tacit security judgments into explicit, falsifiable, and executable specifications, aligning the agent's view of code intent with actual behavior.
2. CODE-AUGUR System: A realization of this paradigm that combines LLM reasoning with guided fuzzing. It commits assumptions as in-source invariants and uses a fuzzer to continuously falsify them, forming a closed loop of reason-falsify-refine.
3. Empirical Effectiveness: Evaluation demonstrates that CODE-AUGUR significantly outperforms state-of-the-art agentic systems (CLAUDE CODE and the AIxCC-winning ATLANTIS) in bug discovery.
4. Real-World Impact: The system discovered 22 previously unknown vulnerabilities in widely-used open-source projects (16 confirmed/fixed, several with bug bounties), including in projects under continuous fuzzing (e.g., gpsd, zlib).
5. Durable Artifacts: The inferred invariants serve as reusable security specifications that persist beyond a single audit, helping to identify incomplete fixes and bug families in subsequent development cycles.

4. Evaluation Results

The authors evaluated CODE-AUGUR on two benchmarks: the AIxCC (DARPA AI Cyber Challenge) and the OSV (Open Source Vulnerability) database.

• Bug Discovery Rate:
  • On the AIxCC benchmark, CODE-AUGUR found 34%–63% more bugs than baselines (CLAUDE CODE and ATLANTIS) across different backing LLMs (Claude Sonnet 4.6 and DeepSeek V4 Pro).
  • On the OSV benchmark, it found 61%–370% more bugs than baselines.
  • Crucially, CODE-AUGUR found new bugs across a broader range of subjects (16/23 challenge groups) compared to baselines.
• Component Contribution: Analysis showed that invariant falsification was the primary discovery lane, accounting for 36–41% of vulnerabilities found with the Claude model. This highlights the value of the reason-falsify-refine loop over pure static analysis or blind fuzzing.
• Cost Efficiency: CODE-AUGUR using the open-weight DeepSeek model outperformed baselines using the proprietary Claude model in total vulnerability count, while operating at a fraction of the cost.
• Case Study (gpsd): The system identified a bug family in gpsd where a single invariant (satellites_visible <= MAXCHANNELS) exposed an incomplete fix, tied together multiple related bugs across different drivers, and eventually revealed that the capacity bound itself was too low. This demonstrated the utility of invariants as long-term regression checks.

5. Significance and Claims

The paper claims that CODE-AUGUR represents a shift from "finding bugs" to "generalizing knowledge." By making security specifications explicit and checkable, the system:

• Reduces Trust Barriers: It provides a transparent, auditable record of why code was deemed secure, addressing the "black box" nature of agentic analysis.
• Bridges Paradigms: It effectively combines the semantic reasoning of LLMs with the exhaustive exploration of fuzzing, leveraging the strengths of both to cover the weaknesses of the other.
• Democratizes Security: It demonstrates that effective agentic vulnerability detection can be built on widely available, open-weight LLMs (like DeepSeek) rather than requiring exclusive access to curated, specialized frontier models (like Claude Mythos).
• Future-Proofs Codebases: The inferred invariants act as durable artifacts that can be maintained and reused, helping to prevent regressions and incomplete fixes in future development cycles.

The authors conclude that as AI increasingly participates in software development, the ability to extract and maintain generalized security knowledge (via specifications) is critical for securing the digital infrastructure.