Technical Summary: Harnessing Code Agents for Automatic Software Verification

Problem Statement

Formal verification offers the strongest guarantees of software correctness but has historically failed to scale due to the prohibitive cost of manual proof construction. Interactive theorem provers (ITPs) like Coq require highly trained experts to compose proofs step-by-step, a bottleneck exacerbated in complex domains like separation logic (e.g., Iris), where concurrency and shared mutable state make proofs difficult and time-consuming.

Existing automated approaches attempt to solve this by embedding fixed, human-designed proof strategies into the system. These systems typically constrain Large Language Models (LLMs) to follow specific patterns, such as predicting one tactic at a time, retrieving premises, or recursively dividing goals. However, these approaches suffer from two critical limitations:

1. Partial Coverage: They achieve success rates of only 12%–48% on available benchmarks.
2. Limited Scope: Benchmarks often focus on sequential programs or simple data types, omitting the difficult dimensions of concurrency and fine-grained memory reasoning that define real-world verification challenges.

Furthermore, these systems rely on "scaffolding"—hand-engineered strategies that dictate how a proof is built. This paper argues that such scaffolding is unnecessary and limiting, preventing the full utilization of general-purpose code agents.

Methodology: The Aria System

The authors propose Aria, a system that pairs a general-purpose LLM code agent (specifically Claude Code running on Claude Opus 4.7) with a verification harness. Unlike prior work, Aria imposes no Coq-specific proof strategy, premise selector, or decomposition heuristic on the agent. Instead, the agent is given a whole lemma and free rein to choose its approach.

Architecture

Aria operates on a three-layer architecture:

1. Model Layer: A stateless LLM backend that generates candidate proofs based on structured prompts containing the current goal, context, and feedback.
2. Agent Layer: An orchestrator that manages the session, retrieves context (definitions, prior lemmas), and interprets model responses. It drives the agent non-interactively via a command-line interface.
3. Harness Layer: The trust anchor. It wraps the Coq kernel and enforces policy checks. It submits candidate proofs to the kernel; if rejected, it returns precise feedback (failing step, error message, pending goal) to the agent for revision.

The Harness Hook Language (HHL)

A key contribution is HHL, a declarative language used to define the verification harness. HHL allows engineers to specify "hooks" that run before or after agent actions (e.g., tool calls like Write, Edit, Bash).

• Pre-hooks: Enforce constraints before an action (e.g., blocking Admitted, banning new imports, or preventing dangerous shell commands).
• Post-hooks: Validate outcomes after an action (e.g., running the Coq kernel to verify the proof, checking for lemma coverage, or enforcing Iris coding conventions via a linter).
• Retry Loop: The system runs in a feedback-driven loop. If the harness rejects a proof, the error is fed back to the agent, which revises the proof. This continues for up to 30 attempts per lemma.

Key Operational Properties

• Soundness: Only proofs accepted by the Coq kernel are considered successful. The kernel acts as an infallible oracle; incorrect outputs cannot slip through.
• Completeness: The harness checks that the target lemma is actually proved and not silently dropped or weakened (e.g., via Admitted).
• Termination: A timeout check prevents divergent, non-terminating tactics from stalling the system.
• Autonomy: The system runs unattended. No human expert intervenes to write tactics, inspect goals, or restart runs.

Experimental Results

The authors evaluated Aria on four distinct benchmarks, achieving 100% coverage with zero failures and zero expert intervention across all settings.

1. Iris Core (Coq): The primary benchmark consists of 4,257 lemmas across the four core modules of Iris (algebra, bi, base_logic, program_logic), representing state-of-the-art higher-order concurrent separation logic.
   • Result: All 4,257 lemmas proved.
   • Efficiency: 79.2% solved on the first attempt; mean retries per lemma was 0.51. Total model time was ~380 hours.
2. RustBelt (Coq): Verified safety proofs for Rust's standard library (concurrency and interior mutability types like Arc, Mutex, RwLock).
   • Result: All 217 lemmas proved.
   • Significance: Demonstrates the approach works on software built on top of Iris, not just the logic itself.
3. reglang (Coq): A library of regular-language theory, previously the worst-performing benchmark for prior LLM provers (Rango proved only ~12.5%).
   • Result: All 318 lemmas proved.
   • Comparison: Prior systems managed barely one in eight; Aria proved all.
4. iris-lean (Lean 4): A port of Iris to the Lean 4 prover, targeting lemmas that had not yet been ported or proved.
   • Result: All 72 lemmas proved.
   • Significance: Shows the approach is not specific to Coq and works with the Lean kernel.

Model Comparison

The paper also compared the private Claude Opus 4.7 against open-source models (Kimi K2.6, DeepSeek V4 Pro). While open-source models could prove simpler lemmas, they struggled with long, complex proofs (e.g., >50 lines), often exhausting the retry budget. The private model reliably constructed proofs exceeding 200 lines.

Key Contributions

1. Harness + Code Agent Paradigm: The paper demonstrates that a general-purpose code agent, paired with a verification harness and free to choose its own strategy, outperforms systems with hard-wired proof strategies. It achieves full coverage on the hardest separation logic proofs, a feat previously unattained.
2. Iris and RustBelt Verification: This is the first empirical evidence that a state-of-the-art LLM can fully and automatically prove an expert-grade, state-of-the-art software verification library (Iris) and its downstream applications (RustBelt).
3. HHL (Harness Hook Language): A declarative language for defining reusable, auditable verification policies around LLM agents, separating the agent's reasoning from the correctness guarantees.

Significance and Claims

The paper concludes that the "scaffolding" of fixed proof strategies is unnecessary. By handing the entire lemma to a capable agent and using a harness to enforce soundness, completeness, and termination, the system can navigate the complex search space of formal proofs autonomously.

The authors claim that this approach removes the manual-proof bottleneck not just for Iris, but for the entire body of software built upon it. The results suggest that current state-of-the-art models are capable of writing proofs for state-of-the-art verified software development fully and automatically, provided they are guided by a rigorous verification loop rather than a constrained, human-designed strategy. The paper does not claim to solve specification generation, focusing strictly on the proof synthesis phase.