💻 computer science

VeriScale: Adversarial Test-Suite Scaling for Verifiable Code Generation

The paper introduces VeriScale, an adversarial framework that expands and distills test suites to create more rigorous benchmarks like VerinaPlus and VerinaLite, which reveal significant weaknesses in state-of-the-art LLMs' code generation and specification capabilities that existing benchmarks fail to detect.

Original authors: Yifan Bai, Xiaoyang Liu, Zihao Mou, Guihong Wang, Jian Yu, Shuhan Xie, Yantao Li, Yangyu Zhang, Jingwei Liang, Tao Luo

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yifan Bai, Xiaoyang Liu, Zihao Mou, Guihong Wang, Jian Yu, Shuhan Xie, Yantao Li, Yangyu Zhang, Jingwei Liang, Tao Luo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a robot chef to cook a complex meal. You give the robot a recipe (the "specification") and ask it to cook the dish (the "code"). To see if the robot is good, you taste the food.

The Problem: The "Easy Quiz" Trap
Currently, the tests we use to check these AI chefs are like a very easy quiz. We only give them a few simple ingredients (like "salt" and "water") and ask for a simple result ("soup"). If the robot makes soup, we give it an A+.

But here's the catch: The robot might be cheating. It might have memorized the answer for "salt + water = soup," but if you give it a weird ingredient like "toothpaste," it might still try to make soup and claim it's correct. Or, it might write a recipe that says "add salt," but forgets to say "don't add poison." Because the test didn't include "toothpaste" or "poison," the robot passes, even though it's actually dangerous or broken.

The paper argues that current tests are too small and too easy, making AI look smarter than it really is.

The Solution: VeriScale (The "Stress Test" Framework)
The authors created a new system called VeriScale. Think of this as a "Stress Test" or a "Red Team" exercise for AI code. Instead of just asking the AI to cook soup, VeriScale tries to trick the AI into failing.

It works in two main steps:

Step 1: The "Obstacle Course" (Expansion)

First, VeriScale creates a massive, chaotic obstacle course of ingredients.

The Seed: It starts with normal ingredients.
The Mutation: It uses a "mutation machine" to twist and turn these ingredients. It turns a cup of water into a bucket of ice, or a pinch of salt into a mountain of salt. It creates weird, edge-case scenarios that the AI has never seen before.
The "Trickster" Chef (Adversarial Synthesis): This is the clever part. VeriScale asks a different, very smart AI to act as a "Trickster." This Trickster tries to write a fake recipe that looks like it follows the rules but actually produces garbage.
- Example: If the rule is "The soup must be hot," the Trickster might write a recipe that outputs "Ice Cream" but claims, "Well, technically, ice cream is a food, so I followed the rule!"
- VeriScale runs these Trickster recipes against the AI's rules. If the AI's rules accept the Ice Cream as "Hot Soup," VeriScale catches the flaw. It generates a huge list of these "gotcha" moments.

Step 2: The "Essential Checklist" (Reduction)

Now, VeriScale has thousands of these tricky test cases. Running all of them on every AI would take forever and cost a fortune. So, it performs a "Reduction."

It asks: "Which of these 1,000 tricks are the most important?"
It keeps the specific tricks that catch the most mistakes and throws away the ones that are just repeats.
The result is a compact, super-challenging test that is small enough to run quickly but still catches every single flaw the massive list would have found.

The Results: The "Truth Serum"
The authors tested this new system on the best AI models available (like GPT-5.5 and others).

The Old Test: The AIs got scores like 96% or 98%. They looked like geniuses.
The VeriScale Test: When put through the stress test, the scores dropped dramatically. One top model's score fell from 96% down to 86% for coding, and from 68% down to 44% for writing the rules (specifications).

The Big Takeaway
The paper shows that the old tests were lying to us. They were overestimating how good AI is at writing safe, correct code. The new "VeriScale" tests reveal that while AI is great at writing code that looks right, it is still very bad at writing rules that catch every possible mistake.

They also released two versions of this new test:

VERINAPLUS: The massive, full-scale stress test (83 times bigger than the original).
VERINALITE: The "lite" version. It's 14 times bigger than the original but uses the "Reduction" trick to be fast and cheap, while still catching the same mistakes.

In short: VeriScale is a tool that stops us from being fooled by AI that only knows how to pass easy tests. It forces the AI to prove it can handle the weird, messy, and tricky real world.

Technical Summary: VeriScale: Adversarial Test-Suite Scaling for Verifiable Code Generation

Problem Statement

As Large Language Models (LLMs) are increasingly deployed for software engineering, evaluating their ability to generate not just functional code but also formally verifiable specifications and proofs has become critical. Existing benchmarks for verifiable code generation, such as VERINA and CLEVER, rely heavily on test-suite evaluation to assess Specification Generation (SpecGen) and Code Generation (CodeGen). However, these benchmarks suffer from a fundamental limitation: they are constrained by a scarcity of high-quality test cases, particularly negative test cases (unexpected inputs and unexpected outputs).

Current benchmarks typically provide only a limited number of positive input-output pairs. Consequently, models can achieve inflated pass rates by satisfying these sparse tests without truly capturing program intent or ensuring the soundness and completeness of their generated specifications. This leads to an overestimation of model capabilities, where incorrect implementations or unsound specifications pass evaluation simply because the test suite fails to expose logical flaws or boundary violations.

Methodology: The VeriScale Framework

To address these limitations, the authors propose VeriScale, a novel framework driven by adversarial implementations to systematically scale and refine test suites for verifiable code generation. The framework operates in two primary stages: Test-Suite Expansion and Test-Suite Reduction.

1. Test-Suite Expansion

The goal of this stage is to construct a diverse and challenging pool of test cases, including expected input-output pairs, unexpected inputs, and unexpected outputs.

Seed Generation and Type-Aware Mutation: Drawing inspiration from EvalPlus, VeriScale uses LLMs to synthesize initial seed inputs. These seeds are then expanded via an iterative, type-aware mutation strategy. The framework formalizes mutation as a five-tuple process ( $M = (T, D, O, p, q)$ ) that preserves input types while introducing structural perturbations. This includes reusing contextual values from an "ingredient pool" or applying type-specific heuristics to generate complex edge cases.
Precondition-Guided Input Classification: Candidate inputs are rigorously classified against ground-truth preconditions using a two-stage verification pipeline in Lean.
- Syntax Filter: Ensures candidates are syntactically valid Lean expressions.
- Bidirectional Verification: Uses #guard decide and plausible proof search to classify inputs as "expected" (satisfying the precondition) or "unexpected" (violating it).
Adversarial Synthesis of Unexpected Outputs: This is the core innovation. Instead of relying on LLMs to hallucinate random incorrect outputs, VeriScale synthesizes adversarial implementations.
- A capable model acts as a "red team," generating formal specifications for a problem.
- Another model (or the same one in a different role) attempts to craft adversarial implementations designed to exploit weaknesses in these specifications.
- These implementations are executed on expected inputs. If an adversarial implementation produces an output that differs from the ground truth but is falsely accepted by the generated postconditions, this output is retained as a high-quality "unexpected output." This process ensures the test cases are genuinely informative for exposing specification unsoundness.

2. Test-Suite Reduction

To manage the computational cost of the expanded suites, VeriScale employs reduction strategies to distill the large pool into compact yet discriminative suites.

Boundary-Preserving Reduction (for Unexpected Inputs): Instead of random sampling, this strategy prioritizes inputs that correspond to common boundary violations (e.g., empty containers, zero values, mismatched lengths). It retains representative examples from critical boundary buckets to ensure coverage of typical precondition violations.
Adversary-Killing Reduction (for Expected Inputs): This strategy repurposes the adversarial implementations synthesized in the expansion phase. Each expected input-output pair is evaluated based on how many "mutant" (adversarial) implementations it can "kill" (i.e., produce a different output from). The reduction is formulated as a set-cover problem, greedily selecting pairs that cover the maximum number of previously un-killed mutants. This ensures the reduced suite retains the ability to distinguish flawed specifications.

Key Contributions

VeriScale Framework: The authors propose a general framework that leverages adversarial implementations to drive both the systematic expansion and reduction of test suites for verifiable code generation.
VERINAPLUS and VERINALITE: The framework is instantiated on the VERINA benchmark to create two new datasets:
- VERINAPLUS: An expanded benchmark with test suites scaled by over 83× (specifically 62.83× for expected inputs, 87.79× for unexpected outputs, and 183.08× for unexpected inputs).
- VERINALITE: A lightweight variant scaled by 14×, optimized via the reduction strategies to maintain discriminative power with lower evaluation costs.
Empirical Validation: The paper demonstrates that the original benchmarks significantly overestimate model capabilities. The enhanced benchmarks reveal substantial weaknesses in state-of-the-art LLMs that were previously hidden.

Experimental Results

The authors evaluated eight state-of-the-art LLMs (including GPT-5.5, Claude-Sonnet-4.6, and DeepSeek-V4-Pro) across VERINA, VERINAPLUS, and VERINALITE.

Exposure of Model Weaknesses: The augmented benchmarks caused sharp performance drops. For instance, GPT-5.5 saw its SpecGen score plummet from 68.78% on VERINA to 44.44% on VERINAPLUS, and its CodeGen score drop from 96.83% to 86.24%. This indicates that the original benchmark's sparse test cases allowed models to pass without true understanding.
Reduction of Evaluation Uncertainty: The expanded test suites significantly narrowed the error bars in SpecGen evaluation. In the original benchmark, indeterminate verification outcomes created wide gaps between upper and lower bounds (e.g., a 13.23% gap for Claude-Opus-4.7). On VERINAPLUS, this ambiguity was resolved, shrinking the gap to 4.76%, providing a more statistically reliable assessment.
Robustness Gap: The results highlight a fundamental disparity: models are significantly more proficient at CodeGen than SpecGen. While CodeGen scores saw moderate declines, SpecGen scores suffered precipitous drops, suggesting that synthesizing sound and complete formal specifications remains a major bottleneck for current LLMs.
Efficiency of Reduction: VERINALITE achieved performance degradation profiles nearly identical to VERINAPLUS but with significantly reduced computational overhead. Evaluation time for VERINALITE was only 1.38× (SpecGen) and 1.05× (CodeGen) the baseline, compared to 1.80× and 1.50× for the full VERINAPLUS suite.

Significance and Claims

The paper claims that VeriScale provides a crucial, highly reliable standard for advancing LLM-based verifiable code generation. By moving beyond sparse positive tests to include adversarially synthesized negative cases, the framework offers a more rigorous evaluation that prevents the overestimation of model capabilities.

The authors position VeriScale as complementary to existing efforts; rather than introducing new tasks, it strengthens existing benchmarks by scaling the quality and quantity of test signals. The work underscores the necessity of adversarial test scaling to expose the fragility of current models in formal reasoning tasks.

Limitations Acknowledged:
The authors note that VeriScale relies on the availability of ground-truth preconditions and reference implementations. Furthermore, the quality of the synthesized unexpected outputs is bounded by the reasoning capabilities of the LLM acting as the "red team." Finally, the construction phase remains compute-intensive, requiring massive LLM sampling and Lean executions.

Future Work:
The authors propose evolving VeriScale into a closed-loop repair system to guide models in patching flawed specifications. They also plan to use VeriScale as an adversarial data engine to synthesize training corpora for dedicated autoformalization models and generalize the framework to other verification backends like Dafny.