RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements

1. Problem Statement

The paper addresses a critical gap in the evaluation of Large Language Model (LLM)-based REST API testing tools.

• Limitations of Current Metrics: Existing tools are primarily evaluated using code coverage and crash-based metrics (e.g., HTTP 5xx errors). These are insufficient for LLM-generated tests because they do not verify if the test actually validates the intended functional behavior defined in Natural Language (NL) requirements.
• The Oracle Problem: While LLMs can generate tests from NL requirements, ensuring these tests correctly reflect the requirements (rather than just the current implementation) is difficult. Traditional regression testing assumes the current implementation is correct; however, requirement-based testing must distinguish between valid and invalid behaviors.
• Lack of Ground Truth: Existing benchmarks lack verified NL requirements and specific mutations tied to those requirements, forcing researchers to rely on manual human judgment, which limits reproducibility and comparability.

2. Methodology: RESTestBench

The authors introduce RESTestBench, a benchmark designed to evaluate requirement-based test generation quantitatively.

A. Benchmark Components

1. REST Services: Three distinct, open-source REST services were selected to represent varying complexities:
   • FastAPI: A full-stack template with complex dependencies and security (JWT).
   • TodoApp: A compact .NET Minimal API for controlled, lower-complexity testing.
   • RealWorld: A NestJS backend with complex workflows (articles, comments, followers).
2. Natural Language Requirements:
   • 106 Human-Validated Requirements: Reverse-engineered from source code and integration tests to ensure they match actual system behavior.
   • Two Variants per Requirement:
     • Precise: Detailed, scenario-based descriptions including preconditions, steps, and expected outputs.
     • Vague: High-level, goal-oriented descriptions lacking operational details (simulating real-world ambiguity).
3. Requirement-Based Mutations (228 Total):
   • Instead of random code mutations, the authors manually designed Property-Based Mutations (PBMT).
   • Mutations are specifically crafted to violate the specific requirement (e.g., swapping email addresses in a user list response).
   • A test is considered effective only if it "kills" the mutant by detecting the specific violation of the requirement, not just any behavioral change.

B. Evaluation Framework

The benchmark uses Property-Based Mutation Testing (PBMT) to calculate a mutation score ($MS_\phi$).

• Non-Refinement Approach: The LLM generates a test in a single step based on the NL requirement and OpenAPI Specification (OAS), without interacting with the Service Under Test (SUT).
• Refinement Approach: The LLM generates a test, executes it against the SUT, and uses a "Judge" (another LLM call) to analyze the execution trace and refine the test iteratively.
• Fault Injection: To test the robustness of refinement, the benchmark evaluates generation against both the Valid SUT and Mutated SUTs. This determines if the refinement process adapts to faulty behavior (a negative outcome) or correctly identifies the fault.

3. Key Contributions

1. RESTestBench Framework: A reproducible benchmark comprising 3 services, 106 requirements (precise/vague), 228 manual mutations, and a Python interface for integrating new generation approaches.
2. Novel Metric: Introduction of a requirements-based mutation score that measures fault detection relative to specific requirements, extending the work of Bartocci et al.
3. Controlled Experiments: Two large-scale experiments evaluating 10 state-of-the-art LLMs (including Llama 3.3, GPT-5 variants, DeepSeek, and Claude) across both non-refinement and refinement strategies.

4. Key Results

The experiments were conducted on 10 LLMs using both precise and vague requirements.

• Impact of Requirement Precision:
  • Precise Requirements: Significantly improve test effectiveness. Top models (e.g., Sonnet 4.5, GPT-5 Nano) achieved mutation scores >90%.
  • Vague Requirements: Performance drops drastically (26–40 percentage points). Even top models struggle, with scores often falling below 60%.
• Refinement vs. Non-Refinement:
  • General Trend: Refinement loops generally improve mutation scores, especially for vague requirements (e.g., GPT-5 Nano improved by 29 points on vague requirements).
  • The "Faulty Adaptation" Risk: When refinement is performed against a mutated (faulty) SUT, the mutation score often drops below the non-refinement score.
  • Key Finding: For precise requirements, interacting with a faulty implementation can cause the LLM to "learn" the bug and treat it as correct, negating the benefits of refinement. In some cases (e.g., Sonnet 4.5), the single-step approach was more effective than refinement against a faulty SUT.
• Cost-Effectiveness:
  • Smaller models (e.g., GPT-5 Nano, GPT-OSS 120B) combined with refinement loops achieved comparable or superior results to massive models (like Sonnet 4.5) at a fraction of the cost (up to 25x cheaper).
  • Sonnet 4.5, while highly accurate, is prohibitively expensive for many use cases.

5. Significance and Implications

• Redefining Evaluation: The paper argues that code coverage is a weak proxy for LLM test quality. Requirement-based mutation testing is necessary to ensure tests validate business logic, not just code paths.
• The Danger of Refinement on Faulty Code: The study reveals a critical flaw in current refinement strategies: if an LLM interacts with a buggy implementation during the refinement loop, it may adapt its assertions to match the bug, effectively "hallucinating" correctness. This suggests that for high-precision requirements, interacting with the SUT might be unnecessary or even harmful if the SUT is not guaranteed to be correct.
• Requirement Engineering is Key: The quality of the input requirement is the single most significant factor in test generation success. Vague requirements remain a major bottleneck for automation.
• Model Selection: The results suggest that smaller, cost-efficient models, when paired with refinement loops and precise requirements, can outperform expensive frontier models in terms of cost-effectiveness.

Conclusion

RESTestBench provides a rigorous, reproducible standard for evaluating LLM-based API testing. It demonstrates that while LLMs are powerful, their effectiveness is heavily dependent on the precision of requirements and the nature of the SUT they interact with. The benchmark highlights that blindly refining tests against a running service can be counterproductive if that service contains undetected faults, emphasizing the need for robust requirement specifications over reliance on implementation behavior.