🤖 machine learning

Benchmarking Web API Integration Code Generation

This paper introduces WAPIIBench, a dataset and evaluation pipeline revealing that current open-source large language models struggle significantly with generating correct web API integration code, achieving less than 40% success due to issues like hallucinated endpoints and incorrect argument usage.

Original authors: Daniel Maninger, Leon Chemnitz, Amir Molzam Sharifloo, Jannis Brugger, Mira Mezini

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daniel Maninger, Leon Chemnitz, Amir Molzam Sharifloo, Jannis Brugger, Mira Mezini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, well-read robot how to order a specific meal from a massive, complex restaurant menu. The menu isn't just a list of dishes; it's a strict set of rules about how to write your order, which buttons to press, and exactly what ingredients to list. If you get even one detail wrong, the kitchen rejects the order.

This paper is about testing how well these "robots" (which are actually Large Language Models, or LLMs—the same technology behind AI chatbots) can write the code needed to send these orders to real-world web services (like Google Calendar, Slack, or Asana).

Here is a breakdown of what the researchers did and what they found, using simple analogies:

The Problem: The "Magic" Isn't Magic Yet

Developers use "Web APIs" to make different software talk to each other. It's like connecting Lego blocks. Usually, humans have to write the instructions to snap these blocks together. The hope was that AI could do this automatically.

The researchers asked: Can an AI look at a simple request like "Add a new event to my calendar" and write the perfect, error-free code to make that happen?

The Experiment: Building a "Driving Test" for AI

To find out, the team built a special test called WAPIIBench. Think of it as a driving test for AI code.

The Course: They created 395 specific "driving scenarios" using four popular real-world services (Asana, Google Calendar, Google Sheets, and Slack).
The Rules: They used the official "driver's manuals" (OpenAPI specifications) for these services to know exactly what a correct order looks like.
The Test: They gave the AI a prompt (e.g., "Create a new calendar") and asked it to write the code.
The Check: Instead of just reading the code to see if it looks right, they actually tried to run it in a safe, controlled sandbox. If the code tried to send an order to a non-existent address or used the wrong ingredients, the test failed.

The Results: The AI Gets Lost Often

The results were a bit disappointing for those hoping for instant automation.

The Score: Even the best open-source AI models only got about 30% to 40% of the tasks completely correct. The very best commercial model (GPT-4o) did better, hitting around 60-77%, but that still means it failed nearly a third of the time.
The "Hallucinations": This is the biggest issue. The AI often made things up.
- Fake Addresses: The AI would invent URL addresses that didn't exist (like telling a taxi driver to go to "123 Fake Street" when the restaurant is actually at "456 Main St"). This happened in up to 39% of cases.
- Wrong Ingredients: The AI would include parameters (ingredients) that the service didn't accept, or miss ones that were required.
Partial Success: The AI was good at remembering the general shape of the code (like knowing it needs to use the POST method, which is like knowing you need to "place an order" rather than "cancel one"). But when it came to the specific details of where to send it and exactly what to say, it struggled to put the pieces together correctly.

A Few Surprising Twists

Bigger Isn't Always Better: Sometimes, a medium-sized AI model performed worse than both a tiny one and a huge one. It's like a student who studied too hard for the wrong test and got confused.
Memory vs. Understanding: The AI seemed to have "memorized" parts of the manuals from its training data. It knew some URLs and argument names, but it couldn't reliably combine them to solve a new, specific problem. It was like a student who memorized the answers to last year's math test but couldn't solve this year's problems.
The "Fill-in-the-Blank" Trick: When the researchers gave the AI the correct address (URL) and just asked it to fill in the rest, the AI did much better. This suggests the AI knows how to follow instructions if it doesn't have to guess the destination first.

The Bottom Line

The paper concludes that while AI is getting good at writing code, it is not yet reliable enough to automatically connect software systems without human supervision. If you let an AI write code to connect your business apps to the internet right now, it will likely send your data to the wrong place or break the connection about 60% to 70% of the time.

The researchers say we need better safety checks and new ways to help the AI "look up" the rules in real-time (rather than relying on memory) before we can trust it to build these connections on its own.

Technical Summary: Benchmarking Web API Integration Code Generation

Problem Statement

Web API integration is a fundamental component of modern software infrastructure, yet writing correct code to invoke these APIs remains a challenging and tedious task for developers. While Large Language Models (LLMs) have demonstrated significant potential in automating software development, their specific efficacy in generating web API invocation code remains largely unexplored. Unlike local function calls, web API invocations involve complex syntax, including specific HTTP methods, parameterized URLs, request bodies, headers, and query parameters, often documented in external specifications (e.g., OpenAPI) that are not always accessible within the model's context window.

The central research question addressed is: How correct is LLM-generated web API invocation code, and what types of errors are commonly present in such code?

Methodology

To answer this question, the authors introduce WAPIIBench, a novel dataset and evaluation pipeline designed to systematically assess LLM capabilities in this domain.

1. Dataset Creation (WAPIIBench)

The dataset consists of 395 task–configuration pairs derived from four real-world APIs adhering to the OpenAPI standard: Asana, Google Calendar, Google Sheets, and Slack Web.

Synthesis: Unlike sourcing from existing repositories (which often lack unambiguous task descriptions), the authors synthesized tasks using Gemini 1.5 Pro. The model was provided with full OpenAPI specifications and detailed instructions to generate a natural language task description ( $t$ ) and a corresponding ground-truth request configuration ( $c$ ).
Validation: The generated data underwent rigorous automated and manual curation to ensure that every task had a unique, solvable configuration and that arguments were correctly reflected.
Structure: Each sample represents an API endpoint, capturing the HTTP method, URL, and arguments (body, headers, query parameters).

2. Evaluation Pipeline

The evaluation follows a four-stage process:

Code Generation: LLMs are tasked with completing starter code (JavaScript using the Axios library) based on a task description. Two setups were tested:
- Full Completion: The model must generate the HTTP method, URL, and all arguments.
- Argument Completion: The model is provided with the correct method and URL and must only generate the arguments.
Code Execution: Generated code is executed in a controlled, sandboxed environment (Mock) that intercepts outgoing requests. This approach prioritizes functional correctness over syntactic similarity, capturing the actual request configuration ( $c'$ ) sent by the code.
Correctness Analysis: The captured configuration ( $c'$ ) is compared against the ground truth ( $c$ ) and validated against the API specification ( $s$ ).
Metrics: The study employs fine-grained metrics, including:
- Correct Implementations: Percentage of executable code matching the ground truth.
- Illegal Implementations: Code violating the API specification.
- URL/Method Accuracy: Correctness of endpoints and HTTP methods.
- Argument Precision/Recall: Accuracy of parameter names and values.

3. Models Evaluated

The benchmark evaluated 21 state-of-the-art open-source LLMs (including StarCoder, DeepSeek-Coder, Qwen-Coder, Code Llama, and Llama 3.1) alongside commercial models (GPT-4o and GPT-4o mini) to establish performance baselines.

Key Results

The experiments reveal that generating correct web API invocation code is a significant challenge for current LLMs, even for the most capable open-source models.

Overall Correctness: None of the evaluated open-source models solved more than 40% of the tasks.
- In the Full Completion setup, the best open-source model (Code Llama 70B) achieved 30% correctness, while GPT-4o reached 60%.
- In the Argument Completion setup (where the endpoint is known), Code Llama 70B improved to 40%, and GPT-4o reached 77%.
Error Types:
- Hallucinated Endpoints: A primary failure mode is the generation of incorrect or non-existent URLs. Up to 39% of URLs were illegal in the full completion setup.
- Argument Errors: Models frequently hallucinate parameter names (up to 31% illegal arguments) or omit required arguments.
- Execution Failures: Several models (e.g., Qwen2.5-Coder, Llama 3.1) produced non-executable code in the full completion setup due to syntax errors or refusal to complete the code, resulting in 0% correctness metrics for those models.
Model Behavior:
- Models generally perform well on predicting HTTP methods (GET/POST) but struggle with the unrestricted nature of URLs and arguments.
- Larger models do not always outperform smaller ones; in some families, medium-sized variants performed worse than both smaller and larger counterparts.
- Performance varies significantly across different APIs, likely correlating with the prevalence of specific APIs in the models' training data.

Significance and Contributions

The paper makes three primary contributions to the field of software engineering and AI:

WAPIIBench Dataset: The first dataset of its kind containing nearly 400 pairs of web API invocation tasks and expected outcomes across four real-world APIs, specifically designed for execution-based correctness analysis.
Open-Source Evaluation Pipeline: A novel, safe, and automated pipeline for evaluating code generation models on web API tasks using fine-grained metrics that distinguish between syntactic correctness, specification compliance, and functional success.
Empirical Insights: The study provides the first systematic evidence that while LLMs possess some memorized knowledge of API specifications, they struggle to synthesize this knowledge into correct, specification-compliant invocations without external context. The high rate of hallucinated endpoints and arguments highlights a critical gap in current model reliability for production-grade API integration.

The authors conclude that while LLMs show promise, significant limitations remain. They emphasize the necessity of integrating quality assurance techniques (such as retrieval-augmented generation, fine-tuning, or constrained decoding) into AI-driven code generation workflows to ensure the correctness and reliability of web API integration code. The benchmark serves as a foundation for future research into improving these capabilities.