Technical Summary: Mitigating Errors in LLM-Generated Web API Invocations

Problem Statement

Integrating web APIs is fundamental to modern software development, yet writing correct invocation code remains a significant challenge due to complex, evolving specifications defined in OpenAPI documents. While Large Language Models (LLMs) show promise in code generation, empirical studies (specifically the authors' prior work, WAPIIBench) have demonstrated that they frequently generate incorrect web API invocations. These errors range from selecting wrong endpoints and HTTP methods to hallucinating illegal arguments or omitting required parameters.

Existing mitigation techniques like Retrieval-Augmented Generation (RAG) and Constrained Decoding (CD) have shown success in other domains but remain insufficiently understood for the specific challenges of web API invocations. Unlike local function calls, web API invocations require constructing explicit HTTP requests (URL, method, headers, body) based on external specifications that evolve independently of client code. This paper addresses the open question of how to effectively mitigate these specific error types using RAG and CD.

Methodology

The authors propose and systematically evaluate two complementary approaches to improve LLM-generated web API code:

1. Retrieval-Augmented Generation (RAG)

• Retriever Design: The authors designed a retriever that processes OpenAPI specifications. Unlike standard document chunking, they split specifications by endpoint, ensuring each chunk contains the complete documentation for a single endpoint (including inlined references, security schemes, and path parameters) to maintain context.
• Embedding and Storage: Chunks are embedded using the all-MiniLM-L6-v2 model and stored in a Chroma vector database. To improve retrieval hit rates, each endpoint is stored with up to five different embedding variants (e.g., varying the inclusion of parameter lists or response schemas).
• Prompt Injection: Retrieved endpoint documentation is formatted into a compact, TypeScript-inspired representation (Listing 2) and injected into the model's prompt. A truncation mechanism handles context length limits.

2. Constrained Decoding (CD)

• Constraint Generation: The authors developed an automatic translator that converts OpenAPI specifications into regular expression (regex) constraints. These constraints are designed to enforce strict compliance with the API specification during token generation.
• Hierarchical Structure: Constraints are organized hierarchically:
  1. HTTP Method: Restricts generation to supported methods.
  2. URL: Limits valid paths to those defined for the selected method.
  3. Parameters: Once an endpoint is identified, constraints restrict parameters to those defined for that endpoint, handling optionality, ordering, and data types.
• Implementation: A custom completion engine dynamically filters the model's next-token probability distribution, setting the probability of tokens that violate constraints to zero. This ensures that generated URLs, HTTP methods, and arguments are always specification-compliant.

3. Evaluation Framework

The study utilizes two datasets:

• Synthetic Dataset: Derived from WAPIIBench, covering 395 tasks across four APIs (Asana, Google Calendar, Google Sheets, Slack) with precise ground truth.
• Real-World Dataset: A new dataset constructed by mining GitHub repositories for Axios-based JavaScript files. It includes 28 tasks across 11 APIs, reflecting common usage patterns and file-level code context.

The evaluation measures Correctness (executable code matching ground truth), Hallucinations (illegal endpoints or arguments), and Executability. Experiments were conducted on 24 open-source LLMs (including CodeLlama, StarCoder, Qwen, DeepSeek) and two commercial models (GPT-4o, GPT-4o mini) across two setups: Full Completion (model selects endpoint and arguments) and Argument Completion (endpoint provided, model selects arguments).

Key Contributions

1. Specialized RAG for APIs: A retrieval method that processes OpenAPI specifications into self-contained, endpoint-level chunks, formatted specifically for LLM token efficiency and comprehension.
2. Automatic Constraint Generation: A system that translates OpenAPI specifications into regex-based constraints, ensuring generated code is strictly compliant with API definitions.
3. Real-World Dataset: A new dataset derived from GitHub repositories to complement synthetic benchmarks, offering diverse, realistic usage scenarios.
4. Systematic Empirical Evaluation: A comprehensive study of RAG, CD, and their combination across 24 models and two distinct datasets.

Results

Retrieval-Augmented Generation (RAG)

• Hallucination Reduction: RAG consistently reduces hallucinated endpoints and arguments across both datasets.
• Correctness Impact: The effect on overall correctness is ambivalent.
  • In the Full Completion setup, RAG significantly improves correctness (e.g., +113% on the synthetic dataset) by helping models select the correct endpoint.
  • In the Argument Completion setup, RAG often decreases correctness (e.g., -3% on synthetic) because models tend to over-include optional parameters found in the retrieved context, even when they are not required for the specific task.
• Model Dependence: The utility of RAG is highly model-dependent; some models leverage the context effectively, while others suffer from "parameter overuse."

Constrained Decoding (CD)

• Zero Hallucinations: CD reliably reduces illegal URLs, HTTP methods, and arguments to zero across all models and datasets.
• Correctness Improvement: CD substantially improves overall correctness. On the synthetic dataset, it yielded average gains of +209% (Full Completion) and +143% (Argument Completion).
• Executability: CD enables models that previously failed to generate syntactically valid code to produce executable implementations, though it slightly increases the rate of non-executable code in some cases due to probability distribution distortion.
• Consistency: Unlike RAG, CD provides consistent improvements across almost all models.

Combination (RAG + CD)

• Peak Performance: The combination achieves the highest absolute correctness scores (e.g., 73% on the synthetic dataset for Argument Completion).
• Trade-offs: While it retains CD's zero-hallucination guarantee, the combination is less consistent than CD alone. For Argument Completion, the "parameter overuse" issue from RAG persists, leading to lower average gains compared to CD alone in this specific setup.

Significance and Claims

The paper claims that Constrained Decoding is the most robust and reliable technique for web API invocation tasks, offering hard guarantees against specification violations and hallucinations. It argues that CD should be unconditionally recommended for this setting.

While RAG is effective at reducing hallucinations and aiding endpoint selection, its tendency to induce parameter overuse makes its net benefit highly dependent on the specific model and task. The authors conclude that CD alone is the safer choice unless maximum correctness is the priority and the specific model's behavior with retrieved context has been validated.

The work underscores the necessity of integrating quality assurance techniques (like CD) into AI-driven code generation workflows to achieve practical reliability, particularly for tasks involving external, evolving specifications like web APIs. The authors position their benchmark and methodology as a foundation for future research into other quality criteria (e.g., security) and different programming ecosystems.