💬 NLP

NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions

This paper introduces NL2SQLBench, the first modular benchmarking framework that decomposes NL2SQL systems into Schema Selection, Candidate Generation, and Query Revision modules to provide fine-grained metrics for evaluating effectiveness and efficiency, revealing significant accuracy and computational gaps in current methods while highlighting critical flaws in existing datasets and evaluation rules.

Original authors: Shizheng Hou, Wenqi Pei, Nuo Chen, Quang-Trung Ta, Peng Lu, Beng Chin Ooi

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shizheng Hou, Wenqi Pei, Nuo Chen, Quang-Trung Ta, Peng Lu, Beng Chin Ooi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly complex library (a database) filled with millions of books, but the books are written in a secret code called "SQL." You want to ask a question like, "Show me all the mystery novels published in 2023," but you don't know the secret code.

NL2SQL (Natural Language to SQL) is the technology that acts as a translator, turning your plain English question into that secret code so the library can find the answer.

Recently, Large Language Models (LLMs) like the ones powering chatbots have gotten really good at this translation. But here's the problem: everyone is racing to build the "best" translator, but nobody has a standardized way to check how they are actually working. Are they good at finding the right books? Are they good at writing the code? Or are they just guessing?

This paper introduces NL2SQLBench, a new "gym" or "testing ground" designed to fix this. Instead of just checking if the final answer is right or wrong, it breaks the translator down into three specific muscles to see which one is weak.

Here is how the paper explains it, using simple analogies:

1. The Three Muscles of the Translator

The authors realized that every NL2SQL system does three distinct things. They call these the Three Core Modules:

Muscle 1: Schema Selection (The Librarian)
- The Job: Before writing the code, the system must figure out which books and which pages are relevant. If you ask about "mystery novels," the system needs to know to look in the "Fiction" section, not the "Cookbooks" section.
- The Problem: If the librarian grabs the wrong section, the rest of the process is doomed.
- The New Test: Instead of just checking the final answer, NL2SQLBench checks: "Did the librarian pick the right books?" It measures Precision (did they grab only relevant books?) and Recall (did they miss any relevant books?).
Muscle 2: Candidate Generation (The Scribe)
- The Job: Once the right books are found, the system writes the secret code (SQL) to ask the library for the info.
- The Problem: The scribe might write a sentence that is grammatically perfect but asks for the wrong thing (e.g., asking for "all books" instead of "2023 books").
- The New Test: The framework categorizes errors. Did the code crash? (Execution Error). Did it run but give the wrong answer? (Incorrect Query). Or was it perfect? (Correct Query).
Muscle 3: Query Revision (The Editor)
- The Job: The system looks at its own draft code, tries to find mistakes, and fixes them.
- The Problem: Sometimes the editor fixes a mistake but accidentally breaks something that was already working.
- The New Test: The framework tracks if the editor actually improved the answer or just made it worse. It asks: "Did this step turn a wrong answer into a right one, or did it turn a right answer into a wrong one?"

2. The "Cost" of Being Smart

The paper also highlights a hidden cost: Time and Money.
Imagine you have a genius translator who can answer your question perfectly, but it takes them 10 hours to think and costs $50 to hire. That's not practical for a real app.
NL2SQLBench measures Efficiency: How many "tokens" (words) did the AI read? How many times did it have to call the "brain" (LLM)?

Finding: Some methods are super accurate but incredibly expensive (like hiring a team of 10 experts). Others are cheap and fast but make more mistakes. The paper helps you find the "sweet spot" where you get a good answer without breaking the bank.

3. The "Broken Map" Discovery

One of the most surprising findings in the paper is that the maps (the datasets used to train and test these systems) are often broken.

The Analogy: Imagine you are training a GPS. You tell it, "Drive to the Eiffel Tower," but the map says the Eiffel Tower is in the middle of a lake. If the GPS drives to the lake, the test says the GPS is wrong. But actually, the map was wrong!
The Reality: The researchers found that many of the "Gold Standard" answers in the popular BIRD dataset were actually incorrect. This means some AI systems were being punished for being right because the test itself was flawed. They also found that some questions were ambiguous (like asking for "the school" when there are five different ways to define a school), making it impossible to have a single "correct" answer.

4. Why This Matters (The Takeaway)

Before this paper, developers were like chefs trying to improve a recipe without a way to taste the ingredients individually. They just tasted the final dish and said, "It's good" or "It's bad."

NL2SQLBench gives them a taste test for:

The Ingredients (Did we pick the right tables?)
The Cooking (Is the code syntax correct?)
The Plating (Did the final edit improve the dish?)

The Result:
The paper reveals that while AI is getting smarter, it's still very inefficient and prone to "hallucinations" (making things up). It also warns us that we need to fix our test data (the maps) before we can trust the drivers (the AI).

In short: This paper builds a better ruler to measure AI translators, finds out that the rulers we were using before were broken, and gives developers a manual on how to build systems that are both smart and affordable.

1. Problem Statement

While Large Language Models (LLMs) have significantly advanced Natural Language to SQL (NL2SQL) capabilities, the rapid development of these methods has outpaced systematic evaluation. Current evaluation practices suffer from three critical limitations:

Holistic vs. Granular Metrics: Existing benchmarks (e.g., Spider, BIRD) rely primarily on end-to-end metrics like Execution Accuracy (EX) or Exact Match (EM). These fail to isolate the performance of individual pipeline stages (Schema Selection, Candidate Generation, Query Revision), making it difficult to identify specific bottlenecks.
Neglect of Efficiency: Most evaluations ignore computational costs (token usage, latency, and LLM call counts), which are critical for real-world deployment but often overlooked in favor of raw accuracy.
Dataset and Rule Flaws: Existing datasets contain inaccurate "gold" SQL annotations, and evaluation rules are often overly strict (requiring exact string matches or single-reference execution), failing to account for semantic ambiguity or multiple valid SQL formulations.

2. Methodology: The NL2SQLBench Framework

The authors propose NL2SQLBench, the first modular, multi-agent benchmarking framework designed to evaluate LLM-enabled NL2SQL solutions systematically.

A. Modular Decomposition

The framework dissects NL2SQL systems into three core modules, allowing for independent evaluation:

Schema Selection: Identifying relevant tables and columns from the database schema.
Candidate Generation: Generating candidate SQL queries based on the natural language query and selected schema.
Query Revision: Refining candidate queries to correct errors and select the final output.

B. Novel Fine-Grained Metrics

Instead of relying solely on end-to-end accuracy, the authors introduce specific metrics for each module:

Schema Selection: Precision, Recall, and F1-score for both table and column retrieval.
Candidate Generation:
- Correct Rate (CR): Proportion of queries yielding correct results.
- Incorrect Rate (IR): Proportion of syntactically valid but semantically wrong queries.
- Error Rate (ER): Proportion of queries causing runtime execution errors.
- Pass@k: The probability of generating at least one correct query among $k$ candidates.
Query Revision:
- Correctness Improvement (CI): Net gain in accuracy after revision.
- Transition Rates: Metrics like I2C (Incorrect-to-Correct) and E2C (Error-to-Correct) to measure repair capability, alongside C2I (Correct-to-Incorrect) to detect regressions.
Efficiency Metrics: Explicit tracking of #Tokens (prompt + completion) and #LLM Calls to quantify computational overhead.

C. Multi-Agent Architecture

The framework utilizes three specialized agents to ensure flexibility and reproducibility:

Result Collecting Agent: Extracts intermediate outputs (schemas, candidates, refined SQL) and usage statistics from diverse NL2SQL pipelines.
Postprocessing Agent: Executes SQL queries, compares results against ground truth, and standardizes outputs into structured JSON.
Benchmarking Agent: Computes the proposed metrics and generates comparative reports across different strategies, datasets, and LLMs.

3. Experimental Setup

Datasets: Evaluated on BIRD (large-scale, cross-domain) and ScienceBenchmark (highly domain-specific, complex schemas).
Models: Ten representative open-source NL2SQL approaches (e.g., DIN-SQL, MAC-SQL, CHESS, TA-SQL, OpenSearch-SQL) were tested using two LLMs: DeepSeek-V3 and GPT-4o mini.
Scope: The study covers the full pipeline from schema linking to final execution, analyzing performance across varying question difficulties (Simple, Moderate, Challenging).

4. Key Results and Findings

A. Module-Level Insights

Schema Selection: Multi-stage pruning and Preliminary-SQL enhanced strategies (e.g., CHESS, TA-SQL) yield the highest F1-scores. However, they often suffer from over-selection (low precision) on complex schemas, leading to high token costs. Single-call approaches (DIN-SQL, TA-SQL) offer a better accuracy-efficiency trade-off.
Candidate Generation: The dominant failure mode is semantic mismatch (Incorrect Rate), not execution errors. Syntactically valid but logically flawed SQLs vastly outnumber runtime errors.
Query Revision: Current revision strategies show limited effectiveness. While they can fix some errors, they often fail to improve the Correct Rate significantly and sometimes introduce regressions (turning correct queries into incorrect ones). The gap between the theoretical upper bound (Pass@k) and actual post-revision performance highlights a bottleneck in selecting the best candidate.

B. Efficiency vs. Accuracy

Cost-Quality Frontier: High-accuracy methods (e.g., OpenSearch-SQL, CHESS variants) often require massive token budgets (100k+ tokens) and numerous LLM calls, rendering them impractical for real-time deployment.
Scalability: Compact pipelines (e.g., TA-SQL, MAC-SQL) achieve competitive accuracy (59–61% on BIRD) with significantly lower costs (single-digit LLM calls, <10k tokens).

C. Critical Flaws in Existing Benchmarks

Inaccurate Gold Annotations: The authors discovered that a significant portion of "incorrect" predictions by NL2SQL models were actually correct, but the benchmark's gold SQL was wrong (e.g., selecting the wrong column).
Strict Evaluation Rules: Current rules penalize semantically equivalent but syntactically different SQLs.
Semantic Ambiguity: Many questions in BIRD have multiple valid interpretations (e.g., "school" referring to different columns), which single-reference evaluation fails to capture.

5. Key Contributions

First Modular Framework: Introduced NL2SQLBench, the first framework to decouple and evaluate the three core modules of NL2SQL systems independently.
Fine-Grained Metrics: Proposed a comprehensive suite of metrics (CR, IR, ER, CI, I2C, etc.) that provide deeper diagnostic insights than traditional end-to-end accuracy.
Comprehensive Benchmarking: Provided a rigorous evaluation of 10 state-of-the-art methods across two datasets and two LLMs, revealing that semantic alignment is the primary hurdle, not syntax.
Dataset Audit: Exposed critical inaccuracies in the BIRD dataset's gold SQL annotations and advocated for multi-reference evaluation and ambiguity-aware rules.
Practical Guidance: Offered actionable recommendations for developers, such as implementing adaptive complexity-based pipelines and "do-no-harm" revision policies to avoid regressions.

6. Significance

This work shifts the paradigm of NL2SQL evaluation from a "black-box" accuracy contest to a transparent, diagnostic, and efficiency-aware engineering process. By identifying that semantic misalignment and dataset noise are the true bottlenecks, the paper guides future research toward:

Developing better semantic verification mechanisms rather than just generating more candidates.
Creating robust, error-free benchmark datasets.
Designing cost-effective, adaptive systems suitable for real-world enterprise deployment.

The framework is open-sourced, enabling the community to reproduce these findings and develop more reliable NL2SQL solutions.