A Cost-Aware Agentic Architecture for NL-to-SQL over Nested Enterprise Schemas, with a New Benchmark
This paper introduces the DevRev NL2SQL benchmark, featuring 900 execution-verified queries with nested structures and a new Semantic Depth Score, alongside a cost-aware agentic architecture that achieves a significant 91.7% answer correctness on this benchmark and remains competitive on Spider 2.0.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a world where you could ask a computer complex questions about your business and receive precise answers in plain English, without needing to know the secret language of databases. For years, researchers have been teaching computers to translate human questions into database commands, a field known as natural-language-to-SQL. Early successes were impressive, but they relied on testing the computers with simple, flat lists of data, like a spreadsheet with rows and columns. In the real world, however, business data is rarely so tidy. Modern enterprise systems are built around complex workflows, where a single item, like a customer support ticket, is connected to a web of other items, nested inside lists, and buried within layers of information. These structures are often described as graph-like, meaning they resemble a map of connections rather than a simple table. When researchers tried to apply their best tools to this messy, real-world data, the systems often failed, unable to navigate the deep, tangled paths of information.
A team of researchers has now tackled this specific challenge by building a new testing ground and a smarter computer system designed to handle it. They created a benchmark called DevRev, which contains 900 real-world questions based on a live business database. Unlike previous tests, this database is filled with deeply nested structures, where information is stored in lists that contain other lists, and where items are linked together in complex ways that change depending on the context. To measure how well a system understands these questions, the team introduced a new scoring method called the Semantic Depth Score. This score does not just count how long a question is; instead, it measures how many layers of reasoning are required to find the answer, such as needing to compare different time periods, calculate new values from old ones, or follow a chain of connections between different types of business objects.
The researchers found that the most advanced systems available at the time, which had performed well on simpler tests, struggled significantly with this new, complex data. These systems, which often try to generate many different possible answers and then pick the best one, could not correctly navigate the nested lists or understand the specific rules for linking different types of business records. Their success rate on the new benchmark hovered around one-third, meaning they got the wrong answer for most questions. The researchers argued that the problem was not just a lack of practice, but a fundamental mismatch between how these systems were built and how the data was structured. The systems were trying to force a flat, simple approach onto a deeply layered reality.
To solve this, the team designed a new computer architecture that works more like a careful investigator than a guesser. Instead of generating dozens of possible answers and hoping one is right, this system follows a single, focused path. It starts by asking a central coordinator to look at the database and figure out which specific pieces of information are needed. If the system gets stuck because it cannot find a table or a column, it does not just guess; it goes back, asks for more details about the structure, and tries again with the new information. This process is guided by a detailed map of the database that includes not just the names of the columns, but also descriptions of what the data inside them actually means. The system also keeps a running log of every mistake it makes, learning from each failure to avoid repeating the same error. It checks its own work before sending it to the database, ensuring that the logic is sound and that it is not accidentally creating a massive, confusing mess of data.
When tested on the 900 complex questions, this new system achieved a success rate of 91.7 percent, a massive improvement over the previous best attempts. It was able to correctly answer questions that required following multiple steps, calculating derived values, and navigating through nested lists of data. The researchers also showed that this approach was far more efficient, costing significantly less to run because it did not waste resources generating and testing dozens of incorrect answers. On a separate, large-scale test of simpler data, the system performed just as well as the leading competitors, proving that its specialized design for complex data did not come at the expense of handling standard data.
The study suggests that the future of asking computers questions lies not in making them try harder or generate more options, but in giving them better tools to understand the structure of the data they are working with. By building systems that can iteratively explore a database, learn from their mistakes, and understand the deep connections between different pieces of information, researchers have created a path toward reliable, intelligent interfaces for the complex, nested data that powers modern businesses. The work highlights that for computers to truly understand our questions, they must first understand the intricate, layered world where our data lives.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.