Bounded Semantic Planning and Deterministic Compilation for Reliable Enterprise Text-to-SQL

In the world of modern business, data is often locked away in vast, complex digital warehouses. To ask a simple question like "How many policies did agent two sell last year?" requires navigating a maze of tables, relationships, and rules that define how the company operates. For years, researchers have tried to teach computers to translate these natural language questions directly into the specific code needed to pull the answers. This is the promise of text-to-SQL: a machine that understands human intent and writes the correct instructions to retrieve the data. However, a significant problem remains. When a computer writes these instructions on its own, it often produces code that runs perfectly but returns the wrong answer. It might connect the wrong pieces of information or count things in the wrong way, creating a result that looks plausible but is fundamentally flawed. This is dangerous in an enterprise setting, where a wrong number can lead to bad business decisions.

A new study explores a different approach to solving this reliability problem. Instead of asking an artificial intelligence to write the entire code from scratch, the researchers built a system where the AI acts as a guide rather than a builder. The AI's job is limited to understanding the question and selecting from a pre-approved list of valid paths and relationships. Once the AI makes its selection, a separate, rigid computer program takes over. This program translates the selection into the final code, ensuring that the connections are made correctly and the counting rules are followed exactly. The researchers tested this method against the traditional "write-it-all" approach using a complex insurance dataset. They found that the guided system was far more reliable, producing correct answers almost every time, while the traditional system frequently generated code that ran but gave the wrong results.

The core of this research lies in changing where the uncertainty lives. In a standard system, the artificial intelligence model must decide everything: which tables to join, how to link them, and how to aggregate the numbers. Because these models generate text based on probability, they can sometimes choose a path that is technically possible but semantically wrong. The new system, which the author calls Semantic Path Compilation, draws a strict line in the sand. The artificial intelligence is allowed to ground the words in the question to specific business concepts and choose from a finite set of options provided by a human-curated map of the data. It cannot invent new connections or write code fragments. Once the AI makes its choice, a deterministic engine takes the wheel. This engine is a piece of software that follows fixed rules to turn the AI's selection into the final database query. It checks for errors, ensures the logic holds up, and only releases the code if it passes every check. If the AI's choice is unclear or violates a rule, the system refuses to answer rather than guessing.

To test this, the researchers used a benchmark based on an insurance company's data, which involves many complex relationships, such as the different roles a person might play (like a policyholder, an agent, or an underwriter) and how those roles connect to policies and claims. They ran two different systems on the same set of thirty-eight questions. The first system was the traditional approach, where the AI generated the code directly from the database structure. The second system was the new guided approach. They ran each question three times to see how consistent the results were. The traditional system succeeded in getting the right answer for all three runs on only twenty-one of the thirty-eight questions. In contrast, the guided system succeeded on thirty-seven of the thirty-eight questions. More importantly, the traditional system produced twenty-nine instances where the code ran successfully but the answer was wrong. The guided system produced zero such errors. When it failed, it simply refused to give an answer, rather than providing a misleading number.

The study also looked at what happens when the data is slightly altered to expose hidden errors. In the insurance dataset, some questions require counting items at a specific level of detail. If the computer joins tables incorrectly, it might accidentally multiply the numbers, creating a result that is too high. The researchers created "counterfactual" versions of the database where these multiplication errors would become obvious. They found that the traditional system often failed to catch these errors, returning inflated numbers that looked correct on the original data but were wrong on the modified data. The guided system, because it was forced to follow the strict rules of the semantic map, avoided these traps. It consistently produced the correct count, even when the data was tweaked to trick a less careful system.

The researchers were careful to note that their success came from the entire system working together, not just the new software engine. The guided system had access to a detailed, human-written map of the business rules, which the traditional system did not have. This means the improvement is a result of combining better knowledge with a more careful process. The study does not claim that the new method will work for every possible question or that it removes all uncertainty. The artificial intelligence still has to make the initial choice, and if the business rules are not properly defined in the map, the system will refuse to answer. However, for the specific type of complex, rule-heavy questions found in this insurance domain, the combination of a guided AI and a rigid compiler proved to be a powerful way to ensure reliability.

The findings suggest a shift in how we might build these tools for the future. Instead of hoping that a large language model will get the complex logic right every time, we can build systems that constrain the model to make safe choices and then use deterministic software to execute them. This approach trades some flexibility for a much higher degree of trust. In a world where data drives critical business decisions, the ability to say "I don't know" rather than "here is a wrong answer" is a significant advantage. The study shows that by moving the heavy lifting of logic and connection from the probabilistic model to a rule-based engine, we can achieve a level of consistency that was previously difficult to reach. The result is a system that is not just smart, but also dependable.