💬 NLP

TraceSQL: Traceable Answerability Estimation for Reference-Free Text-to-SQL Verification

The paper proposes TraceSQL, a lightweight and traceable verification model that utilizes 67 explicit diagnostic features to estimate the answerability of generated SQL queries without ground-truth references, achieving competitive performance while providing interpretable evidence for its predictions.

Original authors: Neelesh Kumar Shukla, Debasmita Panda, Srutanik Bhaduri, Aditya Banerjee, Viji Krishnamurthy

Published 2026-08-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Neelesh Kumar Shukla, Debasmita Panda, Srutanik Bhaduri, Aditya Banerjee, Viji Krishnamurthy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital age, a vast amount of human knowledge is locked inside massive databases, waiting to be asked about. To unlock this information, people use a system called Text-to-SQL, which acts as a translator. When a person types a question in plain English, the system attempts to convert it into a precise set of instructions known as a query, allowing a computer to search the database and return an answer. This technology has advanced rapidly, powered by sophisticated artificial intelligence that can handle complex requests. However, a significant problem remains: once the system generates a query, how can anyone be sure it is correct? In a controlled test environment, experts can compare the computer's work against a known correct answer. But in the real world, where these systems are actually used, there is no reference key to check against. The system must decide for itself whether the query it just wrote is trustworthy, a task that is difficult because the computer might produce a query that looks perfect but answers the wrong question or uses the wrong data.

Researchers at Oracle Corporation have developed a new method to solve this problem of self-verification, creating a system they call TraceSQL. Instead of simply guessing whether a generated query is right or wrong, this new approach builds a detailed report card for every single attempt. The system examines the user's question, the database structure, and the generated query to look for specific signs of trouble. It checks if the question was vague, if the query matched the available data, if the mathematical logic was sound, and if the final result aligned with what the user actually wanted. By breaking down the verification process into sixty-seven distinct, understandable signals, the system can not only predict if a query is correct but also explain exactly why it made that decision.

The researchers tested this method using a large collection of questions and database scenarios. They compared their new system against a leading existing method that relies on a massive artificial intelligence model to judge correctness. The results showed that the new, lighter system was more accurate. It correctly identified valid queries and spotted errors more often than the heavier model, achieving a success rate of 66.47 percent compared to 61.87 percent for the other method. More importantly, the new system provided a clear trail of evidence. When it flagged a query as potentially incorrect, it could point to the specific reason, such as a missing filter, a wrong connection between data tables, or a misunderstanding of a business term. This transparency allows human users or other software to inspect the reasoning behind the decision, rather than just accepting a black-box score.

The study suggests that reliable verification does not require a massive, opaque artificial intelligence to look at the final result. Instead, it can be achieved by carefully measuring specific, logical properties of the question and the query. The system found that the most important clues came from a mix of two types of evidence: the structural details of the query itself, such as whether it included specific commands to limit results or group data, and the semantic alignment, which checks if the query truly matches the intent of the user's question. By combining these concrete structural checks with an understanding of the question's meaning, the system creates a robust way to verify answers without needing a reference key.

This work highlights a shift in how we might trust artificial intelligence in critical tasks. Rather than relying solely on the intuition of a large model, the researchers demonstrated that a system built on explicit, traceable rules can be both more accurate and more understandable. The ability to trace a decision back to its source evidence means that errors can be diagnosed and fixed more easily. In a world where automated systems are increasingly responsible for retrieving and analyzing data, the ability to see the work behind the answer is just as important as the answer itself. The researchers plan to expand their training data to see if these findings hold true across even wider ranges of questions and databases, but the current results show a clear path toward making text-to-query systems more reliable and transparent for everyday use.

Technical Summary: TraceSQL

Problem Statement

Text-to-SQL systems have advanced significantly with Large Language Models (LLMs), yet generated SQL queries often contain subtle semantic, schema-grounding, or structural errors. In real-world deployments, verifying these queries is challenging because ground-truth SQL or reference execution results are unavailable at inference time. Existing verification methods face specific limitations:

LLM-based judges and agents: While they can inspect SQL, their decisions are often difficult to trace, lacking a unified, measurable form of evidence.
Outcome Reward Models (ORMs): Models like GradeSQL learn from execution-labeled candidates to assign correctness scores. However, they primarily output a scalar probability, offering limited visibility into the specific semantic or structural signals driving the decision.

This creates a gap where verification systems can predict correctness but cannot effectively support system analysis, diagnosis, or explainability by tracing decisions back to specific evidence.

Methodology: TraceSQL

The authors propose TraceSQL, a lightweight, learned verification model designed to be interpretable, explainable, and traceable. Instead of relying solely on a final score, TraceSQL constructs a fixed set of 67 explicit diagnostic features derived from five complementary families. These features capture signals regarding question ambiguity, planning, consistency, SQL structure, and intent alignment.

1. Diagnostic Evidence Generation

The system utilizes three upstream modules to generate raw diagnostic evidence:

Ambiguity Detector: Assesses if the user question is sufficiently specified relative to the database context, outputting status, probability, and explanations.
Pair Analyzer: Evaluates the consistency between the question, schema, and candidate SQL using predefined rules (e.g., schema grounding, joins, data types, temporal semantics).
SQL Repair Module: Decomposes the question into structured intents and evaluates the candidate SQL's alignment with these intents in a reference-free manner.

2. Feature Extraction Pipeline

The raw evidence is transformed into the 67-dimensional feature vector ( $z$ ) through three mechanisms:

LLM-based Probe Assessment: For ambiguity, question planning, and intent alignment, specific diagnostic units are evaluated by an LLM (GPT-4o) against predefined probes (e.g., "Metric Clarity," "Output Structure"). The LLM returns structured verdicts (PASS/FAIL/N/A) which are aggregated into features.
Direct Rule Projection: For Pair Analysis, the status of canonical verification rules (e.g., "Join-Key Support") is directly projected into the feature set without additional LLM inference.
Deterministic SQL-Structure Extraction: The candidate SQL is parsed into an Abstract Syntax Tree (AST) using SQLGlot. Ten binary features are extracted based on the presence of structural elements (e.g., DISTINCT, LIMIT, Aggregation, Window Functions).

3. Model Training and Inference

Training: TraceSQL is trained using the FLAML framework on a balanced dataset of 2,000 candidate SQLs (1,000 positive, 1,000 negative) derived from the BIRD ORM training corpus. The selected model is an Extra Trees classifier.
Inference: Given a question, schema, and candidate SQL, the system generates the 67 features and estimates the probability of correctness ( $\hat{p}$ ). A threshold of 0.50 determines the final verification decision.
Traceability: Crucially, the richer diagnostic evidence (explanations, rule statuses, AST nodes) is retained separately. This allows any influential feature to be traced back to its specific diagnostic provenance.

Key Contributions

TraceSQL Model: Introduction of a lightweight verifier that integrates SQL verification with interpretability, explainability, and traceability.
67-Feature Diagnostic Representation: Development of a semantically meaningful feature set derived from five diagnostic families. This representation retains provenance to underlying evidence, allowing for feature-level inspection.
Empirical Evaluation: A rigorous comparison against the GradeSQL-7B ORM baseline on identical held-out generated SQLs from BIRD development databases, demonstrating that verification decisions can be connected to explicit feature-level evidence.

Experimental Results

The authors evaluated TraceSQL and GradeSQL-7B on 1,521 generated SQL candidates from 11 held-out BIRD development databases.

Performance: TraceSQL outperformed the GradeSQL-7B baseline across all metrics:
- F1 Score: 66.47% (TraceSQL) vs. 61.87% (GradeSQL-7B).
- ROC-AUC: 64.48% (TraceSQL) vs. 58.26% (GradeSQL-7B).
Feature Importance: Analysis using permutation importance and SHAP values revealed that the model relies on a combination of deterministic SQL-structure signals (e.g., SQL DISTINCT Usage, SQL LIMIT Usage, SQL Aggregation Usage) and semantic diagnostic evidence (e.g., Additional Grounding Check, Pattern-Based Business Meaning Support).
Generalization: The model demonstrated effective transfer across diverse schemas and query distributions, achieving stronger results on nine of the eleven held-out databases.
Ground-Truth Verification: When tested on 1,534 ground-truth SQLs, TraceSQL accepted 60.17% of queries, slightly lower than GradeSQL-7B (62.84%), indicating that while TraceSQL is effective for candidate verification, its behavior on perfect queries differs slightly from the baseline.

Significance and Claims

The paper claims that TraceSQL addresses a critical limitation in current verification systems: the lack of visibility into why a query is deemed incorrect. By moving beyond scalar scores to a structured diagnostic representation, TraceSQL enables:

Interpretability: Model inputs are named features with explicit semantic meaning.
Explainability: Feature importance methods can identify which signals (semantic vs. structural) drive specific predictions.
Traceability: Decisions can be traced back to the specific diagnostic evidence (e.g., a specific rule failure or AST node) that influenced the model.

The authors conclude that reliable Text-to-SQL verification can be achieved using a lightweight learned model that combines multiple forms of diagnostic information while preserving the evidence necessary for inspection and failure analysis. They position TraceSQL not just as a performance improvement, but as a framework for making the verification process transparent and actionable.