🤖 machine learning

What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study

This study demonstrates that while basic self-consistency and log-probability signals struggle to predict Text-to-SQL correctness beyond a 0.68 AUROC ceiling, verification-based approaches—particularly ensembles of large language model judges—achieve superior performance (up to 0.82 AUROC) and robust generalization across schemas, whereas fine-tuned verifiers fail to transfer effectively to unseen schemas.

Original authors: Robert Richardson

Published 2026-07-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Robert Richardson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but sometimes overconfident, robot assistant. You ask it a question in plain English, and it writes a complex computer command (called a SQL query) to find the answer in a giant database.

The big problem is: How do you know if the robot is right or just making things up?

This paper is like a detective story investigating different "lie detectors" to see which one can actually tell if the robot's answer is correct. The researchers tested these lie detectors on two very difficult puzzles (datasets called BIRD and Spider) where the robot often gets things wrong.

Here is what they found, explained through simple analogies:

1. The "Echo Chamber" Lie Detector (Self-Consistency)

The Idea: One way to check the robot is to ask it the same question 8 times. If it gives you the exact same answer 8 times, you assume it must be right. It's like asking a friend the same question repeatedly; if they say the same thing every time, you trust them.
The Result: This didn't work well. The robot was very good at being consistent, even when it was wrong. It would confidently repeat the same wrong answer 8 times.
The Score: It was only slightly better than flipping a coin (about 67% accurate at spotting the truth).

2. The "Grammar Police" Lie Detector (Log-Probability)

The Idea: This checks how "smooth" the robot's sentence feels. If the robot says something that sounds very natural and grammatically perfect, maybe it's right.
The Result: Also didn't work well. The robot could write a perfectly smooth sentence that was completely nonsense.
The Score: Similar to the echo chamber, it hit a "ceiling" and couldn't get much better.

3. The "Expert Judge" Lie Detector (Verification)

The Idea: Instead of asking the robot to check itself, you show the question, the database rules, and the robot's answer to a different, super-smart AI (a "Judge"). You ask the Judge: "Does this answer actually solve the problem?"
The Result: This was the winner. The Judge didn't just look for repetition; it actually read the logic. It checked if the math was right, if the conditions matched the question, and if the grouping made sense.
The Score: This jumped the ceiling, reaching about 77–78% accuracy.

4. The "Two-Headed" Lie Detector (Ensemble)

The Idea: The researchers realized that even smart Judges make different mistakes. So, they used two different Judges (one from OpenAI and one from Anthropic) and asked them both. If they both agree, you trust the answer even more.
The Result: This was the best method. Because the two Judges made different errors, combining them covered each other's blind spots.
The Score: This reached 82% accuracy and was very reliable. It was the only method that could safely say, "I'm not sure, I'll skip this question," without skipping too many easy ones.

5. The "Student" vs. The "Professor" (Training Verifiers)

The Idea: Can we train a cheaper, smaller AI to be the Judge? They tried teaching a small AI by showing it thousands of examples of right and wrong answers.
The Result:

In the Classroom (Same Database): The student did great! If the database was the same one it studied, it was almost as good as the Professor.
On the Real World (New Database): When they gave the student a new database it had never seen, it failed miserably. It had just memorized the patterns of the old database rather than learning how to reason.
The Professor (Frozen Model): The big, pre-trained "Professor" AI (which wasn't fine-tuned on specific data) was the only one that could handle new, unseen databases effectively.

The Big Takeaway

The paper concludes that for hard computer tasks, repetition isn't proof of correctness. Just because a robot says the same thing twice doesn't mean it's right.

To know if an AI is telling the truth, you need a reasoning expert who actually understands the logic of the question and the answer.

If you are working on a fixed, known system, a trained "student" verifier is cheap and effective.
If you are working in the real world with new, unknown data, you need a powerful, "frozen" reasoning model (a Professor) to act as the judge.

The paper also notes that simply asking the robot to "fix its own work" (self-correction) didn't help much; it just made the robot more confident in its mistakes. You still need an outside judge to decide if the final answer is safe to use.

Technical Summary: What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study

Problem Definition

The paper addresses the problem of selective prediction in text-to-SQL systems: determining when a generated SQL query is correct enough to be executed and when the system should abstain. Correctness is defined in the standard benchmark sense: the generated query must return the same result set as a human-written reference query when executed against the database. The goal is to estimate the probability of correctness ( $P(\text{correct})$ ) to enable a system to choose between answering or declining, rather than iteratively improving the query (self-correction).

Methodology

The study evaluates various signals for predicting correctness on "hard" multi-table text-to-SQL tasks using two benchmarks: BIRD (800 questions, 8 databases) and Spider (490 questions, 20 databases).

Signals Evaluated

The authors categorize signals into two groups:

Black-box Statistical Signals: These rely on the generator's output statistics without external reasoning.
- Self-consistency: String (identical queries), Structural (canonical structure), and Execution (result set) clustering.
- Schema Relevance: Embedding similarity between the question and the tables/columns used.
- Executability: Whether the query runs and returns rows.
- White-box Log-Probability: The mean sequence log-probability of the generated query.
Verification-Based Signals: These involve a separate model (LLM judge) inspecting the question, schema, evidence, and candidate SQL to judge correctness.
- Judges: GPT-4o-mini, GPT-4o, and Claude-Sonnet-4.6.
- Ensemble: A combination of GPT-4o and Claude scores.
- Trained Verifiers: Fine-tuned encoder and generative models (Qwen2.5-1.5B/7B) on execution-labeled pairs.

Evaluation Metrics

AUROC: Area Under the Receiver Operating Characteristic curve, measuring the ability to rank correct queries above incorrect ones.
Calibration: Expected Calibration Error (ECE).
Selective Prediction: Risk-coverage frontiers (error rate among answered questions vs. fraction of questions answered).
Statistical Significance: Paired bootstrap confidence intervals (2,000 samples) to determine if differences between signals are significant.

Key Results

1. The Black-Box Ceiling

Black-box statistical signals and white-box log-probability plateau in a narrow performance band, failing to reliably distinguish correct from incorrect queries on hard multi-table tasks.

Performance Range: AUROC between 0.61 and 0.68.
Strongest Black-Box Signal: String self-consistency (AUROC 0.675).
Log-Probability: Similar to self-consistency (0.669), with no significant difference.
Executability: Essentially random chance (0.500), as incorrect queries often execute successfully.
Conclusion: These signals measure internal consistency or schema alignment but fail to detect logical errors in computation or composition.

2. Verification Breaks the Ceiling

Verification-based signals significantly outperform black-box baselines by reasoning about whether the SQL logic matches the question.

Single Judges:
- GPT-4o-mini: 0.724 AUROC.
- GPT-4o: 0.770 AUROC.
- Claude-Sonnet-4.6: 0.776 AUROC.
Ensemble: Combining GPT-4o and Claude (which make different errors, correlation $r=0.43$ ) yields the best signal: 0.822 AUROC with excellent calibration (ECE 0.031).
Robustness: This pattern holds across different generators (GPT-4o-mini vs. GPT-4.1-mini) and benchmarks (BIRD and Spider).

3. Trained Verifiers: In-Distribution vs. Transfer

The study investigates whether a cheaper, fine-tuned verifier can replace the frozen LLM judge.

In-Distribution: Fine-tuned models (encoders and generative judges) perform well on seen schemas, reaching 0.77–0.80 AUROC.
Cross-Schema Transfer: When tested on unseen databases (Leave-One-Database-Out), performance drops significantly to ~0.66 AUROC.
Scaling & Distillation: Increasing model size (1.5B to 7B), adding schema diversity, distilling rationales, or cross-benchmark training failed to close the transfer gap.
Conclusion: Fine-tuned verifiers capture schema-specific surface regularities rather than robust cross-schema reasoning. For open/cross-schema settings, a large frozen reasoning model remains superior.

4. Selective Prediction and Abstention

Self-Consistency: Cannot form a valid low-risk subset; even its most confident predictions are too error-prone for safe abstention.
Verification Ensemble: Enables useful abstention. For example, at a target risk of 0.30, the ensemble answers 27% of questions with a held-out selective risk of 0.24.
Self-Correction Baseline: A single round of self-correction barely improves accuracy (0.451 to 0.468) and produces severely miscalibrated confidence (mean 0.97, ECE 0.501), confirming that regeneration does not replace the need for a calibrated verifier.

5. Error Analysis

The paper identifies that the primary failure modes in hard text-to-SQL are computation and composition (ratios, percentages, conditional logic, grouping, nesting), not schema linking or join count.

Schema relevance scores are weak because generators often identify the correct tables but fail on the logic.
Verifiers excel because they evaluate the computation expressed by the SQL against the question.
The verifier's advantage is most pronounced on arithmetic questions (AUROC 0.813) and simple queries where self-consistency is often confidently wrong.

Significance and Claims

The paper claims that for hard text-to-SQL tasks:

Correctness uncertainty is real but not captured by cheap signals. Sampling agreement, structural consistency, and log-probability are insufficient for high-stakes deployment.
Reasoning is required. The only signals that break the performance ceiling are those that explicitly reason about the query's logic (verification).
Deployment regimes differ.
- Fixed/In-Domain: A fine-tuned verifier is a cost-effective and accurate tool for abstention.
- Open/Cross-Schema: A large, frozen reasoning model (LLM judge) is currently necessary for reliable generalization.
Ensembles improve reliability. Combining independent judges (different providers) yields the best-calibrated and most discriminative signal, suggesting that verification uncertainty is not an artifact of a single model family.
Self-correction is not a substitute. Even systems that regenerate queries need a calibrated verifier at the end of the pipeline to decide whether to trust the final output.

The study concludes that if a system must decide whether to trust generated SQL, it must invest in a logic-aware verifier and calibrate it for selective prediction, rather than relying on sampling confidence.