🤖 AI

Agent-Agnostic Evaluation of SQL Accuracy in Production Text-to-SQL Systems

This paper introduces STEF, a schema-agnostic evaluation framework that enables continuous, production-native monitoring of Text-to-SQL systems by generating interpretable accuracy scores from natural language inputs and generated SQL without requiring database schemas or ground-truth queries.

Original authors: Taslim Jamal Arif, Kuldeep Singh

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Taslim Jamal Arif, Kuldeep Singh

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot assistant that speaks "Human" (like "Show me sales from last year") and translates it into "Database Language" (SQL code) to pull information from a company's records.

The big problem this paper tackles is: How do you know if the robot is doing a good job when it's working for real people in a real company?

The Old Way: The "Answer Key" Problem

In the past, to test these robots, researchers used a method like grading a math test with an answer key.

The Scenario: You give the robot a question, it gives an answer, and you compare it to a "Gold Standard" answer written by a human expert.
The Flaw: In the real world, you don't have an answer key for every single question a customer asks. Also, the company's database is often a secret, changing, or too complex to share with the testing team.
The Result: Once the robot is deployed in a real office, nobody knows if it's getting worse over time. It's like driving a car with a broken speedometer; you don't know if you're speeding until you crash.

The New Solution: STEF (The "Smart Translator" Inspector)

The authors created a new system called STEF (Schema-agnostic Text-to-SQL Evaluation Framework). Think of STEF as a super-intelligent editor who doesn't need the original book or the database to check if the translation is good.

Here is how STEF works, using simple analogies:

1. The "No-Reference" Rule

STEF doesn't need the "Gold Standard" answer or the database map. It only looks at three things:

What the human asked.
How the robot rephrased that question internally.
The code the robot wrote.
It's like a translator checking if a French sentence makes sense in English, without needing to know the original Spanish source text.

2. The "Deconstruction" (Taking the Soup Apart)

Instead of just comparing the code string-for-string (which is like checking if two sentences have the exact same letters), STEF breaks the request down into ingredients:

What are we looking for? (The columns)
What are we counting or adding? (The math)
What are we filtering out? (The "only show me active users" part)
How are we grouping it? (By country, by year, etc.)

STEF checks if the robot included the right ingredients. If the human asked for "Active Users" and the robot forgot to filter out "Inactive" ones, STEF spots the missing ingredient immediately.

3. The "Human" Judge (The LLM)

STEF uses another AI (a "Judge") to look at the ingredients and decide: "Does this code actually answer the question?"

The Confidence Score: The Judge doesn't just say "Yes" or "No." It says, "I'm 90% sure this is right," or "I'm only 50% sure."
The Penalty: If the Judge is unsure, the final score gets a little penalty. This stops the system from giving a perfect score to a guess.

4. The "Real-World" Rules (Normalization)

This is the cleverest part. In the real world, robots sometimes add extra steps that are actually helpful, not wrong. STEF knows this.

The "Sort" Rule: If a robot sorts the results from highest to lowest (even if the human didn't ask), STEF says, "That's a nice touch, not a mistake."
The "Safety" Rule: If a robot adds a limit like "Show me the top 10,000 results" just to prevent the computer from crashing, STEF says, "Good job, that's safe," instead of "Wrong, you didn't ask for 10,000."
The "Group" Rule: If the math requires grouping data to make sense, STEF accepts that even if the human didn't explicitly say "group by."

5. The "Company Customization" (The Rulebook)

Every company is different. One company might call a column "Region," while another calls it "Territory."
STEF has a configurable rulebook. You can tell STEF: "Hey, in this company, 'Region' and 'Territory' mean the same thing," or "Always ignore the 'Status' filter because it's automatically added." This lets STEF adapt to any company without needing to be reprogrammed.

The Final Score

STEF gives a score from 0 to 100.

90-100: Excellent. The robot is ready for prime time.
50-75: Marginal. The robot is guessing too much; humans should check it.
Below 50: Poor. The robot is failing and needs immediate help.

Why This Matters

Before STEF, companies were flying blind with their AI assistants. They couldn't tell if the AI was slowly getting dumber or if it was making dangerous mistakes. STEF acts like a continuous health monitor for these AI agents. It lets companies fix problems before they affect real business decisions, all without needing to peek inside the secret database or write new answer keys for every question.

1. Problem Statement

The paper addresses a critical gap in the deployment of Text-to-SQL (T2SQL) systems in production environments. While academic benchmarks (e.g., Spider, BIRD) rely on ground-truth SQL queries and fixed database schemas for evaluation, real-world production systems face different constraints:

No Ground Truth: User queries arrive dynamically without pre-annotated "gold standard" SQL.
Schema Dependency: Production databases are often proprietary, multi-tenant, or subject to frequent schema evolution, making static schema access a liability.
Evaluation Vacuum: Without reference queries or schema access, developers cannot systematically monitor accuracy, detect quality degradation, or compare agent versions after deployment. This leads to "silent quality erosion" where errors go undetected.

Existing methods (string matching or execution-based accuracy) fail in this context because they require reference queries or a live, queryable database snapshot, which are often unavailable or impractical in enterprise settings.

2. Methodology: The STEF Framework

The authors propose STEF (Schema-agnostic Text-to-SQL Evaluation Framework), a production-native system that evaluates T2SQL agents using only natural language inputs and the generated SQL, without needing database schemas or reference queries.

The framework operates through a five-stage pipeline:

A. Input Specification

STEF accepts four inputs per query instance:

$Q_u$ : The original user question.
$Q_e$ : The agent's enriched/reformulated question (pre-processing output).
$S$ : The generated SQL query.
$R_{app}$ : A JSON configuration for application-specific rules (e.g., column mappings, benign filters).

B. Semantic Feature Extraction & Normalization

The system extracts structured semantic specifications from both the natural language ( $Q_u, Q_e$ ) and the SQL ( $S$ ) independently.

Dimensions: It decomposes queries into Projections (columns), Aggregations (SUM, AVG), Filters (WHERE clauses), and Grouping (GROUP BY).
Normalization: To handle surface-form variations, STEF normalizes data during extraction:
- Resolves SQL aliases to semantic meanings.
- Normalizes filter values (case, whitespace).
- Maps natural language terms (e.g., "total") to SQL functions (e.g., SUM).
- Applies application-specific column mappings from $R_{app}$ .

C. Specification Alignment & Filter Classification

The extracted features are aligned to determine the Filter Alignment Status. The system classifies the alignment into four categories:

Fully Applied: All required filters present, no extras.
Fully Applied with Extras: Required filters present, but extra filters exist (checked against "benign" rules).
Partially Applied: Some filters missing or incorrect.
Not Applied: Critical filters missing.

D. LLM-Based Semantic Verdict

An LLM evaluator acts as a "judge" to assess holistic semantic alignment that rule-based logic cannot capture (e.g., complex date arithmetic, implicit metric equivalences).

Verdict Scale: The LLM outputs a 4-point ordinal score: Correct, Likely Correct, Potentially Incorrect, Incorrect.
Confidence Scoring: The LLM provides a confidence score ( $\gamma \in [0, 1]$ ) to quantify its certainty, which is crucial for handling evaluator variance.

E. Production Normalization Rules

To prevent false negatives on structurally correct but idiomatically different SQL, STEF applies four universal normalization rules:

Required GROUP BY Inference: If a SELECT mixes aggregates and non-aggregates, the necessary GROUP BY is assumed correct even if not explicitly requested.
Benign GROUP BY: Grouping on a column already constrained to a single value by a WHERE clause is ignored.
ORDER BY Tolerance: Default sorting (e.g., by aggregate metric descending) is accepted if the user didn't explicitly request a sort order.
LIMIT Tolerance: High-cardinality limits (e.g., LIMIT 10000) used as safety guards are not penalized as top-k constraints unless explicitly requested.

3. Composite Scoring Mechanism

STEF produces a single interpretable score ( $\Phi$ ) on a 0–100 scale. The score is calculated as:
$\Phi = \frac{B}{10} \times 100 \times \Gamma$
Where:

$B$ (Base Score): Sum of the Filter Alignment Score (0–5) and Semantic Verdict Score (0–5), plus a Leniency Boost (+1) if extra filters are confirmed benign.
$\Gamma$ (Confidence Multiplier): A penalty factor based on the LLM's confidence ( $\gamma$ $γ$ ).
- $\gamma \ge 0.85 \rightarrow \Gamma = 1.0$
- $0.65 \le \gamma < 0.85 \rightarrow \Gamma = 0.8$
- $\gamma < 0.65 \rightarrow \Gamma = 0.5$
- Significance: This transforms evaluator uncertainty into a scoring signal, penalizing low-confidence assessments to prevent false positives.

4. Key Contributions

Schema-Agnostic Evaluation: The first framework to evaluate T2SQL accuracy in production without requiring database schemas, reference queries, or execution infrastructure.
Enriched Question Validation: Treats the agent's internal reformulation of the user question as a first-class signal for evaluation.
Configurable Rule Injection: Allows enterprises to inject deployment-specific logic (column mappings, benign filters) via prompt templating without code changes or model retraining.
Production-Robust Normalization: Explicitly handles common production SQL idioms (GROUP BY inference, default ORDER BY, safety LIMITs) to reduce false negatives.
Confidence-Calibrated Scoring: Integrates LLM uncertainty directly into the final accuracy score, providing a more reliable metric for continuous monitoring.

5. Results

Empirical results (Section 7) demonstrate the framework's viability across three production agents:

Agent-A: Mean Score 87.4, P90 Score 96.2, Coverage 98.7%.
Agent-B: Mean Score 82.1, P90 Score 93.5, Coverage 97.2%.
Agent-C: Mean Score 91.3, P90 Score 98.1, Coverage 99.1%.
The high coverage and scores indicate that STEF can effectively monitor and differentiate agent performance in a live environment.

6. Significance

This work bridges the gap between academic T2SQL research and industrial reality. By removing the dependency on ground-truth data and static schemas, STEF enables:

Continuous Improvement: Organizations can now monitor agent quality in real-time and identify regression.
Agent Comparison: Different model versions or agents can be objectively compared based on their output quality alone.
Scalability: The framework is deployable across heterogeneous database technologies and multi-tenant environments without modification.

Limitations: The authors note that residual LLM evaluator variance persists on borderline cases and that deeply nested subqueries or complex window functions remain challenging, suggesting a future direction toward hybrid symbolic-neural parsing.