Technical Summary: UNIQL

Problem Statement

Current text-to-SQL benchmarks, such as Spider and BIRD, are predominantly centered on the SQLite dialect. This reliance creates a significant evaluation gap: it remains unclear whether large language models (LLMs) can generalize their capabilities across heterogeneous SQL dialects. Real-world database systems (e.g., PostgreSQL, MySQL, Oracle, Hive) differ substantially in syntax, built-in functions, type systems, and execution semantics. Consequently, the same natural language intent often requires dialect-specific SQL realizations. Existing benchmarks either focus on a single dialect or cover multiple systems without aligning the same natural language intents across them, making it impossible to isolate dialect variation from differences in schemas or tasks. This limitation leads to an overestimation of the robustness and universality of current text-to-SQL systems.

Methodology

The authors introduce UNIQL, a human-verified, executable benchmark designed for cross-dialect text-to-SQL evaluation. The construction of UNIQL involves a hybrid pipeline that extends the BIRD development set to 16 SQL dialects.

Dataset Construction Pipeline

1. Database Migration: The original BIRD databases (SQLite-based) are migrated to 15 target database systems (including ClickHouse, Doris, Drill, Druid, DuckDB, Hive, MySQL, Oracle, PostgreSQL, Presto, Spark, StarRocks, Teradata, Trino, and T-SQL). This process involves schema adaptation, type mapping, identifier normalization, and namespace adaptation to ensure executability while preserving data semantics.
2. Automated Translation & Verification:
   • Rule-Based Translation: A deterministic tool (SQLGlot) is used to generate initial target dialect queries.
   • Execution-Guided Verification: Translated queries are executed on the target database. A query is automatically accepted only if it executes successfully and produces results strictly equivalent to the source SQLite query. The protocol preserves ordering semantics and duplicate multiplicities, avoiding the false positives common in unordered set comparisons.
   • LLM-Based Translation: For queries failing rule-based translation, an LLM (GPT-5-mini) generates translations conditioned on the source SQL, target schema, and current rule set.
   • Self-Reflection: If LLM translations fail equivalence verification, the model enters a bounded self-reflection loop (up to 3 rounds) using execution feedback (errors or result mismatches) to refine the query.
3. Iterative Rule Evolution: Failed translations are analyzed to abstract recurring error patterns into reusable dialect transformation rules. These rules are updated iteratively (using Gemini-2.5-Pro) to handle long-tail dialect-specific issues such as function rewriting, type conversion, and aggregation behavior.
4. Human Verification: Cases that cannot be resolved through automated translation, reflection, or rule refinement are routed to human annotators. Two independent annotators review executability and semantic consistency, rewriting queries as necessary to ensure they match the original natural language intent.

Benchmark Statistics

• Source: 1,534 natural language questions from the BIRD development set.
• Coverage: 16 SQL dialects (1 source + 15 targets).
• Total Annotations: 24,544 dialect-specific, executable SQL queries.
• Alignment: All dialects share identical natural language intents, aligned schemas, and database contents.

Evaluation Protocol

The paper evaluates models using Execution Accuracy (EX). Unlike prior benchmarks that convert outputs to unordered sets, UNIQL's protocol preserves ordering when explicit ORDER BY clauses exist and maintains duplicate multiplicities for unordered outputs. Models are evaluated in an inference-only setting without task-specific fine-tuning, few-shot selection, or dialect-specific adaptation.

Key Results

The authors evaluated a broad set of open-source (e.g., Qwen3, Llama-3, DeepSeek) and closed-source (e.g., GPT-5, Gemini, Claude) LLMs.

1. Limited Dialect Universality: Even the strongest models (e.g., Claude-4.5-Sonnet) achieve an average execution accuracy of only ~54.63%. This indicates that executable cross-dialect text-to-SQL generation is far from saturated.
2. SQLite-Centric Bias: Performance on SQLite does not reliably predict performance on other dialects. For instance, a model might achieve 59.84% on SQLite but only 37.74% on Teradata. Single-dialect benchmarks fail to capture these significant model-dialect interactions.
3. Scaling Effects: Larger models within families (e.g., Qwen3-1.7B to Qwen3-32B) show improved performance, but scaling primarily boosts overall capability rather than eliminating dialect-specific weaknesses. Difficult dialects like Druid, Presto, and Teradata remain challenging even for larger models.
4. Dialect Familiarity: Performance correlates with model familiarity. Oracle, which has abundant public documentation and training exposure, yields higher accuracy than Teradata or Druid, which may be underrepresented in pretraining corpora and involve complex system-specific functions.
5. Cross-Dialect Consistency: High average accuracy does not imply dialect-universal correctness. For the best model, only ~20% of questions are answered correctly across all 16 dialects. Furthermore, success on the SQLite realization of an intent does not guarantee success on other dialects (e.g., only ~33% of SQLite-correct questions remain correct across all other dialects for Claude-4.5-Sonnet).

Significance and Contributions

The paper makes three primary contributions:

1. UNIQL Benchmark: The introduction of a human-verified, executable benchmark that aligns the same natural language intents with SQL realizations across 16 dialects, enabling controlled evaluation of dialect generalization.
2. Construction Framework: A novel SQL translation framework integrating tool-based translation, LLM-based translation, execution-based verification, self-reflection, iterative rule evolution, and human validation to construct high-quality cross-dialect datasets.
3. Diagnostic Evaluation: Comprehensive evaluation showing that current LLMs struggle with dialect-universal text-to-SQL generation. The findings highlight that robust cross-system querying requires explicit evaluation and modeling of cross-dialect robustness, rather than relying solely on SQLite-centric benchmarks.

The authors conclude that while current models show promise, they are not yet dialect-universal. The substantial variation in performance across database systems underscores the need for aligned cross-dialect benchmarks and the development of more dialect-aware text-to-SQL methods.