Technical Summary: PRACTIQ

Problem Statement

Existing text-to-SQL datasets and systems primarily focus on user questions with clear, unambiguous intentions that can be answered using the provided database schema. However, real-world interactions frequently involve queries that are ambiguous (having multiple valid interpretations given the schema) or unanswerable (lacking relevant data in the database). Current state-of-the-art (SoTA) systems, while performing well on clean benchmarks like Spider and BIRD, struggle with these practical scenarios. This performance gap is largely attributed to the unavailability of practical training data that encompasses ambiguous and unanswerable queries, as previous datasets often exclude such cases or lack fine-grained categorization and conversational context.

Methodology

The authors introduce PRACTIQ, a practical conversational text-to-SQL dataset designed to address these gaps. The construction of PRACTIQ involves a three-stage framework applied to the Spider dataset (specifically train_spider.json and dev.json), leveraging Large Language Models (LLMs) for data generation and refinement:

1. Question Categorization & Database Modification:
   The authors identified eight distinct categories inspired by real-world user questions:

   • Ambiguous Categories: Ambiguous SELECT Column, Ambiguous WHERE Columns, Ambiguous Values Within Column, and Ambiguous Filter Criteria.
   • Unanswerable Categories: Nonexistent SELECT Column, Nonexistent WHERE Column, Nonexistent Filter Value, and Unsupported Join.
   • Procedure: Instead of modifying user questions, the framework programmatically modifies the database schemas (e.g., removing columns, adding semantically similar but distinct columns, or removing specific cell values) to render originally answerable queries ambiguous or unanswerable.

2. Conversational Data Generation (Reverse Generation):
   The dataset is structured as four-turn conversations:

   • Turn 1: Initial user question.
   • Turn 2: Assistant response seeking clarification (or providing a helpful multi-faceted SQL response for certain ambiguous types).
   • Turn 3: User clarification response.
   • Turn 4: Assistant's clarified SQL response, SQL execution results, and a natural language explanation of the results.
   • Technique: The authors employ a "reverse generation" approach. They first generate the final clarified SQL and execution results programmatically, then use an LLM to generate the user's clarification response that logically leads to that specific SQL. This ensures the SQL is executable and accurate.

3. Refinement & Quality Control:

   • Conversation Refinement: A 3-shot prompt is used to improve the naturalness and coherence of the dialogue and to add natural language explanations for the execution results.
   • Filtering: An LLM-based binary classifier verifies that each generated question-database pair strictly adheres to its intended category. Execution checks are performed to discard non-executable SQLs.

Key Contributions

• Dataset Construction: The creation of PRACTIQ, a comprehensive dataset containing 2,812 conversations (1,802 ambiguous/unanswerable and 1,034 answerable). It covers eight fine-grained categories of ambiguity and unanswerability, significantly expanding the scope beyond previous works like CoSQL or AmbiQT.
• Conversational Format: Unlike static datasets, PRACTIQ models the resolution process, including assistant clarification questions, user responses, and natural language explanations of SQL results. It also includes cases where the assistant provides helpful SQL responses covering multiple interpretations without explicit clarification.
• Benchmarking: The authors established baselines for two sub-tasks:
  1. Question Category Classification: Classifying queries into one of the nine categories (8 ambiguous/unanswerable + 1 answerable).
  2. Clarification SQL Prediction: Generating the correct SQL after the clarification turn.
     Experiments were conducted using various SoTA LLMs (including Claude 3.5 Sonnet, Llama-3.1, and Mixtral) and the DIN-SQL framework.

Results

• Human Annotation: Human experts (SQL specialists) evaluated the dataset. The binary classification accuracy for category correctness was 93.75%, and the 9-way classification accuracy was 88.33%. Quality metrics for naturalness, factuality, and helpfulness were high (mean scores close to 1 on a 1-5 scale, where 1 is best).
• Model Performance:
  • Classification: Even with Oracle cell values (perfect retrieval), the best model (Claude 3.5 Sonnet) achieved only 77.4% accuracy in 9-way classification. Without Oracle values, accuracy dropped to 74.3%. This indicates that detecting fine-grained ambiguity remains a significant challenge for current LLMs.
  • SQL Prediction: On the clarification SQL task, Mixtral-large-v2 and Claude 3.5 Sonnet achieved the highest average execution accuracy on ambiguous/unanswerable questions (~72%). However, performance varied significantly across categories, with some open-source models (e.g., Llama-3.1 70B) lagging behind by up to 9% on specific ambiguous categories compared to the best models.
  • Baseline Failure: When tested on ambiguous/unanswerable queries without clarification (a standard DIN-SQL setup), the framework frequently hallucinated non-existent columns or generated incorrect SQLs, confirming that standard systems are not equipped for these queries.

Significance and Claims

The paper claims to be the first work to study text-to-SQL systems in a conversational setting where user queries can be answerable, ambiguous, or unanswerable simultaneously. The authors emphasize that:

• Current SoTA systems are not well-equipped for real-world data containing ambiguity or missing information.
• The gap in performance between answerable and ambiguous/unanswerable queries highlights a critical area for improvement in LLM-based text-to-SQL systems.
• The proposed framework provides a scalable method to generate synthetic practical data, which can be used to fine-tune open-source models to better handle real-world text-to-SQL tasks.
• The dataset serves as a benchmark to reveal that while LLMs are approaching human-level performance on clean data, they still struggle with the nuances of ambiguity and unanswerability found in practical applications.

The authors maintain a modest stance, acknowledging that their data generation relies on programmatic methods and LLMs, and that further improvements could be achieved through agentic workflows or fine-tuning open-source models on this specific data, which they leave as future work.