Technical Summary: EntSQL – A Benchmark for Grounding Text-to-SQL in Long-Context Enterprise Knowledge

1. Problem Definition

While Large Language Models (LLMs) have advanced Text-to-SQL capabilities, existing benchmarks (e.g., Spider, BIRD, Spider 2.0) primarily evaluate schema generalization, large-scale database handling, and workflows relying on public documentation. These benchmarks largely overlook enterprise-specific scenarios where SQL generation depends on private business knowledge.

In realistic enterprise Business Intelligence (BI) settings, answering a query often requires interpreting internal metric definitions, reporting conventions, fiscal-year rules, and organizational mappings that are not recoverable from public knowledge or the database schema alone. The core challenge is not merely generating syntactically valid SQL, but grounding SQL construction in long-context, proprietary enterprise documents.

2. Methodology: The EntSQL Benchmark

To address this gap, the authors introduce EntSQL, an enterprise-oriented benchmark designed to evaluate whether models can ground SQL generation in long-form private business knowledge.

Data Construction

The benchmark was constructed from real enterprise operational data through a four-stage pipeline:

1. Data Collection: Real user Q&A logs were collected from five core business domains: Treasury, Finance, Business Management, Party Building (Union), and Human Resources.
2. Context Synthesis & Expert Annotation: Domain experts annotated each instance with:
   • Gold SQL: Reflecting intended business logic.
   • Evidence Snippets: Concise identification of key business rules.
   • Domain Documents: Synthesized long-form documents describing the relevant business context.
3. Anonymization & Translation: Sensitive entities (dates, monetary values, org units) were replaced with curated mappings to preserve structure while protecting privacy. Original Chinese materials were translated and validated into English, creating aligned bilingual inputs.
4. Quality Assurance: Multi-stage manual review by experts and specialists ensured consistency across questions, SQL, evidence, and documents.

Benchmark Composition

• Scale: 1,066 aligned Chinese-English semantic examples.
• Complexity: 96.0% of examples require external domain knowledge beyond the question and schema. Gold SQL queries average 388.7 tokens, frequently utilizing Common Table Expressions (CTEs), joins, and nested subqueries.
• Difficulty Levels: Categorized by SQL token length: Easy (<200), Medium (200–400), and Hard (>400). Most queries fall into Medium and Hard categories.

Experimental Setup

• Models Evaluated: Eight systems, including seven standalone LLMs (e.g., Claude Opus/Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro, Qwen 3.6 Max, Kimi K2.6, GLM 5.1) and one interactive coding agent (Claude Code).
• Input Settings:
  • Base: Question + Schema.
  • w/Doc: Question + Schema + Long-form Domain Document.
  • w/Evidence: Question + Schema + Concise Expert-curated Evidence.
• Metric: Execution Accuracy (EX), comparing the result set of the predicted SQL against the gold SQL on the target database.

3. Key Results

The evaluation reveals significant challenges for current state-of-the-art systems in enterprise contexts:

• Low Overall Performance: Even with long-form documents provided, the best-performing system (Claude Code) achieved only 15.9% execution accuracy on the full English benchmark.
• Complexity Impact: Performance drops sharply as SQL complexity increases. In the "Base" setting (Question only), the best accuracy on Hard queries was merely 0.8%.
• The Value of Evidence: Providing concise, expert-curated evidence significantly outperforms providing full long-form documents.
  • w/Doc: Best Avg Accuracy = 15.9%.
  • w/Evidence: Best Avg Accuracy = 21.4%.
  • This indicates that while enterprise knowledge is useful, current models struggle to identify and apply relevant information from long documents without explicit localization.
• Expert Gap: A comparison with human experts on a sampled subset showed a substantial gap. Experts achieved 84.0% accuracy with evidence, compared to 20.3% for the best agent (Claude Code), highlighting limitations in business-rule grounding and multi-step composition.
• Domain Variance: Performance is highly skewed. Human Resources queries saw higher accuracy (up to 30.4% with evidence), while Finance queries remained near zero, suggesting difficulty is driven by domain-specific query patterns and rule complexity, not just SQL length.
• Error Analysis: The dominant failure mode (54.6%) was Wrong Filter, where models failed to correctly ground filtering constraints from the question and documents. Other major errors included Wrong Scope (14.4%) and Wrong Aggregation (10.0%).

4. Significance and Claims

The paper claims that EntSQL complements existing benchmarks by shifting the focus from schema generalization to enterprise long-context grounding.

• Distinct Practical Gap: The benchmark exposes a specific gap where faithfully grounding SQL generation in private enterprise knowledge remains a substantial challenge, distinct from the difficulties of schema matching or short-context reasoning.
• Current Limitations: The results suggest that simply increasing context length is insufficient; the key bottleneck is effective knowledge selection and grounding. Current agents struggle to translate localized business rules into executable SQL, even when provided with relevant evidence.
• Future Directions: The authors propose that future research should focus on explicit intermediate representations for query planning, stronger grounding to domain-specific rules, and post-generation validation modules to check schema usage and document constraints.

The authors maintain a modest stance, noting that while EntSQL is constructed from real workloads, it covers only five domains from a single enterprise setting and may not represent all enterprise Text-to-SQL scenarios. They emphasize that the benchmark is designed to support research into more reliable, knowledge-grounded systems for realistic enterprise scenarios.