Technical Summary: BEAVER: An Enterprise Benchmark for Text-to-SQL

Problem Statement

Existing text-to-SQL benchmarks (e.g., Spider, BIRD) are constructed from public databases with well-structured schemas, explicit join relationships, and relatively simple queries. While Large Language Models (LLMs) achieve high accuracy on these benchmarks, their efficacy in private enterprise environments remains unproven. Enterprise data warehouses present distinct challenges:

• Schema Complexity: Schemas are large, heterogeneous, and continuously evolving, often containing hundreds of tables with cryptic column names and limited documentation.
• Implicit Relationships: Join relationships are frequently implicit rather than declared via foreign keys.
• Domain Knowledge: Resolving natural language queries often requires extracting domain-specific knowledge (e.g., mapping "Stata building" to an internal ID) that is not explicitly encoded in the schema.
• Query Sophistication: Enterprise queries are highly analytical, involving deep nesting, Common Table Expressions (CTEs), advanced window functions, and multiple joins.
• Evaluation Limitations: Standard "execution accuracy" (all-or-nothing) metrics fail to diagnose specific failure modes when multiple compounded challenges exist simultaneously.

Methodology

1. Dataset Construction (BEAVER)

The authors introduce BEAVER, the first text-to-SQL benchmark derived from private enterprise data warehouses.

• Source Data: The dataset is built from three anonymized enterprise warehouses (DW, NW, SP) covering 19 diverse domains (e.g., education, facilities, networking). It includes 812 tables and 9,128 question-SQL pairs sourced from real user query logs and production reports.
• Data Synthesis (Structural Template Recomposition): To overcome the scarcity of accessible enterprise logs due to privacy constraints, the authors propose a systematic pipeline:
  1. Template Extraction: Recurring analytical patterns (Base SELECT-FROM-JOIN-WHERE-AGGREGATION, Nesting, CTEs, and Domain Knowledge predicates) are extracted from real logs.
  2. Iterative Composition: These templates are instantiated with concrete values (tables, columns, instances) and iteratively composed using LLMs.
  3. Challenge Isolation: The pipeline generates queries that isolate specific challenges: high complexity without domain knowledge, low complexity with domain knowledge, and combinations of both.
  4. Verification: All synthesized queries undergo expert verification for executability and correctness.

2. Evaluation Framework

BEAVER moves beyond binary execution accuracy by introducing five critical subtasks with human annotations and specific metrics:

1. Multi-table Retrieval: Identifying the correct subset of tables (F1 score).
2. Join Key Detection: Inferring implicit join conditions between tables (F1 score).
3. Column Mapping: Mapping natural language concepts to obscure column names (F1 score).
4. Domain Knowledge Extraction: Identifying instance values and domain facts not in the final SQL (F1 score).
5. Query Decomposition: Breaking complex questions into sub-problems (CTEs/Sub-queries) (Score 0-100).

Key Contributions

1. BEAVER Benchmark: A large-scale dataset (9,128 queries) reflecting the "messiness" of private enterprise environments, featuring schemas averaging 101.5 tables and 869.4 columns per database, significantly larger than existing benchmarks like Spider 2.0 or BIRD.
2. Structural Template Recomposition: A principled method for expanding enterprise datasets under strict privacy constraints while maintaining high fidelity to real-world workloads.
3. Fine-Grained Evaluation: A framework providing human annotations for five subtasks, enabling diagnostic analysis of model failures beyond coarse execution accuracy.
4. Comprehensive Error Taxonomy: A detailed analysis of residual errors, categorizing failures into analytical/structural errors, schema linking errors, and predicate errors.

Experimental Results

The authors evaluated seven recent LLMs (including GPT-5.2, Claude-4.5, Qwen3) and four text-to-SQL methods (ReFoRCE, DIN-SQL, DAIL-SQL, Few-shot) on BEAVER.

• Performance Gap: There is a significant drop in performance compared to existing benchmarks. The state-of-the-art agentic framework (ReFoRCE) using GPT-5.2 achieved only 10.8% execution accuracy on BEAVER, compared to ~80% on BIRD/Spider.
• Subtask Analysis:
  • Models performed worst on Join Key Detection, Domain Knowledge, and Query Decomposition.
  • Join key detection is particularly difficult due to the lack of explicit foreign keys and cryptic naming.
• Oracle Hints: When provided with annotations for all five subtasks as "oracle hints," execution accuracy improved to 30.1%. This confirms that correctly resolving these subtasks is a major bottleneck, though significant residual errors remain.
• Query Categories: Models struggled most with queries combining high structural complexity and domain knowledge. Purely complex queries (without domain knowledge) were slightly easier, while domain-specific queries with low complexity were the most manageable.

Error Analysis

Even with oracle hints for subtasks, residual errors persist, categorized as:

• Analytical & Structural Errors (46.8%): Mistakes in grouping, ordering, advanced functions (window, scalar, aggregate), and query decomposition (e.g., incorrect use of ROLLUP).
• Schema Linking Errors (35.2%): Incorrect table/column selection and join types, often due to opaque naming conventions.
• Predicate Errors (18.0%): Incorrect literal values or omitted filters, reflecting a failure to apply domain-specific conventions.

Significance

The paper positions BEAVER as a critical step toward developing robust, enterprise-ready SQL agents. It demonstrates that current text-to-SQL methods, while successful on simplified public benchmarks, are not yet capable of handling the intricate schemas, implicit domain knowledge, and sophisticated analytical constructs required in real-world private enterprise settings. By providing a challenging dataset and a fine-grained diagnostic framework, BEAVER identifies specific bottlenecks (particularly in subtask resolution and advanced SQL construction) that must be addressed to advance the field.