Technical Summary: Business Logic-Driven Text-to-SQL Data Synthesis for Business Intelligence

Problem Statement

Evaluating Text-to-SQL agents within private Business Intelligence (BI) environments is hindered by the scarcity of realistic, domain-specific data. While synthetic data generation offers a scalable alternative to manual annotation, existing methods (e.g., OmniSQL, SQL-Factory) primarily optimize for broad schema coverage and syntactic SQL diversity. Consequently, these methods fail to capture business realism: the generated questions often lack the context of real-world workflows, job roles, and business logic. This disconnect results in evaluation datasets that do not faithfully reflect how non-technical users (analysts, managers) actually query enterprise data, limiting the reliability of agent assessments in private BI settings. Furthermore, current benchmarks often define query difficulty via SQL syntactic features (e.g., join count), which poorly correlates with the actual complexity of business reasoning required to answer a question.

Methodology

The authors propose a Business Logic–Driven Data Synthesis framework designed to generate high-fidelity Text-to-SQL evaluation data grounded in structured business contexts. The framework operates through four primary modules:

1. Business Logic Modeling

Instead of starting from a database schema, the framework models business logic hierarchically to reflect how questions arise in real BI workflows:

• Personas: Realistic job roles (e.g., Sales Development Manager) defined by responsibilities, goals, and pain points.
• Work Scenarios: Recurring business situations where a persona engages with data (e.g., pipeline pacing review).
• Workflows: A sequence of analytical tasks decomposing a scenario into concrete steps (e.g., filtering data, comparing metrics), which naturally give rise to specific analytical questions.

2. Business Logic-Driven Schema Selection

Given production-scale databases (often containing hundreds of tables), the framework mimics human analyst behavior by selecting only relevant tables. It scores tables (0–3) based on their relevance to the specific business logic context (persona, scenario, workflow). Only high-relevance tables are retained, reducing schema complexity while preserving necessary entities for realistic query generation.

3. Text-to-SQL Data Generation

The generation process follows a SQL-first strategy, where executable SQL queries are generated first and then converted to natural language.

• Business Reasoning Complexity Control: The framework introduces a strategy to diversify query difficulty based on business reasoning steps rather than SQL syntax. It defines four complexity levels:
  1. Single Metric Analysis: Simple filtering and aggregation.
  2. Comparative Metric Analysis: Comparing metrics across dimensions (time, region).
  3. Derived Metric Analysis: Computing ratios, percentages, or applying rule-based subsets.
  4. Compositional Task Analysis: Combining multiple operations (ranking, segmenting, comparing) in a single query.
• Execution-Based Refinement: Generated SQL queries undergo an iterative refinement loop using database execution feedback to ensure executability while preserving the original business intent.
• SQL-to-Question Conversion: Executable SQL is converted into natural language questions conditioned on the persona and scenario. The questions use business-oriented terminology, avoiding explicit references to tables or SQL constructs.

4. Data Quality Evaluation

The framework employs an LLM-as-a-Judge approach to evaluate synthesized data on two dimensions:

• Question–SQL Alignment: Measures semantic correctness (result completeness, constraint consistency, structural alignment).
• Question Realism: Measures business plausibility (relevance to business goals, use of domain language, naturalness, and decision value).
• Persona–Question Alignment: (Used in analysis) Evaluates whether the question aligns with the specific job role's scope and responsibilities.

Key Contributions

1. Framework Proposal: A novel synthesis framework that grounds data generation in a hierarchical structure of personas, work scenarios, and workflows, shifting the focus from schema coverage to business realism.
2. Complexity Control Strategy: A method to define and control query difficulty via business reasoning steps and patterns, enabling systematic generation of data across varying analytical complexities.
3. Empirical Validation: A study on a production-scale Salesforce database demonstrating that the synthesized data achieves high business realism and semantic correctness, while revealing significant performance gaps in state-of-the-art Text-to-SQL models.

Experimental Results

The framework was evaluated on a private, production-scale Salesforce database (710 tables) in the sales analytics domain.

• Data Quality Comparison:

  • Business Realism: The proposed method achieved 98.44% question realism (evaluated by Gemini-3-Pro), outperforming OmniSQL (+19.5%) and SQL-Factory (+54.7%).
  • Question–SQL Alignment: The method maintained strong alignment at 98.59%, comparable to or exceeding existing methods.
  • Human Evaluation: In a human evaluation of 100 samples, 98% were judged as realistic, and 89% passed the Question–SQL alignment criterion.

• Model Performance on Synthetic Data:

  • State-of-the-art models (including GPT-5, Claude-4.5, and Gemini-3-Pro) were evaluated on the synthesized dataset.
  • Even the strongest models showed significant performance degradation on complex queries. The best-performing model (excluding GPT-5, which was used in generation) achieved only 65.79% overall execution accuracy.
  • On the most complex level (Compositional Task Analysis), the best model achieved only 42.86% execution accuracy, indicating the dataset is challenging and not trivially solvable.

• Ablation Study:

  • Removing business logic guidance (w/o BL) caused question realism to drop from 97.34% to 63.38%, confirming that schema-driven generation without business context produces unrealistic queries.
  • Removing complexity control (w/o CC) reduced the diversity of reasoning complexity levels, showing that the control strategy is essential for generating a balanced dataset.

Significance and Claims

The paper claims that current Text-to-SQL evaluation methods in private BI settings are insufficient because they prioritize syntactic diversity over business realism. By explicitly modeling business logic (personas, scenarios, workflows) and controlling for reasoning complexity, the proposed framework generates data that is both semantically correct and business-realistic.

The authors assert that their synthetic dataset serves as a more rigorous benchmark for private BI environments. The experimental results reveal that even advanced LLMs struggle with complex, multi-step business reasoning, suggesting that the field must move beyond simple SQL pattern matching to address the nuances of real-world enterprise decision-making. The paper concludes that future work should extend this framework to support multi-turn, interactive analytical tasks to better mirror iterative BI workflows.