💻 computer science

SemanticAgent: A Semantics-Aware Framework for Text-to-SQL Data Synthesis

SemanticAgent is a novel framework that improves text-to-SQL data synthesis by replacing simple execution-based validation with a three-stage, semantics-aware reasoning process involving analysis, synthesis, and verification, thereby generating higher-quality synthetic data that enhances downstream model performance.

Original authors: Qiang Gao, Zhenping Li, Anqi Zhuo, Yingxiao Zhao, Weibo Geng, Xiaosong Li

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qiang Gao, Zhenping Li, Anqi Zhuo, Yingxiao Zhao, Weibo Geng, Xiaosong Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly literal-minded robot how to talk to a giant, complex library of information (a database). You want the robot to understand your questions like "Show me the schools with the most free lunches" and turn them into the exact computer code (SQL) needed to get that answer.

The problem is, existing methods are like teaching a robot to drive by only checking if the car starts and moves. If the car moves, the teacher says, "Good job!" But they don't check if the car is driving into a wall, going the wrong way down a one-way street, or trying to drive a boat on a highway. The car works, but it's doing the wrong thing.

This paper introduces SemanticAgent, a new framework that acts like a super-vigilant driving instructor who doesn't just check if the car moves, but checks if the logic of the drive makes sense.

Here is how it works, broken down into simple analogies:

1. The Problem: The "Running but Wrong" Car

In the old way of making training data for these robots, they would generate a question and an answer, run the code, and if the code didn't crash (it was "executable"), they kept it.

The Flaw: Imagine asking, "What is the average name of the students?" The robot might write code that tries to add up the letters in names (A+B+C...). The code might run without crashing, but the answer is nonsense. The robot learned to drive, but it learned to drive badly.

2. The Solution: SemanticAgent (The Three-Step Team)

Instead of just one robot guessing and checking, SemanticAgent uses a team of three specialized experts who work together in a loop. Think of them as a Research Team, a Writer, and a Fact-Checker.

Step 1: The Researcher (The Analyzer)

Before writing anything, this agent studies the database like a detective studying a city map.

What it does: It doesn't just look at the street names (table names); it learns the rules of the city. It learns that "CDSCode" is a school ID (like a license plate), not a number you can average. It learns that "Free Meal Count" has to be divided by "Total Enrollment" to get a percentage.
The Analogy: It's like reading the local traffic laws and understanding that you can't drive a school bus on a bike path, before you even start the engine.

Step 2: The Writer (The Synthesizer)

This agent is the one who actually writes the question and the code.

What it does: It writes the question and the SQL code, but it has to follow the "rulebook" created by the Researcher. It writes down its thought process (a "rationale") explaining why it chose specific tables and columns.
The Analogy: It's like a chef writing a recipe. But instead of just guessing ingredients, the chef is forced to write, "I am using salt because the recipe says the soup needs seasoning," ensuring they aren't accidentally adding sugar.

Step 3: The Fact-Checker (The Verifier)

This is the most important new part. This agent reviews the Writer's work before it's saved.

What it does: It compares the Writer's code against the Researcher's rulebook.
- Did the Writer try to average a school ID? Stop! That's a violation.
- Did they join the tables correctly? Stop! That's a violation.
The Analogy: This is the editor who catches the mistake before the book is printed. If the Writer says, "I averaged the school IDs," the Fact-Checker says, "No, you can't do that. School IDs are names, not numbers. Fix it."

3. The Result: A Better Driver

Because this team checks the meaning of the code, not just if it runs, the training data they create is much higher quality.

Old Way: The robot learns from 100 examples where 10 were "driving into walls" but didn't crash. The robot learns bad habits.
SemanticAgent Way: The robot learns from 100 examples where every single one is logically sound. The robot learns the rules of the road, not just how to turn the key.

Why Does This Matter?

The paper tested this on difficult real-world scenarios (like medical records or complex school data).

The Outcome: When they trained new robots using the data from SemanticAgent, those robots were much better at answering tricky questions. They didn't just get the right answer by luck; they understood the logic.
The Metaphor: It's the difference between teaching a student to pass a test by memorizing answers (which fails when the question changes slightly) versus teaching them the underlying math so they can solve any problem.

In short: SemanticAgent stops the AI from "faking it" by ensuring that every piece of code it learns actually makes sense in the real world, not just inside the computer.

1. Problem Statement

Current Text-to-SQL data synthesis pipelines suffer from a critical flaw: they conflate executability with semantic validity.

The Issue: Existing methods rely heavily on syntactic checks and execution-based validation (i.e., does the SQL run without error?). However, a query can execute successfully while violating the underlying database semantics or domain rules.
Example: A query might calculate AVG(CDSCode) (an identifier) or join tables on mismatched columns. While the SQL executes, the result is semantically meaningless.
Consequence: This leads to synthetic training data that contains "silent errors," which degrades the performance of downstream models, particularly in domain-specific scenarios (e.g., healthcare, finance) where strict semantic constraints exist.
Gap: Current LLM-based synthesis methods excel at structural alignment but lack mechanisms for holistic semantic consistency and traceable reasoning during generation.

2. Methodology: SemanticAgent

The authors propose SemanticAgent, a tool-augmented framework that replaces implicit semantic modeling with explicit semantic supervision. The framework operates on a structured Semantic Knowledge Base ( $K$ ) derived from database schemas and instances, organized into three specialized modules and a three-stage protocol.

A. Core Components (Tools)

Data Analysis Tool (DA): Acts as a domain analyst. It extracts domain-specific rules, entity semantics, and constraints from the database to build a structured knowledge base.
Data Synthesis Tool (DS): Acts as a data author. It generates question-SQL-rationale triples under the explicit guidance of the knowledge base.
Diagnosis Tool (DT): Acts as a semantic reviewer. It inspects outputs against domain constraints, identifies inconsistencies, and repairs invalid samples.

B. The Three-Stage Protocol

Semantic Analysis (Knowledge Extraction):
- The system infers a multi-level semantic knowledge base ( $K$ $K$ ) through a six-stage hierarchical process:
  - Stages 1-2: Extract schema metadata and domain constraints.
  - Stages 3-5: Analyze field types, column semantics, and table constraints.
  - Stage 6: Infer cross-table relations and foreign key dependencies.
- This creates a rich context (e.g., "CDSCode is a unique identifier, not a numeric value for averaging") that goes beyond standard DDL or M-Schema.
Controlled Authoring (Trace-Grounded Generation):
- The DS tool generates triples $(q, s, r)$ where $q$ is the question, $s$ is the SQL, and $r$ is a reasoning trace.
- Generation is controlled by domain context, task type, and SQL complexity levels (L1–L4).
- Crucially, the model must produce a reasoning trace that explicitly justifies table selection, column operations, and strategy formulation based on $K$ .
Diagnosis and Refinement:
- The DT tool validates the generated triple against $K$ .
- Diagnosis: Checks for invalid columns, type mismatches, incorrect joins, and logic errors (e.g., ORDER BY ASC when DESC is required).
- Retrieval & Correction: If errors are found, the system retrieves specific evidence from $K$ and instructs the LLM to correct the SQL and the reasoning trace.
- This iterative loop continues until the sample is semantically valid or a max iteration limit is reached.

3. Key Contributions

Knowledge-Guided Framework: Introduction of SemanticAgent, which integrates explicit semantic supervision into the synthesis pipeline, moving away from reliance on implicit assumptions.
Explicit Semantic Guidance: The framework decomposes synthesis into knowledge extraction, instruction synthesis, and verification, ensuring domain rules are enforced at every step.
Traceable Reasoning: By requiring a reasoning trace ( $r$ ) and validating it against the knowledge base, the system makes error sources detectable and correctable, rather than treating generation as a black box.
Empirical Validation: Demonstrated that synthetic data generated with semantic awareness leads to superior downstream fine-tuning performance compared to state-of-the-art baselines.

4. Experimental Results

The framework was evaluated on multiple benchmarks (Spider, BIRD, Spider2.0, EHRSQL, ScienceBenchmark) using Qwen2.5-Coder and Qwen3 backbones.

Downstream Performance (RQ1):
- Models fine-tuned on SemanticAgent data consistently outperformed baselines (CodeS, SynQL, OmniSQL).
- Key Gains: On the BIRD benchmark (knowledge-intensive), SemanticAgent improved execution accuracy (EX) by 2.6 points over OmniSQL (using Qwen2.5-Coder-7B). On Spider2.0-SQLite, gains were up to 3.4 points.
- The improvements were most pronounced in complex, domain-specific, and robustness-oriented settings.
Data Quality (RQ2):
- Executability: SemanticAgent achieved the highest Successful Execution Rate (SER) on the most challenging benchmark (Spider 2.0), indicating it can handle complex constraints better than baselines.
- Complexity: It generated a higher proportion of moderately complex and complex SQL queries (L2–L4), offering broader structural coverage.
- Diversity: It achieved higher semantic diversity (measured by Mean L2 and 1-NN distances in embedding space), reducing redundancy in the training set.
Scaling Behavior (RQ3):
- As the volume of synthetic data increased, SemanticAgent showed superior data efficiency. The performance gains emerged early in the low-resource regime and remained stable, suggesting the data is higher quality and more informative per sample.

5. Significance and Conclusion

Paradigm Shift: The paper argues that execution success is insufficient for reliable synthetic supervision. Explicit semantic verification is necessary to prevent "semantically invalid but executable" data from polluting training sets.
Domain Adaptability: The framework is particularly valuable for specialized domains (e.g., clinical, scientific) where schema structure alone cannot capture necessary business logic.
Limitations: The approach is computationally expensive (requiring ~2,880 GPU-hours for corpus construction) and currently relies on a single teacher model, which may introduce bias.
Future Direction: The work suggests that future Text-to-SQL research must prioritize semantic grounding and traceable reasoning over mere syntactic correctness to achieve robust, real-world performance.