💬 NLP

FlexSQL: Flexible Exploration and Execution Make Better Text-to-SQL Agents

FlexSQL introduces a text-to-SQL agent that outperforms stronger models on the Spider2-Snow benchmark by employing a flexible design principle that allows for dynamic schema exploration, data inspection, and diverse execution plans with a two-tiered repair mechanism throughout the reasoning process.

Original authors: Quang Hieu Pham, Yang He, Ping Nie, Canwen Xu, Davood Rafiei, Yuepeng Wang, Xi Ye, Jocelyn Qiaochu Chen

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Quang Hieu Pham, Yang He, Ping Nie, Canwen Xu, Davood Rafiei, Yuepeng Wang, Xi Ye, Jocelyn Qiaochu Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of a crime scene, your "crime scene" is a massive, chaotic library containing millions of books, files, and data points. Your boss gives you a vague clue: "Find all the patents about materials science filed in early 2014 and count how many older patents they mention."

Most current computer programs trying to solve this act like a rigid robot. They are given a map of the library once at the start. They pick a few shelves based on that map, write a list of instructions (a query), and then try to execute it. If they get stuck or the answer is wrong, they can only fix small typos in their instructions. They cannot go back and say, "Wait, I picked the wrong shelves entirely," or "I didn't realize 'materials science' was hidden under a code like 'MS-01'." They are stuck with their initial mistake.

FlexSQL is a new kind of detective agent that acts more like a curious human explorer. Instead of being rigid, it is flexible. Here is how it works, using simple analogies:

1. The "Flexible Explorer" (Flexible Interaction)

Instead of looking at the library map once and committing to a path, FlexSQL is allowed to wander around the library while it thinks.

The Old Way: The robot looks at the map, guesses "Materials Science" is in the "Science" section, and starts writing its list.
FlexSQL's Way: The agent says, "I'm not sure where 'Materials Science' is. Let me walk over to the 'Technology' section, peek inside a box, and read the labels."
It uses special tools to browse the shelves (schema), read the actual contents of the boxes (data values), and run small test searches to see if its guesses make sense. If it realizes it's in the wrong aisle, it can immediately turn around and go to the right one. It doesn't get stuck in a dead end.

2. The "Brainstorming Team" (Diverse Planning)

The clue "earlier patents each one cites" is tricky. Does it mean only domestic patents? Foreign ones too? Pending applications?

The Old Way: The robot picks one interpretation (e.g., "Only domestic") and sticks to it. If it's wrong, the whole answer is wrong.
FlexSQL's Way: It acts like a team of detectives brainstorming. It generates multiple different theories (plans) at once.
- Plan A: Count only domestic citations.
- Plan B: Count domestic and foreign.
- Plan C: Count everything, including pending apps.
  It runs all these theories and then lets the "majority vote" decide which answer is the most likely to be correct. This way, even if one theory is wrong, the team might still get the right answer.

3. The "Bilingual Translator" (Flexible Execution)

Sometimes, a task is too complicated to write as a single, straight-line instruction (SQL). It might require a loop, a "if-then" decision, or a step-by-step calculation that is messy to write in a database language.

The Old Way: The robot tries to force everything into one rigid SQL sentence, often making it too complex and prone to breaking.
FlexSQL's Way: It is bilingual. It can write the instructions in SQL (the language databases speak) OR in Python (a flexible programming language).
- If the task is a simple list, it uses SQL.
- If the task requires a complex loop or a step-by-step calculation, it writes a Python script first because it's easier to think that way.
- Once the Python script works perfectly, it translates the final result back into SQL so the database understands it. It's like writing a complex recipe in a notebook first, then translating it into the official kitchen order.

4. The "Do-Over" Button (Backtracking)

If FlexSQL writes a plan and then realizes, "Oh no, I misunderstood the question entirely," it doesn't just try to fix the code. It hits the backtrack button. It goes all the way back to the beginning, re-examines the library shelves, changes its entire strategy, and starts a new plan. This prevents it from wasting time trying to fix a broken foundation.

The Results

The paper tested this "flexible detective" on a very difficult benchmark called Spider2, which simulates massive, real-world corporate databases.

The Score: Using a model called gpt-oss-120b, FlexSQL scored 65.4%.
The Comparison: This score was higher than other top systems that used much larger, more powerful models (like DeepSeek-R1 or gpt-o3).
The Takeaway: By being flexible—wandering the library, brainstorming multiple theories, speaking two languages, and hitting "undo" when needed—FlexSQL solved problems better than the "rigid robots," even with a smaller brain.

In short, FlexSQL proves that in the complex world of big data, flexibility beats rigidity. It's better to be an agent that can explore, adapt, and change its mind than one that stubbornly follows a single, pre-written map.

Technical Summary: FlexSQL

Problem Statement

Text-to-SQL systems face increasing challenges as analytical databases grow in scale and complexity. Modern data warehouses (e.g., Snowflake, BigQuery) often contain hundreds of tables organized into multiple schemas, with some databases exceeding 60,000 columns. Current systems typically follow a fixed pipeline: they retrieve schema elements once upfront, generate a query based on that static snapshot, and only revisit the database for post-hoc repair if errors occur.

This rigidity creates two primary failure modes:

Inability to recover from early mistakes: If a model selects the wrong tables or misunderstands a column's meaning during the initial schema linking phase, downstream repair mechanisms cannot recover because the agent is trapped in a loop of fixing code rather than revising its fundamental assumptions.
Query Ambiguity: Natural language queries often admit multiple valid interpretations (e.g., which subset of citation tables to use). Fixed pipelines often commit to a single interpretation, missing valid alternative solutions.

Methodology: FlexSQL Framework

FlexSQL introduces a text-to-SQL agent designed around flexible database interaction. Instead of a fixed pipeline, the agent can explore schema structures, inspect data values, and run verification queries at any point during reasoning.

Core Components

Tool-Based Database Interaction:
The agent is equipped with a suite of tools to interact with the database incrementally:
- Schema Navigation: GetSchema (list tables in a schema) and GetTableCol (inspect columns and sample values) allow top-down exploration.
- Data Inspection: GetColValues and FindRows enable the agent to ground decisions in actual data values (e.g., discovering that "materials science" corresponds to specific codes like "MS-01") rather than relying solely on metadata.
- Code Execution: SQLExecutor and PythonExecutor allow the agent to test subqueries, verify intermediate results, and debug partial solutions iteratively.
Diversity-Enforced Plan Generation:
To handle query ambiguity, FlexSQL generates $K$ diverse execution plans using verbalized sampling. These plans cover different interpretations regarding table choices, join paths, and predicate readings. The agent actively explores the database to ground each plan in real data before finalizing it.
Bilingual Program Synthesis (SQL & Python):
FlexSQL allows each plan to be implemented in either SQL or Python, depending on the task's complexity.
- SQL is used for declarative relational queries.
- Python is employed for procedural workflows, multi-step transformations, and iterative logic that is cumbersome in pure SQL.
- Python solutions are subsequently transpiled into SQL for the final output.
Two-Tiered Repair and Backtracking:
- Code-Level Repair: If a generated program fails (syntax errors, wrong aggregation), the agent regenerates the code with error feedback.
- Plan-Level Backtracking: If the error stems from a deeper reasoning flaw (e.g., wrong table selection or missing joins), the system backtracks to the Plan Generation stage. The agent revises the plan itself, potentially re-exploring the database, rather than just patching the code.
Majority Voting:
After generating $K$ programs (a mix of SQL and Python), the system groups them by execution output. The consensus answer (the largest equivalence class) is selected. If the consensus is a Python solution, it is transpiled to SQL.

Key Contributions

Flexible Interaction Architecture: A text-to-SQL agent that replaces fixed pipeline stages with on-demand tools for schema navigation, data inspection, and code execution throughout the entire reasoning process.
Two-Stage Plan-to-Program with Backtracking: An architecture that allows agents to revise not only code but also their underlying plans and schema assumptions when errors are detected.
Diversity and Bilingual Generation: A combination of diversity-enforced plan sampling to handle ambiguity and bilingual (SQL/Python) generation to flexibly implement varied strategies.
Empirical Validation: Strong performance on the Spider2.0 benchmark, demonstrating effectiveness across different model sizes and test-time scaling.

Experimental Results

The authors evaluated FlexSQL on Spider2-Snow (544 questions over Snowflake databases) and Spider2-SQLite (135 questions over SQLite databases), comparing against strong open-source baselines (ReFoRCE and DSR-SQL) using gpt-oss-120b and gpt-oss-20b.

Performance: Using gpt-oss-120b, FlexSQL achieved 65.44% Majority@8 on Spider2-Snow. This outperformed DSR-SQL (using DeepSeek-R1) and ReFoRCE (using gpt-o3), despite FlexSQL using a significantly smaller backbone model.
Parameter Efficiency: The gpt-oss-20b variant of FlexSQL surpassed or remained comparable to baselines using the larger 120b models.
Test-Time Scaling: FlexSQL showed consistent gains as the sampling budget ( $K$ ) increased from 8 to 64, confirming the effectiveness of the diversity-enforced approach.
Ablation Studies:
- Removing Python support caused the largest performance drop, highlighting the necessity of bilingual generation for complex logic.
- Removing diverse planning reduced Majority@8 accuracy, confirming that covering multiple interpretations improves consensus quality.
- Removing backtracking reduced performance, showing the value of revising high-level plans rather than just fixing code.
Generalization: When integrated as skills into a general-purpose coding agent (Claude Code), FlexSQL yielded over 10% relative improvement on Spider2-Snow.

Significance and Claims

The paper claims that flexibility is a key design principle for next-generation Text-to-SQL agents. By allowing agents to explore schemas, inspect data, and backtrack from errors at any stage, FlexSQL overcomes the limitations of fixed pipelines that fail to recover from early reasoning mistakes.

The authors emphasize that their approach:

Grounds decisions in actual data: Moving beyond static schema metadata to inspect real values resolves ambiguities that metadata alone cannot.
Leverages inference-time compute: By generating diverse plans and using majority voting, the system effectively utilizes increased sampling budgets to improve robustness.
Bridges the gap between declarative and procedural logic: The bilingual approach (SQL/Python) allows the agent to choose the most effective tool for the specific reasoning task, translating complex procedural logic back into SQL for execution.

The work suggests that for large-scale, complex analytical databases, the ability to iteratively explore and revise assumptions is more critical than simply generating a query from a pre-retrieved schema snapshot.