💬 NLP

Residual Skill Optimization for Text-to-SQL Ensembles

The paper introduces DivSkill-SQL, a residual skill optimization framework that constructs complementary Text-to-SQL ensembles by iteratively training new skills on previous failures without model fine-tuning, thereby significantly improving Pass@K accuracy and reducing hallucinations across diverse SQL dialects and tasks.

Original authors: Jiongli Zhu, Haoquan Guan, Parjanya Prajakta Prashant, Nikki Lijing Kuang, Seyedeh Baharan Khatami, Canwen Xu, Xiaodong Yu, Yingyu Lin, Zhewei Yao, Yuxiong He, Babak Salimi

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiongli Zhu, Haoquan Guan, Parjanya Prajakta Prashant, Nikki Lijing Kuang, Seyedeh Baharan Khatami, Canwen Xu, Xiaodong Yu, Yingyu Lin, Zhewei Yao, Yuxiong He, Babak Salimi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a very difficult puzzle, like translating a complex human question into a specific computer language called SQL. You have a super-smart AI assistant (a Large Language Model) that can try to solve it.

Usually, if you ask this AI once, it might get it right, or it might get it wrong. To improve the odds, many systems try a "shotgun approach": they ask the AI to generate eight different answers at once and then pick the best one. This is called an ensemble.

However, the paper argues that the current "shotgun approach" has a major flaw. It's like asking the same person to write eight different essays on the same topic by just telling them to "write faster" or "change their mood." They will likely write eight essays that are all very similar, and if they make a mistake in the first one, they will probably make that same mistake in the other seven. They are all failing in the same way.

The Solution: The "Specialized Team" (DIVSKILL-SQL)

The authors of this paper, DIVSKILL-SQL, propose a smarter way to build a team of AI assistants. Instead of just asking for random variations, they train a team of eight distinct specialists, each with a unique "skill set" or personality.

Here is how they do it, using a simple analogy:

1. The "Residual" Training Method (Learning from Failure)

Imagine you are a coach training a soccer team.

The Old Way: You tell all eight players to practice the same drill. If they keep missing the goal, you just tell them to try harder or run faster. They keep missing in the exact same spot.
The DIVSKILL-SQL Way: You watch the game. You see that Player A keeps missing the goal because they are too aggressive. So, you don't just tell Player A to try again; you create a new, specific training drill just for Player A that fixes that specific weakness.
Then, you look at the remaining problems. Maybe Player B is great at shooting but bad at passing. You create a new drill just for Player B to fix the passing.

In the paper, this is called Residual Skill Optimization. The system looks at the questions the current team failed to solve, and then "optimizes" a new skill specifically to solve those remaining difficult cases. It doesn't try to make every player perfect at everything; it tries to make each player perfect at fixing the specific holes left by the others.

2. The "Skill Cards" (No Heavy Lifting)

The paper emphasizes that they don't need to retrain the entire AI brain (which is expensive and slow). Instead, they just change the instruction card (the prompt) given to the AI.

One card might say: "Be a careful explorer. Check the data first, then write the code."
Another card might say: "Be a fast coder. Write the code immediately, run it, and fix errors as they happen."
A third might say: "Break the problem into tiny pieces before building the big picture."

By giving the AI different "personality cards" for each of the eight attempts, they ensure that the eight answers are truly different from each other. If one approach fails, the others are likely to succeed because they are thinking about the problem differently.

3. The "Tournament" (Picking the Winner)

Once the team of eight specialists has generated their eight answers, the system needs to pick the winner.

Instead of just guessing or picking the first one, they use a pairwise tournament.
Imagine a boxing match: Answer A fights Answer B. The AI judge picks the winner. Then the winner fights Answer C.
The one with the most "wins" gets to be the final answer. This is more reliable than just picking one randomly.

What Did They Find?

The authors tested this on real-world database problems (using benchmarks like Spider2-Lite and BIRD-Critic). Here are the key takeaways:

Better Accuracy: Their "Specialized Team" solved significantly more problems than the best existing methods. On some difficult databases, they improved accuracy by over 11 points.
Fewer Hallucinations: Because the skills are focused on solving specific types of errors, the AI made fewer "wild guesses" (like inventing fake database tables that don't exist).
True Diversity: The paper shows that the eight answers weren't just slightly different versions of the same sentence. They were generated using completely different reasoning paths (e.g., one checked the data first, another wrote code immediately). This meant they covered more ground.
Transferable Skills: Interestingly, a team trained on one type of database (Snowflake) worked just as well on completely different databases (BigQuery, SQLite) without needing to be retrained. The "skills" (like "break the problem down") were useful everywhere.

Summary

The paper introduces a method to make AI teams smarter not by making them bigger, but by making them more diverse. Instead of asking the AI to guess eight times in a row, they teach the AI eight different ways to think, ensuring that if one way fails, another way is likely to succeed. It's the difference between asking one person to try eight times, versus asking eight different experts with unique strengths to solve the problem together.

Technical Summary: Residual Skill Optimization for Text-to-SQL Ensembles

Problem Statement

Text-to-SQL systems aim to translate natural language questions into executable SQL queries. While Large Language Models (LLMs) perform well on standard benchmarks, real-world scenarios involving large schemas, dialect-specific syntax, and multi-step logic remain challenging. Current agentic systems, which iteratively inspect schemas and repair errors, still fail to solve every query reliably.

To address this, Text-to-SQL ensembles generate multiple SQL candidates ( $K$ ) and select the best one. However, the effectiveness of any ensemble is fundamentally bounded by Pass@K—the probability that at least one of the $K$ candidates is correct. Existing methods attempt to increase diversity through stochastic decoding (high-temperature sampling) or hand-designed prompt variants. The authors argue these approaches are insufficient because they often produce candidates with correlated failures. In agentic systems, early planning randomness propagates through long reasoning trajectories, resulting in noisy variants of the same incorrect path rather than genuinely complementary solutions. Consequently, Pass@K plateaus, and additional candidates yield diminishing returns.

Methodology: DIVSKILL-SQL

The paper introduces DIVSKILL-SQL, a framework that constructs complementary agentic Text-to-SQL ensembles without model fine-tuning. Instead of relying on chance or heuristics for diversity, it treats candidate complementarity as an explicit optimization target using Residual Skill Optimization.

Core Concept: Skills and Residuals

Skills: A "skill" is defined as a high-level instruction file (system prompt) that controls an agent's decomposition style, schema-exploration policy, drafting strategy, and repair logic.
Residual Optimization: The framework operates on the principle that a new skill should not necessarily be globally better than its predecessors, but rather should solve the specific examples the current ensemble misses.
- Let $D_{train}$ be the training set.
- After selecting skills $s_1, \dots, s_{j-1}$ , the residual set $R_{j-1}$ is defined as the subset of examples where the current ensemble fails.
- The next skill $s_j$ is optimized to maximize success specifically on $R_{j-1}$ .
- This process greedily targets the marginal contribution of a new skill to the overall Pass@K.

Algorithm Flow

Seed Pool Initialization: The process starts with a diverse pool of $K$ manually curated seed skills (e.g., decompose, direct_coder, explore_heavy, adversarial_checker), each encoding a distinct reasoning strategy.
Iterative Optimization:
- The algorithm runs in batches. For each skill in the pool, it identifies the current residual failures (examples the ensemble has not solved).
- Using Reflective Prompt Optimization (specifically the GEPA technique), an LLM analyzes failure traces (schema inspections, execution errors) and proposes a refined prompt to address these specific failures.
- The refinement is accepted only if it improves recovery on the residual set.
- The residual set is updated to remove examples newly solved by the refined skill.
Inference: At test time, the $K$ learned skill-conditioned agents run in parallel. Candidates are deduplicated by execution output, and the final SQL is selected via pairwise comparison using an LLM judge, which is more reliable than direct 1-of-K selection.

Key Contributions

Formulation: The authors formulate Text-to-SQL ensembling as a residual Pass@K optimization problem over agent skills, aiming to learn complementary behaviors that cover different failure modes.
Framework: They propose DIVSKILL-SQL, a method that improves candidate-set coverage without model fine-tuning or high-temperature sampling, making it applicable to both open and closed LLMs.
Theoretical Guarantee: The paper provides a proof (Proposition A.1) showing that under the population objective, the greedy residual optimization strategy achieves at least a $(1 - 1/e)$ -approximation of the optimal $K$ -skill ensemble, leveraging the submodularity of the Pass@K objective.
Empirical Validation: Extensive evaluation across multiple benchmarks and SQL dialects demonstrates that optimizing skills on unresolved examples directly targets marginal contributions to Pass@K.

Experimental Results

The framework was evaluated on Spider2-Lite (covering SQLite, Snowflake, and BigQuery) and BIRD-Critic (PostgreSQL debugging), using Opus-4.6 and GPT-5.4 as base models.

Performance Gains:
- On Spider2-Lite (Snowflake), DIVSKILL-SQL improved selected accuracy by +11.1 points over the strongest baseline (CHASE-SQL).
- On Spider2-Lite (BigQuery), it improved by +8.3 points.
- On BIRD-Critic, it improved selected accuracy by +2.6 points, demonstrating transfer to a debugging-style task formulation.
Transferability: Skills optimized on a single dialect (Snowflake) transferred effectively to BigQuery and SQLite without retraining, indicating the learned strategies capture high-level problem-solving logic rather than dialect-specific syntax.
Error Reduction: Error diagnostics revealed up to 3× fewer hallucinated schema references and unsupported function calls compared to baselines.
Behavioral Diversity: Trajectory analysis showed that DIVSKILL-SQL reduces redundancy in agent behavior by 19%–28%. Unlike repeated sampling which produces correlated failures, the learned skills induce genuinely different reasoning paths (e.g., distinct schema inspection and decomposition patterns).
Efficiency: To achieve the same Pass@K coverage as DIVSKILL-SQL's 8 candidates, baseline methods required 3 to 8 additional passes, highlighting the cost-efficiency of the residual approach.

Significance and Claims

The paper claims that DIVSKILL-SQL represents a shift from relying on stochastic diversity to structured, complementary diversity. By explicitly optimizing for the "residual" cases that previous agents miss, the system constructs an ensemble where the whole is greater than the sum of its parts.

The authors emphasize that their gains stem from genuinely reliable complementary skills rather than surface-level variations. This approach addresses the fragility of agentic systems where early randomness leads to unstable reasoning. Furthermore, the method's ability to transfer across dialects and task types (from generation to debugging) suggests that the optimized skills encode robust, generalizable strategies for interacting with relational databases. The framework offers a practical path to improving Text-to-SQL reliability without the computational cost of fine-tuning multiple models.