Technical Summary: SIRIUS-SQL

Problem Statement

Text-to-SQL systems struggle with complex schemas in real-world databases, where single-pass generation often fails due to schema heterogeneity, implicit constraints, and ambiguous value references. While recent approaches mitigate this by generating multiple SQL candidates and using voting to filter errors, the paper identifies three coupled structural weaknesses in existing multi-candidate pipelines:

1. Diminishing-Return Sampling (L1): Sampling more candidates from a single generator yields increasingly redundant outputs because the generator's distribution is fixed by its training. Simply increasing the sample budget ($k$) does not reliably surface new, correct hypotheses outside the generator's preferred region.
2. Wasted Execution Signal (L2): Existing pipelines apply a single, generic correction strategy to all non-clean execution results. However, different failure modes—runtime errors (syntax/type mismatches), timeouts (inefficient plans), and empty results (semantic drift)—indicate different distances from correctness and require distinct repair strategies.
3. Voting–Oracle Gap (L3): Even when the correct SQL exists within the candidate pool, majority voting often fails to select it. This occurs because selection requires joint judgment in two spaces: the result space (where different SQLs may yield identical outputs) and the logic space (where the same intent admits many surface forms). Existing selectors typically operate in only one of these spaces.

Methodology: SIRIUS-SQL

SIRIUS-SQL addresses these limitations by anchoring every stage of the pipeline in execution feedback. The system consists of three core components:

1. Source-Complementary Candidate Construction

To overcome diminishing returns (L1), the system constructs an initial candidate pool from two complementary sources:

• Specialist ($G_{sp}$): A task-specialized model (SIRIUS-32B) trained with Reinforcement Learning from Verifiable Execution Rewards (RLVR). The training employs a difficulty-smoothed data synthesis strategy to focus on instances where the policy can improve, and a two-stage DAPO schedule with threshold-gated entropy regularization to prevent entropy collapse while maintaining diverse, executable candidates.
• Generalist ($G_{ge}$): An off-the-shelf strong LLM (e.g., Qwen3-235B, Gemini-3.1) prompted with few-shot demonstrations to cover long-tail phrasing and implicit constraints that the specialist might miss.
  The union of these sources ensures that candidates expose different failure modes, reducing error correlation.

2. Execution-Grounded Candidate Lifecycle

Instead of treating execution as a final binary check, the system uses execution outcomes to drive a dynamic repair loop (L2):

• Pre-execution Screening: Deterministic rule-based filters remove malformed SQL (e.g., syntax errors, schema mismatches) before execution to save repair budget.
• Diagnosis & Typing: Surviving candidates are executed and classified into four categories: clean, runtime error, timeout, and empty result.
• Type-Specific Repair:
  • Runtime Errors: The engine's exception text is fed verbatim to a corrector.
  • Timeouts: The rewrite preserves intent but restructures join orders or subquery shapes to improve efficiency.
  • Empty Results: A two-stage approach is used: first, high-precision structural operators (e.g., literal re-binding, predicate adjustment) attempt local recovery; if that fails, a diagnosis-conditioned LLM fallback is triggered.
• Pool Consolidation: Repaired candidates are re-executed and re-admitted to the pool. If the pool collapses, a recovery generator uses condensed negative feedback to regenerate candidates.

3. Confidence-Gated Hybrid Selection

To bridge the voting-oracle gap (L3), the system employs a two-stage selector that combines result-level and logic-level signals:

• Stage 1 (Result-Level Scorer): Computes a coarse score ($S_{coarse}$) by summing:
  • Tuple-level Consensus ($S_{maj}$): Frequency of cell-level triples across the candidate set.
  • Position-Debiased Pairwise Preference ($S_{rm}$): An LLM judge compares representatives of result-signature equivalence classes to rank logic quality.
• Stage 2 (Structural-Support Reranker): A confidence-gated mechanism. If the top candidates are tied across all signals (low margin), a deterministic, LLM-free reranker intervenes. It analyzes the AST-level structural signature (tables, projections, aggregations) and selects the candidate supported by the most distinct provenance groups (cross-generator agreement). This acts as a zero-cost tie-breaker.

Key Contributions

1. Difficulty-Smoothed RL Recipe: A training methodology for SIRIUS-32B that produces a broad yet reward-validated candidate distribution, effectively mitigating diminishing returns in sampling.
2. Execution-Grounded Lifecycle: A repair framework that classifies execution outcomes and applies targeted, type-specific repairs (runtime, timeout, empty) before re-admitting candidates, maximizing the utility of execution signals.
3. Confidence-Gated Hybrid Selector: A selection mechanism that fuses tuple-level consensus with pairwise SQL preference, escalating only ambiguous cases to a deterministic structural check, thereby resolving the voting-oracle gap without excessive computational cost.

Results

SIRIUS-SQL achieves state-of-the-art performance on standard benchmarks:

• BIRD Dev: 75.88% execution accuracy (using SIRIUS-32B + Gemini-3.1 Pro), surpassing the strongest published multi-candidate system, Agentar-Scale-SQL (74.90%), by 0.98 percentage points. It also shows significant gains over single-pass baselines (e.g., +11.54% over Qwen3-Coder).
• SPIDER Test: 91.20% execution accuracy, demonstrating strong generalization without task-specific retuning.
• Ablation Studies:
  • The difficulty-smoothed data synthesis improved validation rewards by 1.8 pp on BIRD and 0.5 pp on SPIDER during training.
  • Complementary candidate construction (Specialist + Generalist) outperformed single-source scaling, proving that diverse failure modes are more valuable than sheer sample count.
  • The type-specific lifecycle outperformed unified correction strategies by 0.13–0.71 pp.
  • The hybrid selector (merge + gate) provided a 0.20 pp gain over single-signal selectors, primarily by resolving low-margin ties.

Significance

The paper argues that the primary bottleneck in multi-candidate Text-to-SQL is not the generation of more candidates, but the systematic under-utilization of execution signals across training, repair, and selection. By anchoring the entire pipeline in execution feedback—training a specialist to produce diverse executable candidates, repairing failures based on specific error types, and selecting based on a hybrid of result consensus and structural logic—SIRIUS-SQL demonstrates that execution-grounded refinement is a critical lever for improving reliability on complex schemas. The system's ability to surpass strong baselines with two of three generalist pairings suggests a robust, partner-agnostic architecture for high-stakes Text-to-SQL applications.