Technical Summary: TAHOE (Text-to-SQL with Automated Hint Optimization from Experience)

1. Problem Statement

While Large Language Models (LLMs) have advanced Text-to-SQL capabilities, bridging the gap between prototype demos and robust production deployment remains a significant challenge. Real-world database environments present three specific hurdles that vanilla LLMs struggle to handle:

1. Strict Dialect Constraints: Industrial environments (e.g., Snowflake) enforce specific syntax rules (such as case-sensitive quoting) that standard SQL-trained models frequently violate, leading to immediate compilation errors.
2. Massive Schemas: Real-world databases contain hundreds of tables with obscure column names, causing models to hallucinate join paths or misinterpret definitions due to context window limitations.
3. Evolving User Intent: User preferences and business logic definitions (e.g., how to handle ties in "top products" or specific metric calculations) are dynamic and often ambiguous, creating a "moving target" that static model weights cannot track without expensive retraining.

Existing solutions face distinct limitations:

• Test-Time Scaling (Agentic Workflows): Iterative refinement and multi-turn agent loops improve accuracy but incur prohibitive latency and computational costs. They also suffer from "total amnesia," failing to retain lessons learned from previous sessions.
• Supervised Fine-Tuning (SFT): While effective for static tasks, SFT is rigid. Adapting to schema changes, new dialects, or evolving user preferences requires expensive retraining, and the resulting weights are opaque and non-transferable across different base models.
• Retrieval-Augmented Generation (RAG): Simple retrieval of raw documentation or examples often fails due to "Context Noise," where excessive metadata buries critical information, leading to hallucinations.

2. Methodology: The Tahoe System

Tahoe reframes prompt optimization from a transient computational process into a persistent data management problem. Instead of modifying model parameters, Tahoe maintains an external, evolving Hint Bank that stores distilled logic derived from past failures and successes.

Core Architecture

The system operates on a two-phase lifecycle:

1. Development Phase: Bootstraps the Hint Bank using a small offline dataset with ground-truth SQL and execution results.
2. Deployment Phase: Adapts via asynchronous updates using compiler feedback (for syntax) and human-in-the-loop preference signals (for semantics).

The system comprises three core modules:

• Hint Learning Module (The "Writer"): An error-driven pipeline that processes failures to generate Atomic Diffs. It uses a temporary bank for iterative refinement, converting compiler errors and execution mismatches into structured hints.
• Hint Bank Management Module (The "Curator"): Consolidates temporary deltas into the global bank. It handles deduplication, scope-aware merging, and Strategy Attribution. Crucially, it models conflicting user intents as competing Strategies under a single trigger, ranked by empirical success, harm, and inertness statistics.
• Hint-Guided Inference Module (The "Reader"): At inference time, it retrieves relevant hints and guides the LLM through a two-stage process:
  1. Logic Planning: Selects and combines strategies (resolving conflicts based on attribution statistics) to form a solution plan.
  2. SQL Synthesis: Generates the final query using the plan and full dialect-specific syntax hints.

Data Model: The Hint Bank

The Hint Bank ($H$) is split into two disjoint registries:

• Syntax Hints ($H_{syn}$): Rule-based tuples $(R, E)$ capturing dialect-specific compilation rules (e.g., "Quote every identifier exactly as stored"). These are injected fully for the target dialect.
• Semantic Hints ($H_{sem}$): Structured tuples $(T, S, \Sigma)$ containing:
  • Trigger ($T$): A natural language pattern for retrieval.
  • Scope ($S$): Classification as General, Database-specific, or User-specific.
  • Strategies ($\Sigma$): A set of competing solutions for a trigger. Each strategy includes a rationale, contrastive examples, and statistical metadata:
    • Recency: When the strategy was last distilled.
    • Eval Stats: Post-learning attribution summarizing empirical success, harm, and inertness across actual generations.

Learning Pipeline

The learning process follows an Execution $\to$ Feedback $\to$ Reflection $\to$ Re-evaluation cycle:

1. Multi-Sampling: Generates multiple candidates.
2. Sequential Feedback:
   • Syntax: Uses compiler feedback to fix dialect errors, generating syntax diffs.
   • Semantic: Compares execution results against ground truth (or user-accepted candidates in deployment) to identify logic gaps, generating semantic diffs.
3. Suggestion & Clustering: Converts diffs into hint suggestions, which are then clustered and applied to a temporary bank.
4. Attribution: Periodically evaluates the bank on a frozen set to update strategy statistics (success/harm rates), ensuring the system distinguishes between "learned" strategies and "useful" ones.

3. Key Contributions

1. Hint-Based Long-Term Memory Paradigm: Tahoe shifts the burden of reasoning and correction from the inference path to a background data management process. By converting transient feedback into a persistent Hint Bank, it mitigates context noise and alleviates the latency bottlenecks of agentic workflows while maintaining compatibility with different base LLMs.
2. Structured and Transferable Hint Architecture: The system decouples syntax from semantics and organizes logic by scope. The Strategy Layer explicitly models conflicting interpretations as competing strategies, allowing the system to resolve ambiguity at runtime based on flexible signals (recency and empirical attribution) rather than static retrieval.
3. Empirical Effectiveness and Cross-Model Transferability: The approach demonstrates that a Hint Bank learned by a strong model can significantly boost the performance of weaker models without retraining, proving the externalization of dialect and logic knowledge.

4. Experimental Results

Evaluations were conducted on the Spider 2.0–Snow-0212 benchmark (113 supervised development examples and 434 held-out examples).

Performance on Development Set (113 examples) with GPT-5.5:

• Pass Rate: Increased from 61.95% (Vanilla) to 79.42% (+17.47 pp).
• Pass@4: Increased from 72.57% to 87.61% (+15.04 pp).
• Syntax Pass Rate: Achieved 100% (up from 96.24%).
• Efficiency: Reduced average compiler-feedback critic rounds from 2.79 to 0.12 per candidate.

Cross-Model Transferability:
Using the same Hint Bank (learned by GPT-5.5) on weaker backbones without retraining yielded significant gains:

• Doubao-2.0-lite: Pass rate improved from 29.42% to 49.12% (+19.70 pp).
• GPT-5: Pass rate improved from 42.70% to 58.85% (+16.15 pp).

Generalization to Held-Out Examples:

• Syntax: Transfer remained strong, with syntax pass rate rising from 90.21% to 99.14% and critic rounds dropping significantly.
• Semantics: Gains were more modest (+1.90 pp pass rate), suggesting semantic benefits depend heavily on how well the development set covers the target workload's specific logic.
• Comparison: Tahoe outperformed a reproduced SQLGenie-style RAG baseline, which actually degraded performance on held-out data, highlighting the superiority of distilled, structured hints over raw example injection.

Ablation Study:
Removing the Strategy Attribution mechanism (which ranks strategies by empirical success/harm) resulted in a drop in Pass Rate from 79.42% to 69.03%, confirming that distinguishing "learned" from "useful" strategies is critical for performance.

5. Significance and Claims

The paper claims that Tahoe offers a practical framework for robust and adaptable database interfaces by addressing the "Compute Trap" of agentic scaling and the "Rigidity Trap" of SFT. Its primary significance lies in:

• Decoupling Adaptation from Parameters: By externalizing knowledge into a Hint Bank, the system allows for continuous evolution and model agnosticism without the cost of retraining.
• Handling Ambiguity: The Strategy Layer provides a structured mechanism to manage conflicting user intents and domain-specific conventions, resolving them at runtime based on empirical evidence rather than static rules.
• Production Viability: By shifting the computational cost of error correction to the background (Development Phase or batch updates), Tahoe enables low-latency inference suitable for real-time applications.

The authors note that while the current evaluation focuses on the Development Phase, the architecture is designed to support deployment-time updates via human feedback, positioning Tahoe as a step toward self-improving database interfaces that rely on continuous external knowledge construction rather than internal parameter updates.