💬 NLP

Reliable Answers for Recurring Questions: Boosting Text-to-SQL Accuracy with Template Constrained Decoding

The paper introduces TeCoD, a system that improves Text-to-SQL accuracy and reduces latency by leveraging recurring query patterns to select reusable templates and enforce them via grammar-constrained decoding.

Original authors: Smit Jivani, Sarvam Maheshwari, Sunita Sarawagi

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Smit Jivani, Sarvam Maheshwari, Sunita Sarawagi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Overconfident Intern"

Imagine you have a very smart but inexperienced intern (a Large Language Model or LLM) whose job is to translate your English questions into a specific computer language called SQL, which databases understand.

When you ask a simple question like "Show me sales from last week," the intern usually does a great job. But when you ask a complex or slightly unusual question, the intern starts to guess. Sometimes the guess is wrong, and the computer crashes or gives the wrong answer. This is especially bad for big companies (enterprises) that have their own private databases that the intern has never seen before.

The current best trick to help the intern is In-Context Learning (ICL). This is like giving the intern a cheat sheet with a few examples of past questions and their correct answers right before they answer a new question. It helps, but the paper shows that even with this cheat sheet, the intern still makes mistakes on about 30-40% of the questions.

The Solution: The "Template Library" (TeCoD)

The authors, Smit Jivani, Sarvam Maheshwari, and Sunita Sarawagi, noticed something important about how people ask questions in real life: We repeat ourselves.

If you work at a bank, you might ask, "How many customers in London have over $5,000?" today. Next week, you might ask, "How many customers in Paris have over $10,000?" The words change, but the structure of the question is identical.

The team built a system called TeCoD (Template Constrained Decoding) that acts like a Master Recipe Book for these recurring questions.

How TeCoD Works (The Three Steps)

1. The Recipe Extraction (Template Creation)
Instead of saving every single question and answer exactly as they are, TeCoD looks at past correct answers and strips out the specific details (like "London" or "$5,000"), leaving behind a "skeleton" or template.

Analogy: Imagine a cookie cutter. The specific shape of the cookie (the numbers and names) changes, but the cutter (the template) stays the same. TeCoD saves the cutter.

2. The Matchmaker (Template Selection)
When a new question comes in, TeCoD doesn't just guess. It runs a special "Matchmaker" (a smart AI model) to see if the new question fits any of the saved cookie cutters.

The Innovation: The paper says simple math (comparing word similarity) isn't good enough. So, they trained their Matchmaker to act like a logic detective. It asks: "Does this new question logically fit the pattern of this old recipe?"
Result: This step is incredibly accurate (91%), ensuring they only use a template if it's a perfect fit.

3. The Strict Chef (Constrained Decoding)
If a match is found, TeCoD doesn't let the intern guess the answer freely. Instead, it forces the intern to fill in the blanks of the recipe using a strict grammar rule.

Analogy: Imagine you are filling out a form. You can't write whatever you want; you must follow the lines and the specific type of ink allowed. TeCoD forces the AI to only generate the specific words that fit the "cookie cutter" shape, preventing it from making up weird SQL code that doesn't work.

Why This is a Big Deal

1. It's Much More Accurate
The paper tested this on standard benchmarks (BIRD and Spider).

Old Way (Cheat Sheet only): The intern got about 60% of the recurring questions right.
TeCoD Way: The system got nearly 90% right.
Metaphor: It's like upgrading from a student guessing on a test to a chef following a proven recipe.

2. It's Faster
Because the system knows the structure of the answer in advance, it doesn't have to "think" about the whole sentence from scratch. It only has to fill in the blanks.

Result: The system is 2.2 times faster for these matching questions.

3. It Doesn't Need to "Re-Learn" Everything
Usually, to get better at a specific company's database, you have to retrain the AI (fine-tuning), which is expensive and slow. TeCoD doesn't need that. It just adds new "recipes" to its library as it sees them. It's like adding a new page to a cookbook rather than rewriting the whole book.

What Happens When There Is No Match?

What if you ask a brand new question that doesn't fit any recipe?

TeCoD has a safety net. If the Matchmaker says, "No, this doesn't fit any of our recipes," the system simply falls back to the standard method (the cheat sheet/In-Context Learning). It doesn't break; it just reverts to the slower, less accurate method for that specific question.

The Bottom Line

The paper argues that for real-world businesses, most questions are just variations of questions asked before. TeCoD exploits this by turning past successes into rigid templates. By forcing the AI to stick to these templates when possible, they get near-perfect accuracy and faster speeds without needing to retrain the AI model.

Key Takeaway: Don't let the AI guess the whole answer every time. If you've seen the question before, give it a template to fill in, and it will be much more reliable.

1. Problem Statement

While Large Language Models (LLMs) have improved Text-to-SQL generation, their deployment in enterprise environments faces two critical challenges:

Inconsistent Accuracy: Performance varies significantly across different database schemas, often failing on complex or unseen schemas. Standard benchmarks (like Spider or BIRD) often evaluate zero-shot or few-shot performance on unseen schemas, which does not reflect the reality of enterprise workloads where specific schemas are queried repeatedly.
Invalid SQL Generation: LLMs frequently generate syntactically invalid SQL or SQL that executes but returns incorrect results due to hallucinations in constants or logic.
Limitations of Current Adaptation:
- Fine-tuning: Requires large labeled datasets, is expensive, and risks catastrophic forgetting.
- In-Context Learning (ICL): While effective, it is inconsistent. The paper demonstrates that even with relevant examples in the prompt, LLMs often fail to replicate the correct SQL structure if the query differs slightly in constants (e.g., changing a city name or a number), leading to structural errors.

Core Premise: Enterprise workloads contain a high frequency of recurring query patterns. While natural language phrasing varies, the underlying SQL structure for common business questions is often identical, differing only in specific literals (constants).

2. Methodology: TeCoD System

The authors propose TeCoD (Template Constrained Decoding), a system that leverages historical labeled workloads to improve accuracy and latency. The system operates in two phases:

A. Preparation Phase (Offline)

Template Extraction: Historical (Natural Language Query, SQL) pairs are converted into templates. Constants and literals in the SQL are masked (e.g., replaced with [string] or [number]), creating a parameterized SQL structure.
Synthetic Augmentation: To create robust search keys, the system uses an LLM to generate multiple synthetic paraphrases of the original Natural Language Query (NLQ) that map to the same template.
Masking & Indexing: The synthetic NLQs are "masked" (removing specific literal mentions) to create a generic search key. These masked NLQs are embedded and indexed alongside their corresponding compiled grammars.
Grammar Compilation: The SQL template is converted into a flexible Context-Free Grammar (CFG) or Regular Expression. Unlike fixed strings, this grammar allows for variations in keyword casing, whitespace, and formatting to align with the specific LLM's tokenization preferences.

B. Inference Phase (Online)

Template Matching:
- When a user query arrives, the system first retrieves top- $k$ candidate templates using cosine similarity on embeddings.
- NLI Validation: A fine-tuned Natural Language Inference (NLI) model (e.g., DistilBERT or Qwen) validates the candidates. It determines if the user query entails the masked template (meaning the SQL structure matches), is neutral, or contradicts it. This step filters out false positives that simple similarity search might catch.
Conditional Generation:
- If a match is found: The system triggers Template Constrained Decoding.
- If no match is found: The system falls back to standard In-Context Learning (ICL) or Zero-Shot generation.
Partitioned Decoding (Efficiency Optimization):
- Instead of decoding the entire SQL token-by-token under strict grammar constraints (which is computationally expensive), TeCoD uses a two-phase approach:
  - Phase 1 (Static): The static parts of the template (keywords, structure) are pre-computed and their token IDs are cached. The LLM's KV-cache is initialized with these tokens.
  - Phase 2 (Dynamic): The LLM only generates the specific masked literals (strings/numbers) using constrained decoding with a simple regex, guided by the pre-computed context.
- This significantly reduces latency by avoiding redundant computation for the static parts of the query.

3. Key Contributions

Template Constrained Decoding Paradigm: Introduced a novel approach to Text-to-SQL that combines historical workload patterns with grammar-constrained decoding to ensure syntactic validity and structural consistency.
Robust Template Matcher: Designed a high-precision matching module using an NLI model and synthetic data augmentation. This achieves ~~91% selection/rejection accuracy, significantly outperforming simple cosine similarity baselines (~~73%).
Efficient Partitioned Decoding: Developed a two-phase decoding algorithm that separates static template tokens from dynamic literals. This reduces inference latency by 2.2× compared to standard constrained decoding libraries while maintaining accuracy.
Flexible Grammar Representation: Proposed converting SQL templates into flexible grammars (handling case, whitespace, and formatting variations) rather than fixed strings, ensuring the LLM can generate valid SQL regardless of its specific formatting biases.

4. Results and Evaluation

The system was evaluated on the BIRD and Spider benchmarks using various LLMs (including CodeS, Llama-3, Granite, and XiYanSQL).

Accuracy Gains:
- For matched queries (queries fitting a stored template), TeCoD achieved ~90% Execution Match (ExM) accuracy, compared to ~60% for standard In-Context Learning (ICL).
- This represents an accuracy jump of up to 36% over ICL baselines.
- TeCoD allowed smaller models (e.g., Granite-8B) to match or exceed the performance of larger, specialized models (e.g., QwenCoder-14B).
Latency Improvements:
- TeCoD reduced inference latency by 1.5× to 2.2× compared to standard constrained decoding methods.
- For reasoning models (like Arctic-R1), latency dropped from ~20 seconds to ~6 seconds per query.
Robustness:
- The system maintained high accuracy even when SQL templates had varying formatting (case, spacing), whereas "Fixed Template" approaches failed significantly under formatting perturbations.
Real-World Relevance: Analysis of a large bank's workload showed that >50% of queries could be matched to a recurring template, validating the system's applicability to enterprise settings.

5. Significance

Enterprise Viability: TeCoD offers a practical solution for enterprises with private schemas and limited labeled data. It improves accuracy without requiring expensive fine-tuning, making it suitable for multi-tenant environments.
Reliability: By enforcing structural constraints via grammar, TeCoD eliminates the risk of generating syntactically invalid SQL, a major hurdle in production deployment.
Scalability: The system is designed for incremental learning; as new expert-verified queries arrive, they can be converted into templates and added to the pool, continuously improving coverage and accuracy over time.
Cost-Effectiveness: By enabling smaller, faster LLMs to achieve high accuracy through constrained decoding, TeCoD reduces the computational cost and latency associated with using massive, specialized models for Text-to-SQL tasks.

In conclusion, TeCoD bridges the gap between the flexibility of LLMs and the rigidity required for reliable database interactions, proving that leveraging recurring patterns in enterprise workloads is a highly effective strategy for boosting Text-to-SQL performance.