Cost Trade-offs of Reasoning and Non-Reasoning Large Language Models in Text-to-SQL

This paper demonstrates that reasoning Large Language Models significantly reduce cloud query execution costs and data consumption compared to non-reasoning models in Text-to-SQL tasks, while revealing that execution time is a poor proxy for cost efficiency and highlighting the substantial financial risks posed by non-reasoning models' tendency to generate inefficient queries.

The Gist

Here is an explanation of the paper, translated from "academic speak" into everyday language with some creative analogies.

🧠 The Big Idea: Speed vs. Price

Imagine you hire two different chefs to cook a meal for a party.

• Chef A (The "Reasoning" Chef) takes a long time to think, taste, and plan the recipe before cooking.
• Chef B (The "Standard" Chef) rushes to the stove immediately, throwing ingredients together as fast as possible.

In the world of AI and databases, we usually care about how fast the food is ready (Execution Time). But this paper asks a different question: "How much did the ingredients cost?"

The authors found that while the "Standard" chefs are fast, they often waste massive amounts of expensive ingredients. The "Reasoning" chefs, who take a moment to think first, actually save you a lot of money, even if they take a tiny bit longer to start cooking.

---

🏢 The Setting: The Cloud Kitchen

The researchers tested this in a "Cloud Kitchen" called Google BigQuery.

• The Menu: They used a massive dataset called StackOverflow (230 GB of data), which is like a library containing every question and answer ever posted on the site.
• The Bill: In this kitchen, you don't pay by the hour. You pay by how much food you eat. If you scan a whole table of data but only need one column, you pay for the whole table. It's like paying for a whole pizza just because you wanted one slice.

🧪 The Experiment

The team asked 6 different AI chefs (3 "Reasoning" models and 3 "Standard" models) to write 30 different SQL queries (recipes) to get specific information from that 230 GB library.

They measured:

1. Correctness: Did the recipe actually make the dish?
2. Speed: How fast was it served?
3. Cost: How many bytes of data did the AI scan to get the answer? (This is the money metric).

---

🔑 The Big Discoveries

1. The "Thinker" Chefs Save Money

The Reasoning Models (the ones that pause to think) were 44.5% cheaper to run than the Standard Models.

• Analogy: Imagine you need to find a specific book in a library.
  • The Standard Model runs into the library, grabs every single book off the shelves, and starts flipping through them until it finds the one. It's fast to start, but it moves a mountain of books (high cost).
  • The Reasoning Model walks to the desk, asks the librarian exactly which shelf the book is on, and walks straight there. It moves fewer books (low cost).
• Result: The Reasoning models scanned significantly less data, saving money, while still getting the right answer 96%–100% of the time.

2. Speed is a Trap (The "Fast & Expensive" Illusion)

The biggest surprise was that speed does not equal savings.

• The Correlation: The paper found a very weak link (0.16) between how fast a query ran and how much it cost.
• Analogy: Imagine a race car driver who drives 200 mph but gets stuck in a traffic jam of 100 miles. They finish the race quickly, but they burned a fortune in gas.
• Reality: A query can finish in 2 seconds because the cloud computer is super powerful (parallel processing), but if it scanned 36 GB of data to do it, it cost a fortune. Don't trust speed as a sign of efficiency.

3. The "Wild Card" Problem

Some of the Standard Models were unpredictable.

• The Outlier: One model (GPT-5.1) had a "bad day" where it generated a query that scanned 36 GB of data for a single question. That's 20 times more than the best model!
• Why? It forgot to filter by date (missing partition filters) or grabbed unnecessary columns (like SELECT *).
• Analogy: It's like asking a waiter, "How many people ordered pizza?" and the waiter goes to every single table in the restaurant, asks everyone what they ate, and brings you a list of 500 people, even though only 5 ordered pizza.

4. The Mistakes They Made

The researchers found common "bad habits" in the Standard models:

• SELECT * (The "Grab Everything" habit): Instead of asking for just the "Name" column, the AI asked for the "Name," "Address," "Phone," "Email," and "Full Biography" columns.
• Missing Filters: Asking for "All questions ever" instead of "Questions from 2023."
• Cartesian Products: Accidentally creating a "Cross Join," which is like taking every user and pairing them with every single post, creating a massive, useless list of combinations.

---

💡 What Should You Do? (The Takeaway)

If you are building an AI system that talks to databases (Text-to-SQL) for a business, here is the advice from the paper:

1. Pick the "Thinkers": Use Reasoning Models. They might cost a tiny bit more to think (inference cost), but they save you a huge amount of money on execution (database cost).
2. Stop Watching the Clock: Don't just look at how fast the query runs. A fast query can still bankrupt your budget. Look at how much data was scanned.
3. Install Safety Guards: Set up automatic rules that say, "If a query tries to scan more than 10 GB, stop it!" This prevents the AI from accidentally ordering a 36 GB pizza.
4. Check for Bad Habits: Make sure the AI isn't using SELECT * or forgetting to filter by dates.

🏁 In a Nutshell

This paper proves that in the cloud, being smart is cheaper than being fast. The AI models that take a moment to "think" before they speak generate cleaner, more efficient code that saves companies real money, while the "fast" models often waste resources by scanning way more data than necessary.

Technical Summary

Here is a detailed technical summary of the paper "Cost Trade-offs of Reasoning and Non-Reasoning Large Language Models in Text-to-SQL" by Deochake and Mukhopadhyay.

1. Problem Statement

While Large Language Models (LLMs) have achieved high accuracy in generating SQL queries (Text-to-SQL), existing evaluation benchmarks (e.g., Spider, BIRD) primarily focus on correctness and execution time.

• The Gap: In cloud data warehouses (like Google BigQuery, Snowflake), costs are consumption-based, determined by the volume of data scanned (bytes processed) rather than wall-clock execution time.
• The Misconception: The paper argues that execution time is a poor proxy for cost. A query can execute quickly due to massive parallelization but incur high costs if it scans unnecessary data.
• The Risk: Inefficient SQL generation (e.g., missing partition filters, SELECT *) can lead to exponential cost increases in production environments, yet current benchmarks do not measure this financial impact.

2. Methodology

The authors conducted a controlled, cloud-native experiment to evaluate the cost efficiency of LLM-generated SQL.

• Platform: Google BigQuery (US multi-region), chosen for its consumption-based pricing model ($6.25 per TB scanned).
• Dataset: The StackOverflow public dataset (230 GB), comprising 597 million rows across 6 tables (e.g., posts, users, votes). This dataset was selected for its real-world complexity and scale.
• Workload: 30 natural language questions categorized by complexity:
  • Simple (10): Single-table filtering/aggregation.
  • Medium (10): Multi-table joins (2-3 tables).
  • Complex (10): Subqueries, window functions, CTEs, or 4+ table joins.
• Models Evaluated: Six LLMs from three vendors, split evenly into Reasoning (extended thinking) and Standard (optimized for speed) categories:
  • Reasoning: Opus 4.5R (Anthropic), GPT-5.2R (OpenAI), Gemini ProR (Google).
  • Standard: Sonnet 4.5 (Anthropic), GPT-5.1 (OpenAI), Gemini Flash (Google).
• Experimental Setup:
  • Zero-shot prompting with full schema definitions.
  • Query caching explicitly disabled to ensure fresh scans.
  • 180 total executions (30 queries × 6 models).
• Metrics:
  • Primary: Bytes Processed ($B_p$), Estimated Cost ($C$).
  • Secondary: Execution Time, Slot Seconds (compute), Shuffle Volume, Correctness.
  • Derived: Coefficient of Variation (CV) for cost consistency.

3. Key Contributions

1. Cloud-Native Cost Evaluation: Introduced a methodology for measuring Text-to-SQL performance based on bytes processed and financial cost on production infrastructure, moving beyond local execution time.
2. Reasoning vs. Standard Analysis: Provided empirical evidence that reasoning models generate significantly more cost-efficient SQL than standard models.
3. Variance Quantification: Identified extreme cost variance in standard models, with outliers scanning up to 36 GB per query (20× the average of the best model).
4. Inefficiency Pattern Identification: Characterized specific SQL anti-patterns (e.g., missing partition filters, SELECT *, unintended cross joins) that drive cloud costs.
5. Decoupling of Time and Cost: Demonstrated that execution time correlates weakly with query cost ($r = 0.16$), invalidating speed as a metric for cost efficiency.

4. Key Results

A. Cost Efficiency of Reasoning Models

• 44.5% Cost Reduction: Reasoning models processed 2,140 MB on average, compared to 3,857 MB for standard models.
• Financial Impact: This translates to an average cost of $0.0134/query for reasoning models vs. $0.0241/query for standard models.
• Statistical Significance: The difference is statistically significant ($p = 0.003$) with a medium effect size (Cohen's $d = 0.52$).
• Mechanism: Reasoning models applied partition filters in 89% of applicable queries versus 67% for standard models, and used explicit column lists more frequently.

B. Cost Variance and Outliers

• High Variance in Standard Models: The standard model GPT-5.1 exhibited the highest variance (Standard Deviation: 11,659 MB).
• Extreme Outliers: GPT-5.1 produced four queries exceeding 5 GB, with one specific query scanning 36.6 GB (costing ~$0.23) due to missing filters and selecting full post bodies.
• Predictability: Reasoning models showed lower coefficients of variation (CV 1.38–1.58) compared to standard models (CV 1.85–1.93), indicating more predictable billing.

C. Correctness vs. Efficiency

• Correctness: Both model types achieved high correctness (96.7% to 100%).
• Differentiation: Since correctness is comparable, efficiency is the primary differentiator for production deployment.

D. Correlation Analysis

• Time vs. Cost: The correlation between execution time and bytes processed is weak ($r = 0.16$). A fast query can be expensive if it scans large datasets in parallel.
• SQL Length vs. Cost: No correlation ($r = 0.05$). A verbose query with explicit columns and filters can be cheaper than a concise SELECT * query.

E. Common Inefficiency Patterns

The study identified specific anti-patterns generated by LLMs:

1. Missing Partition Filters: The most critical issue. In 50% of applicable queries, models failed to filter by date/partition, forcing full table scans.
2. SELECT *: OpenAI models occasionally generated SELECT *, forcing scans of all columns (including large text fields) even when unnecessary.
3. Unintended Cross Joins: Occurred when join conditions were omitted, creating Cartesian products.

5. Significance and Implications

• Financial Risk Mitigation: The paper highlights that deploying Text-to-SQL without cost-awareness can lead to unpredictable and massive cloud bills.
• Model Selection: For analytical workloads, reasoning models are recommended despite potentially higher inference latency, as the savings in execution costs (44.5%) outweigh the inference costs.
• Benchmark Evolution: Future benchmarks must include cloud cost metrics (bytes scanned) rather than just execution time to reflect production realities.
• Operational Guidelines:
  • Implement cost guardrails (pre-execution cost estimation and rejection thresholds).
  • Monitor for anti-patterns (e.g., missing filters, SELECT *).
  • Do not rely on execution time as a proxy for cost efficiency.

Conclusion

This study establishes that reasoning capabilities in LLMs directly translate to financial efficiency in cloud data warehouses. By generating SQL that better utilizes partition pruning and column selection, reasoning models reduce cloud compute costs by nearly half compared to standard models, while maintaining high accuracy. The authors conclude that organizations must shift their evaluation metrics from "speed" to "cost-per-query" to ensure sustainable production deployments.