💬 NLP

Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows

This paper introduces Spider 2.0-AIFunc, a new benchmark of 465 verified instances across 125 real-world databases designed to evaluate large language models' ability to generate AI-native SQL queries on the Snowflake platform, revealing that while top proprietary models achieve 67-70% accuracy, current agent frameworks optimized for traditional text-to-SQL tasks do not effectively transfer to this emerging setting.

Original authors: Tianyang Liu, Canwen Xu, Fangyu Lei, Nikki Lijing Kuang, Jixuan Chen, Tao Yu, Julian McAuley, Zhewei Yao, Yuxiong He

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tianyang Liu, Canwen Xu, Fangyu Lei, Nikki Lijing Kuang, Jixuan Chen, Tao Yu, Julian McAuley, Zhewei Yao, Yuxiong He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, high-tech library (a cloud database) where people store everything from customer reviews to sales records. For years, asking questions about this library required speaking a very strict, robotic language called SQL. It was like trying to order a complex meal at a restaurant by only speaking in a code of numbers and symbols.

Recently, the library owners (like Snowflake) added a new feature: AI Functions. Think of these as "smart assistants" built right into the ordering system. Now, instead of just asking for "all red apples," you can ask the system to "read the reviews of these apples and tell me if people are happy or angry," or "find reviews that sound similar to this one." This turns the database into an AI-Native system, where you can mix standard data commands with smart, language-based thinking in a single sentence.

The Problem: The Old Map Doesn't Work

The researchers noticed that while these new "smart assistants" are powerful, the tests used to train and check computer brains (AI models) were outdated. The old tests only checked if the computer could speak the strict robotic language. They didn't check if the computer could use the new "smart assistants." It was like testing a driver only on a straight, empty road, when the real world now has traffic, pedestrians, and construction zones.

The Solution: Spider 2.0-AIFunc

To fix this, the team created a new, tougher driving test called Spider 2.0-AIFunc.

The Transformation: They took 513 existing "driving tests" (questions about data) and rewrote them. They forced the questions to require the new "smart assistants."
- Old Question: "Show me all negative reviews." (Requires a human to read them first).
- New Question: "Show me all reviews where the AI assistant says the sentiment is negative." (The AI does the reading inside the database).
The Construction Crew: They used a team of AI agents (digital workers) to rewrite these questions. These agents acted like editors and mechanics:
- They checked that the instructions were clear (e.g., "What exactly do you mean by 'negative'?").
- They made sure the "smart assistant" had all the right tools (parameters) to do the job.
- They ran the tests over and over again to make sure the answers were stable and didn't change just because the AI assistant had a bad day.
The Final Exam: The result is a benchmark of 465 verified questions across 125 different databases. It covers six types of "smart assistant" tasks, like sorting reviews by emotion, finding similar text, or extracting specific details from a paragraph.

The Results: Who Passed the Test?

The researchers put 10 different AI models (the "drivers") through this new exam to see how well they could write these complex, AI-powered queries.

The Top Drivers (Proprietary Models): The big, closed-source models (like Claude Opus and Gemini) did the best. They got about 67% to 70% of the answers right. They were good at figuring out which "smart assistant" to use and how to ask it the right question.
The Open-Source Drivers: The best open-source models (like Kimi K2.5) scored around 58%. They were decent, but they made more mistakes.
The Gap: The main reason the open-source models struggled wasn't that they couldn't write the basic code. They stumbled on the "smart" parts:
- Misunderstanding the rules: Picking the wrong table or column.
- Bad instructions: Forgetting to tell the AI assistant exactly what to look for (e.g., not listing all the possible categories for a classification task).
- Logic errors: Mixing up the order of operations (e.g., filtering data after asking the AI instead of before).

The Surprising Discovery: Less is More

The researchers also tested whether using complex "agent frameworks" (elaborate toolkits that help the AI plan its steps) helped.

The Finding: Surprisingly, the most complex toolkits didn't help much. In fact, a minimal setup—just a simple tool to look at the database and a tool to run the query—performed just as well as, or even better than, the fancy frameworks.
The Metaphor: It's like giving a master chef a tiny, simple knife versus a massive, complicated Swiss Army knife. For this specific type of cooking (AI-Native SQL), the master chef didn't need the extra gadgets; they just needed to know how to use the knife. The complex frameworks designed for old-school data tasks actually got in the way or didn't add value for these new, smart tasks.

Summary

This paper introduces a new, realistic test for AI models that checks if they can use modern "AI-powered" database features. It found that while the best AI models are getting good at this, there is still a gap between the top-tier models and the open-source ones. The key to success isn't using more complex planning tools, but rather getting the basic instructions and parameters for the AI functions exactly right.

Technical Summary: Spider 2.0-AIFunc

Problem Statement

Major cloud data platforms (e.g., Snowflake, BigQuery, Databricks) have begun exposing Large Language Model (LLM) capabilities as native SQL functions. These "AI-native" functions allow analysts to perform semantic operations—such as classification, sentiment analysis, similarity search, and information extraction—directly within standard SQL queries. However, existing Text-to-SQL benchmarks (e.g., Spider 1.0, Spider 2.0, BIRD) evaluate only conventional SQL over structured data. They provide no signal regarding a model's ability to generate AI-native SQL, which requires not only standard query construction but also the correct selection, parameterization, and integration of AI function calls.

Methodology

The authors introduce Spider 2.0-AIFunc, a benchmark designed to evaluate Text-to-SQL capabilities in the context of AI-native workflows. The construction and evaluation methodology involves three core components:

1. Benchmark Construction via Agent Pipeline

The benchmark is derived from 513 instances of the Spider2-Snow dataset (an enterprise-level subset of Spider 2.0 hosted on Snowflake). The authors employ a reusable, agent-based pipeline (using Claude Opus 4.5) to transform these traditional tasks into AI-native versions:

Transformation: Agents rewrite the target SQL to incorporate Snowflake Cortex AI Functions and simultaneously refine the natural language instructions to make the intended AI usage explicit.
Ambiguity Reduction: For tasks with multiple valid gold SQLs, instructions are refined to specify the exact AI function and parameters (e.g., complete class label sets for classification) to ensure a single correct answer.
Two-Round Process:
- Main Round: Agents select AI functions freely, resulting in a distribution dominated by AI_CLASSIFY, AI_FILTER, and AI_SIMILARITY.
- Diversity Round: Agents are instructed to prioritize underrepresented functions (AI_SENTIMENT, AI_EXTRACT, AI_AGG) to ensure broad coverage of the six supported AI function types.

2. Determinism Verification

A critical challenge in benchmarking AI functions is execution determinism, as LLM outputs can exhibit minor inconsistencies even at zero temperature. To address this:

Multi-Pass Verification: Each candidate instance undergoes four passes of execution (totaling $\ge$ 35 executions). A verification agent repairs instances with inconsistent results by adjusting SQL, instructions, or evaluation configurations.
Temporal Stability: Surviving instances undergo a final stability check involving 30 executions across three temporally separated time windows. Instances producing inconsistent results across these windows are discarded.
Final Dataset: The process yields 465 verified instances across 125 real-world databases.

3. Evaluation Protocol

Metric: Execution Accuracy (EX). Unlike traditional benchmarks where gold results are pre-computed and static, Spider 2.0-AIFunc requires both the gold SQL ( $S_n$ ) and the predicted SQL ( $\hat{S}_n$ ) to be executed within the same time window to ensure they are evaluated against the same underlying model state.
Comparison Logic: Results are compared using column-subset matching (gold columns must appear in predicted results; predicted columns may contain extras).
Frameworks: Models are evaluated using the Spider-Agent framework with a minimal tool set (bash for schema exploration, SQL execution, and task termination). The setup includes reference documentation for Snowflake Cortex AI Functions.

Key Contributions

Spider 2.0-AIFunc Benchmark: The first benchmark specifically for AI-native Text-to-SQL, covering 465 verified instances, 125 databases, and six types of AI functions (Classification, Filtering, Sentiment, Similarity, Extraction, Aggregation).
Agent-Based Construction Pipeline: A novel, automated pipeline that transforms existing enterprise Text-to-SQL tasks into AI-native versions while simultaneously refining instructions and verifying execution determinism.
Empirical Analysis of AI-Native SQL: A comprehensive evaluation of 10 state-of-the-art models (5 proprietary, 5 open-source) and 3 agent frameworks, revealing specific failure modes and the limitations of traditional Text-to-SQL strategies in this new setting.

Results

The evaluation of 10 models on Spider 2.0-AIFunc yields the following findings:

Proprietary vs. Open-Source Gap: Proprietary models significantly outperform open-source models.
- Top Proprietary: Claude Opus 4.6 (70.3%), Claude Sonnet 4.6 (69.0%), and Gemini 3.1 Pro (67.1%).
- Top Open-Source: Kimi K2.5 (58.1%).
- The gap is driven primarily by errors in predicate specification, schema grounding, and AI function parameterization.
Execution Success: Most models achieve high "Exec." rates (90–99%), indicating they can generate syntactically valid AI-native SQL that runs without error. The performance gap lies in semantic correctness.
Agent Framework Transferability: Agent frameworks designed for traditional Text-to-SQL challenges (e.g., schema retrieval, table selection) do not transfer effectively to AI-native SQL.
- Complex frameworks (AutoLink, ReFoRCE, DSR-SQL) matched but did not exceed the performance of a minimal Spider-Agent setup when using the same backbone model (Claude Sonnet 4.6).
- This suggests that strategies for traditional SQL scaffolding are less critical than the ability to make precise semantic decisions regarding AI function usage (e.g., correct parameterization, filtering order).
Interaction Behavior: Open-source models require significantly more interaction rounds (19–30 vs. 10–12 for proprietary) and more SQL execution attempts to reach a correct result, often failing to terminate correctly.

Significance and Claims

The paper claims that Spider 2.0-AIFunc represents a necessary evolution in Text-to-SQL evaluation, moving from purely structured data manipulation to AI-native workflows that integrate semantic operations directly into SQL.

The authors emphasize that:

Current Benchmarks are Insufficient: Existing benchmarks fail to capture the complexity of integrating LLM capabilities into enterprise data pipelines.
Determinism is Critical: The verification protocol highlights that benchmarking AI functions requires rigorous stability checks due to the inherent non-determinism of LLM outputs, even in controlled settings.
Strategy Shift: The failure of complex agent frameworks to outperform minimal setups suggests that the primary bottleneck in AI-native SQL is not schema retrieval or decomposition, but rather the precise specification and parameterization of AI functions. As models improve, the value of elaborate scaffolding for traditional SQL components diminishes relative to the need for accurate semantic function invocation.

The work serves as a foundational step for evaluating and improving models capable of building "AI-native" data analysis workflows, a capability increasingly required by enterprise data platforms.