💬 NLP

SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks

The paper introduces SPENCE, a syntactic probing framework that reveals significant training contamination in older NL2SQL benchmarks like Spider through sensitivity to syntactic variants, while demonstrating that newer datasets like BIRD remain largely uncontaminated.

Original authors: Mohammadtaher Safarzadeh, Hitesh Laxmichand Patel, Afshin Orojlooyjadid, Graham Horwood, Dan Roth

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mohammadtaher Safarzadeh, Hitesh Laxmichand Patel, Afshin Orojlooyjadid, Graham Horwood, Dan Roth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher giving a final exam to a class of very smart students (the AI models). The exam consists of math word problems. The students get perfect scores, so you assume they are geniuses who truly understand math.

But then, a suspicious thought crosses your mind: Did they just memorize the answers to this specific exam, or do they actually understand the math?

This is exactly the problem the paper "SPENCE" tackles, but instead of math word problems, it's about translating human questions into computer database queries (SQL).

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Cheat Sheet" Effect

Large Language Models (LLMs) are like students who have read every book in the library. When they take a test (a benchmark like Spider or BIRD), they often score incredibly high.

The worry is Contamination. This happens when the "exam questions" the students are taking were accidentally included in the books they studied earlier. If a student memorized the exact wording of the question, they can answer it perfectly without actually knowing how to solve the problem.

The Old Exams (Spider, SParC, CoSQL): These are like exams from 2018 and 2019. They have been around for a long time, so it's highly likely the AI "students" have seen them before.
The New Exam (BIRD): This is a fresh exam from 2023. It's less likely the AI has seen it yet.

2. The Solution: The "Paraphrase Game" (SPENCE)

The authors created a tool called SPENCE to test if the students are cheating (memorizing) or actually learning.

How it works:
Imagine you give the student a question: "How many singers are older than 20?"
If they get it right, you don't just move on. You play a game of "How many ways can we say this?"

SPENCE takes that original question and rewrites it 10 times, making it slightly different each time:

Rank 1: "What is the count of singers over 20?" (Very similar)
Rank 5: "List the countries of singers who are past the age of 20." (Different words, same meaning)
Rank 10: "I need to know the number of vocalists whose age exceeds two decades." (Very different structure, same meaning)

The Test:

If the student is genuinely smart, they should get the answer right for all 10 versions. They understand the concept, not just the words.
If the student is cheating (memorizing), they will ace the first few versions (because they look like the cheat sheet) but will fail the later, stranger versions because they don't recognize the pattern.

3. The Results: Who Passed the Test?

The researchers ran this "Paraphrase Game" on several famous AI models using four different datasets.

The Old Datasets (Spider, SParC, CoSQL):
- Result: The models started strong but quickly crashed as the questions got reworded.
- The Metaphor: It's like a student who can recite the answer to Question #1 perfectly but freezes when you ask the same question using a thesaurus.
- Conclusion: These older benchmarks are likely "contaminated." The models are likely memorizing the specific phrasing of these old questions rather than learning how to translate language to code.
The New Dataset (BIRD):
- Result: The models stayed steady. Even when the questions were rewritten in weird ways, their performance didn't drop much.
- The Metaphor: These students actually understood the math. They could handle the rewording because they hadn't memorized the specific exam questions yet.
- Conclusion: The newer benchmarks are a more honest test of real intelligence.

4. Why This Matters

The paper argues that we can't just look at the top scores on leaderboards anymore. A high score might just mean the AI has a "photographic memory" of the test questions.

SPENCE is like a "lie detector" for AI benchmarks. By shaking up the wording of the questions, it reveals whether the AI is truly generalizing (learning the rules) or just overfitting (memorizing the answers).

Summary in One Sentence

SPENCE is a tool that rewrites test questions in increasingly strange ways to see if AI models are actually smart enough to understand the meaning, or if they are just cheating by memorizing the exact wording of old exams.

The takeaway: If an AI fails when you change the words, it probably didn't learn the lesson; it just memorized the homework.

1. Problem Statement

Large Language Models (LLMs) have demonstrated high performance on Natural Language to SQL (NL2SQL) benchmarks. However, there is a growing concern that these reported accuracy scores are artificially inflated due to data contamination. Contamination occurs when models have been exposed to benchmark queries (or structurally similar patterns) during pretraining or fine-tuning, leading to memorization rather than genuine compositional generalization.

Existing detection methods often rely on string overlap or simple retrieval checks, which may miss "near-memorization" where models recognize the underlying structure but fail on syntactic variations. The paper addresses the need for a robust method to distinguish between true generalization and contamination-like behavior (memorization of specific query patterns) in NL2SQL tasks.

2. Methodology: The SPENCE Framework

The authors introduce SPENCE (Syntactic Probing and Evaluation of NL2SQL Contamination Effects), a controlled framework designed to detect contamination by measuring model sensitivity to syntactic divergence.

Core Workflow

Data Selection: The study focuses on four widely used NL2SQL benchmarks with varying release dates:
- Spider (Sept 2018)
- SParC (June 2019)
- CoSQL (Oct 2019)
- BIRD (May 2023)
Paraphrase Generation:
- For each natural language (NL) query in the development sets, the system generates 10 syntactic paraphrases using an LLM (GPT-5).
- Crucially, the generation prompt includes the Gold SQL and the Database Schema to ensure semantic fidelity (the paraphrase must mean the same thing as the original).
Syntactic Ranking (Tree Edit Distance):
- Paraphrases are parsed into dependency trees.
- Tree Edit Distance (TED) is calculated between the original query's tree and each paraphrase's tree.
- Paraphrases are ranked from 1 (closest syntactic similarity) to 10 (most distant).
- Filtering: Embedding-based cosine similarity is used to filter out semantically divergent paraphrases, ensuring that accuracy drops are due to syntactic changes, not meaning shifts.
Evaluation:
- The target NL2SQL models are evaluated on both the original queries and the ranked paraphrases.
- Metric: Execution Accuracy (does the generated SQL run successfully and return the correct result?).
- Analysis: The change in accuracy ( $\Delta$ Accuracy) is plotted against the paraphrase rank.
Statistical Quantification:
- Kendall's $\tau$ is computed to measure the correlation between paraphrase rank and accuracy change.
- Bootstrap Confidence Intervals (95%) are used to determine statistical significance.
- Hypothesis:
  - $H_0$ (Robustness): Accuracy is invariant to paraphrasing (no contamination).
  - $H_1$ (Contamination): Accuracy degrades as syntactic distance increases (indicating memorization).

3. Key Contributions

SPENCE Framework: A novel, model-agnostic pipeline for quantifying contamination via controlled syntactic probing.
Syntactic vs. Semantic Isolation: Unlike previous methods that conflate lexical overlap with contamination, SPENCE specifically isolates structural divergence (via TED) while controlling for semantic equivalence.
Temporal Gradient Analysis: The study correlates benchmark release dates with contamination sensitivity, providing empirical evidence that older benchmarks are more susceptible to memorization effects.
Comprehensive Benchmarking: Evaluation across six diverse LLMs (including GPT-4o-mini, O3-mini, LLaMA-3.1/3.3, and GPT-OSS variants) and four major datasets.

4. Key Results

The results reveal a clear temporal gradient in model robustness:

Older Benchmarks (Spider, SParC, CoSQL):
- Exhibit strong negative Kendall's $\tau$ values (typically between -0.6 and -0.95).
- Observation: Models show high accuracy on the original queries and close paraphrases (Rank 1-3) but suffer sharp, monotonic degradation as syntactic distance increases (Rank 8-10).
- Magnitude: On Spider, accuracy drops by 10–15 percentage points for distant paraphrases across all models.
- Interpretation: This suggests models have memorized specific query patterns from these older, widely used datasets.
Newer Benchmark (BIRD):
- Exhibits near-zero Kendall's $\tau$ values (ranging from -0.11 to -0.46, often statistically indistinguishable from zero).
- Observation: Model performance remains stable across all paraphrase ranks.
- Interpretation: As a newer dataset (released 2023), BIRD shows minimal evidence of contamination, suggesting models are generalizing rather than memorizing.
Control Experiments:
- Query Length: Ruled out as a confounding factor; query lengths were consistent across ranks.
- Lexical Overlap: The degradation persists even when controlling for Jaccard lexical similarity, confirming the issue is syntactic, not just lexical.
- Schema Linking: Manual inspection confirmed that paraphrases retained correct schema links (table/column names), ruling out schema-linking failure as the cause of errors.
- Paraphrase Generator: Re-running experiments with a different generator (LLaMA-4 Maverick) yielded similar trends, proving the signal is not an artifact of the paraphrase generator.

5. Significance and Implications

Reliability of Benchmarks: The paper demonstrates that standard leaderboard scores for older NL2SQL benchmarks (like Spider) may be significantly inflated due to contamination. High scores may reflect memorization of surface forms rather than true reasoning capabilities.
Evaluation Best Practices: SPENCE provides a necessary complement to standard evaluation. A model that performs well on original queries but fails on syntactic variants is likely overfitting.
Future Directions: The authors suggest that future benchmarks must be designed with "contamination resistance" in mind, and evaluation protocols should include syntactic probing to verify generalization.
Broader Impact: The methodology can be extended to other domains (text-to-code, QA) to audit LLMs for data leakage and memorization, ensuring that AI systems are truly learning to reason rather than simply recalling training data.

In conclusion, SPENCE offers a rigorous, empirical method to detect "contamination-like" behavior, revealing that while LLMs appear powerful on static benchmarks, their robustness to syntactic variation varies significantly depending on the age and exposure history of the dataset.