Technical Summary: SrDetection

Problem Statement

The evaluation of Code Large Language Models (Code LLMs) is increasingly compromised by data leakage, where benchmark samples are inadvertently included in a model's pre-training corpus. When models memorize these samples, their performance metrics are artificially inflated, reflecting memorization rather than genuine generalization. This undermines the fairness of model comparisons and complicates the assessment of true innovation.

Existing detection methods suffer from three primary limitations:

1. Proprietary Data Dependency: Overlap-based methods (e.g., n-gram matching) require access to the model's training corpora, which are typically proprietary and undisclosed.
2. Brittle Heuristics: Timestamp-based filtering assumes that code released after a model's training cutoff cannot be in the corpus. This fails in practice due to code reuse, inheritance, and forking.
3. Threshold Instability: Confidence-based approaches (e.g., Perplexity) rely on external reference sets to calibrate decision thresholds. These thresholds are often tuned on natural language data and do not generalize well to the structural regularities of code, or they rely on the same flawed temporal heuristics.

Methodology: SrDetection

The authors propose SrDetection, a unified, self-referential framework for detecting data leakage in both gray-box (access to model logits) and black-box (access only to model outputs) settings. The core hypothesis is that if a model has memorized a specific code sample, it will exhibit significantly more predictable behavior (lower perplexity or higher generation similarity) on the original surface form compared to semantically equivalent variants.

The framework operates in two main stages:

1. Code Transformation Module

This module generates a set of $n$ semantically equivalent variants ($V = \{x_1, ..., x_n\}$) from an original benchmark sample ($x_0$) using an auxiliary LLM. The transformations are minimal and localized to preserve execution behavior while altering lexical and syntactic forms:

• Function Name Renaming ($\tau_f$): Replaces function names and call sites with semantically equivalent alternatives (e.g., solve $\to$ compute).
• Variable Name Renaming ($\tau_v$): Systematically replaces variable identifiers and parameters (e.g., nums $\to$ arr).
• Test Case Transformation ($\tau_t$): Generates new test inputs semantically similar to the originals, validates them via execution, and replaces the original test cases.

2. Leakage Decision Module

The framework compares the model's behavior on the original sample ($x_0$) against its variants ($V$) without relying on external thresholds.

• Gray-box Setting (Perplexity-based):
  The detection score $S_M(x)$ is the Perplexity (PPL) of the sample. If the model has memorized $x_0$, its PPL will be lower than that of any variant.
  $$\text{Leakage}_{gray}(x_0) = \mathbb{I}[S_M(x_0) < \min_{x_i \in V} S_M(x_i)]$$
  Where $\mathbb{I}$ is the indicator function.

• Black-box Setting (Generation Similarity-based):
  The detection score $S_M(x)$ is the N-gram Overlap (NGO) between the model's generated suffix and the ground-truth suffix when prompted with a prefix. If memorized, the model generates a suffix closer to the ground truth for the original form than for the variants.
  $$\text{Leakage}_{black}(x_0) = \mathbb{I}[S_M(x_0) > \max_{x_i \in V} S_M(x_i)]$$

Controlled Testbed Construction

To rigorously evaluate these methods, the authors constructed a controlled leakage testbed:

1. Dataset Preparation: A public code benchmark was filtered to remove samples already showing signs of leakage (where the original was already the easiest among variants).
2. Simulated Leakage: Models were subjected to continued pre-training on a subset of the data (the "training" split) mixed with a general web corpus, while the remaining data served as the "held-out" test set. This creates a ground truth where leakage is explicitly controlled and known.

Key Contributions

1. Self-Referential Framework: SrDetection eliminates the need for external reference data or manually defined thresholds by using within-sample contrast between a code snippet and its semantics-preserving variants.
2. Automatic Augmentation: A tailored method for generating code variants that expose memorization under common edits (renaming, refactoring) while preserving semantic functionality.
3. Controlled Evaluation: The introduction of a training-based testbed that allows for precise measurement of leakage detection performance across different models and training stages, avoiding the unreliability of temporal heuristics.

Experimental Results

The authors evaluated SrDetection against strong baselines (e.g., PPL, Min-K% Prob, Zlib, VeilProbe, DPDLLM) on continued pre-trained Qwen2.5 and LLaMA3.1 models.

• Performance Gains: SrDetection significantly outperformed baselines.
  • Gray-box: Improved average F1 by 21.52 points over strong baselines.
  • Black-box: Improved average F1 by 14.46 points.
• Robustness: Unlike baselines that showed unstable performance or relied on specific epochs, SrDetection remained consistent across different models and training stages.
• Sensitivity: Performance improved with a higher number of variants ($n$), and the method remained stable across different code-to-text mixing ratios in the pre-training corpus.
• Real-world Application: A study of 15 widely used Code LLMs on four benchmarks (APPS, HumanEval, MBPP, BigCodeBench) revealed benchmark-specific leakage patterns. Notably, BigCodeBench showed high leakage rates in the gray-box setting, suggesting behavior-based detection can identify contamination missed by strict overlap analysis.

Significance and Claims

The paper claims that SrDetection addresses a critical gap in Code LLM evaluation by providing a threshold-independent and corpus-independent detection mechanism. By reframing leakage detection as a self-referential problem, the method avoids the brittleness of temporal assumptions and the inaccessibility of training data.

The authors assert that their findings demonstrate leakage is not merely a marginal artifact but a systematic confounder in current evaluation practices. The ability to detect leakage without external references suggests that benchmark-specific contamination patterns are more prevalent and complex than previously understood through overlap-based analyses alone. The work underscores the necessity for leakage-aware benchmark design and reporting to ensure the reliability of Code LLM evaluations.