💻 computer science

Learned or Memorized ? Quantifying Memorization Advantage in Code LLMs

This paper introduces a perturbation-based method to quantify memorization advantages in code LLMs, revealing that memorization risks are highly dependent on specific models and tasks while demonstrating that widely suspected benchmarks like CVEFixes and Defects4J actually exhibit low memorization, suggesting models rely more on learned generalization.

Original authors: Djiré Albérick Euraste, Kaboré Abdoul Kader, Jordan Samhi, Earl T. Barr, Jacques Klein, Tegawendé F. Bissyandé

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Djiré Albérick Euraste, Kaboré Abdoul Kader, Jordan Samhi, Earl T. Barr, Jacques Klein, Tegawendé F. Bissyandé

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Did the Student Cheat?

Imagine you are a teacher giving a final exam to a class of brilliant students (these students are AI Code Models). You want to know if they truly understand the math and logic behind the problems, or if they just memorized the answers from a specific textbook they were allowed to study.

In the world of AI, this is a huge problem. Because we don't know exactly what data these AI models were trained on, we often suspect they are "cheating" by having seen the test questions before. If they get a perfect score, is it because they are geniuses, or because they memorized the answer key?

This paper introduces a clever way to test if the students are memorizing or learning.

The Method: The "Slight Nudge" Test

Instead of just giving the students the same test again, the researchers decided to play a game of "spot the difference." They used a technique called Perturbation.

Think of it like this:

The Original Question: "Write a function to calculate the sum of numbers from 1 to 10."
The Nudge (Perturbation): "Write a function to calculate the total of integers starting at 1 and ending at 10."

To a human, these are the exact same question. The meaning hasn't changed at all. But to a robot that just memorized the answer to the first version, the second version might look like a completely different puzzle.

The Analogy of the Parrot vs. The Musician:

The Parrot (Memorization): If you ask a parrot to say "Hello," it says "Hello." If you ask it to say "Hullo" (a slight change), it might freeze or say something nonsense. It only knows the exact sound it heard.
The Musician (Generalization): If you ask a musician to play a song, and you ask them to play it slightly faster or in a different key, they can still play it. They understand the music, not just the notes.

The researchers "nudged" the code prompts in many ways (changing variable names, rephrasing instructions, shuffling code structure) and watched how the AI's performance dropped.

Big Drop in Performance? The AI was likely memorizing (The Parrot).
Small Drop in Performance? The AI was generalizing (The Musician).

What They Found: The Results

The researchers tested 8 different AI models on 19 different types of coding tasks. Here is what they discovered, using our classroom analogy:

1. The "Star Student" who might be cheating

One model, called StarCoder, did great on most tests but suddenly struggled when the questions were "nudged" on a specific dataset called APPS.

The Analogy: It's like a student who gets 100% on a math test but fails immediately if you change the font of the question. This suggests StarCoder might have memorized the specific answers for the APPS dataset rather than truly understanding the logic.

2. The "Surprising" Clean Benchmarks

There was a lot of worry in the community that popular datasets like Defects4J (a list of bugs) and CVEFixes (security patches) were "contaminated" (i.e., the AI had seen them before).

The Finding: The researchers found the opposite! When they nudged these questions, the AI's performance didn't crash.
The Analogy: People thought the students were cheating on the "Security Exam," but the test showed they actually understood the security concepts deeply. They weren't just reciting answers; they were solving the problems.

3. The Easiest vs. Hardest Subjects

Code Summarization (The Easy Subject): When asked to explain what code does in plain English, the AI models were very robust. Even when the code was slightly changed, they still understood it. It's like a student who can summarize a story no matter how you rephrase the title.
Test Generation (The Hard Subject): When asked to write tests for code (checking if the code works), the models struggled more with the "nudges." This suggests that writing tests requires a deeper, more fragile understanding that is harder to generalize.

4. The Security Domain is a Wild Card

In the field of security (finding bugs and vulnerabilities), the results were all over the place. Some models were super robust, while others fell apart with the slightest nudge.

The Analogy: It's like a classroom where some students are experts in history but terrible at geography. There is no "one size fits all" for security; it depends heavily on how the specific AI was built.

Why Does This Matter?

This paper is a wake-up call for the AI community.

Don't just look at the score: A high score on a test doesn't always mean the AI is smart. It might just mean the AI memorized the test.
We need better tests: We shouldn't just use static, unchanging tests. We need tests that change slightly every time (dynamic benchmarks) to force the AI to prove it actually understands the logic.
Design matters: Two AI models might have the same size (number of "brain cells"), but if one was trained differently, it might be a "Musician" while the other is just a "Parrot."

The Bottom Line

The authors built a tool to measure how "brittle" an AI's knowledge is. If the AI breaks when you slightly change the question, it's memorizing. If it stays strong, it's learning.

They found that while some AI models are indeed memorizing specific datasets, others are genuinely learning to code. The key takeaway? We need to stop trusting the raw scores and start testing how well these models handle the unexpected.

1. Problem Statement

The rapid advancement of Code Large Language Models (Code LLMs) is hindered by a lack of transparency regarding training datasets. This opacity creates significant challenges in distinguishing between true generalization (learning underlying programming concepts) and memorization (recalling specific training examples).

Data Leakage Concerns: Many standard benchmarks (e.g., HumanEval, APPS, CVEFixes) are publicly available and likely included in training corpora. High performance on these benchmarks may stem from data leakage rather than genuine capability, leading to inflated evaluation results.
Detection Difficulty: Traditional methods to detect memorization require access to internal model weights or specific training data details, which are often proprietary or unavailable.
The Gap: There is a need for a robust, black-box methodology to quantify the "memorization advantage"—the performance gap between a model handling familiar (seen) data versus novel (unseen/perturbed) data.

2. Methodology

The authors propose a Perturbation-Based Approach grounded in the Perturbation Sensitivity Hypothesis (PSH). The core intuition is that models relying on memorization will suffer drastic performance drops when inputs are slightly perturbed, whereas models relying on generalization (interpolation) will remain robust.

A. Definition of Memorization Advantage ($ma$)

The memorization advantage is defined as the absolute difference in performance probability between a model $M$ processing an original input $x$ versus a perturbed input $x'$ :
$ma(M, x, y) = |p_\theta(y|x) - p_\theta(y|x'(x))|$
Where $x'$ is a function returning a similar but unseen sequence.

B. Experimental Framework

Models Evaluated: 8 state-of-the-art open-source Code LLMs (ranging from 7B to 33B parameters), including DeepSeek-Coder-V2, Qwen2.5-Coder, StarCoder, CodeLlama, Codestral, OpenCoder, WizardCoder, and Magicoder.
Benchmarks: 19 datasets across 5 software engineering tasks:
- Code Generation: HumanEval, APPS, MBPP, LBPP, CodeContest, XLCost.
- Test Generation: BigCodeBench, TestEval, QuixBugs.
- Program Repair: QuixBugs, Defects4J, ConDefects.
- Vulnerability Detection: CVEFixes, VulDetectBench, VulnPatchPairs, Devign, ReVeal, DiverseVul.
- Code Summarization: CodeSearchNet, TLCode-Sum.
Perturbation Techniques:
- Natural Language Perturbations (for NL2Code): Using the BART model to generate paraphrased prompts with increasing cosine distance from the original, preserving core requirements.
- Code Perturbations (for Code2Code/Code2NL): Systematic renaming of variables (alpha-renaming) with random identifiers to alter syntax while preserving semantics.
Sensitivity Metric:
- For each input, 5 levels of perturbation intensity are applied.
- The model generates outputs for each perturbed input (3 samples per prompt to ensure statistical significance).
- Sensitivity Score: Calculated as the maximum performance drop between consecutive perturbation levels. High sensitivity indicates memorization; low sensitivity indicates generalization.
Statistical Analysis: The Mann-Whitney U test with Bonferroni correction is used to identify statistically significant differences in sensitivity distributions across models and benchmarks.

3. Key Contributions

Empirical Framework: Introduced a prompt perturbation sensitivity analysis framework to empirically distinguish between memorization and generalization in Code LLMs without requiring access to training data.
Comprehensive Evaluation: Evaluated 8 diverse Code LLMs across 19 benchmarks covering generation, repair, testing, security, and summarization.
Challenge to Prevailing Assumptions: Provided evidence that widely suspected "contaminated" benchmarks (specifically CVEFixes and Defects4J) actually exhibit low memorization advantage, suggesting models are genuinely generalizing on these tasks.
Model Architecture Insights: Demonstrated that models with similar parameter counts (e.g., StarCoder vs. CodeLlama) exhibit vastly different sensitivity patterns, highlighting the impact of architectural design and training methodology over sheer scale.
Task-Specific Boundaries: Identified that code summarization allows for the strongest generalization, while test generation and context-dependent program repair present the highest generalization challenges.

4. Key Results

A. Model-Specific Patterns

StarCoder: Exhibited the highest sensitivity scores, particularly on the APPS benchmark (sensitivity $\approx$ 0.8), suggesting significant memorization or data contamination specific to that dataset.
QwenCoder & CodeLlama: Consistently showed the lowest sensitivity scores (<0.4) across most benchmarks, indicating superior generalization capabilities.
Variability: Significant variations were found even among models with comparable parameter counts, proving that training data composition and architecture matter more than size alone.

B. Task-Specific Patterns

Code Summarization: Showed the lowest sensitivity (<0.3) across all models, indicating robust generalization. This is likely due to the abundance of code-documentation pairs in pre-training data.
Test Generation: Showed significantly higher sensitivity (0.4–0.7) compared to basic code generation, suggesting that generating tests requires deeper semantic understanding that current models struggle to generalize.
Program Repair:
- Defects4J: Surprisingly low sensitivity (0.2–0.4), challenging the assumption that it is a "leaked" benchmark.
- ConDefects: High sensitivity (0.5–0.8), indicating that context-dependent bugs (spanning multiple functions) are a major generalization bottleneck.

C. Security Benchmarks (Vulnerability Detection)

CVEFixes: Showed exceptionally low sensitivity (<0.1) across all models. This contradicts community concerns about data leakage, suggesting models have learned generalized patterns of common vulnerabilities rather than memorizing specific CVEs.
Other Security Benchmarks: Showed moderate to high sensitivity (0.4–0.8), indicating that generalizing security knowledge remains a challenge.

5. Significance and Implications

Re-evaluating Benchmarks: The study suggests that high performance on benchmarks like CVEFixes and Defects4J is a sign of genuine capability, not just data leakage. Conversely, high performance on APPS for StarCoder may be inflated.
Evaluation Standards: The paper advocates for Perturbation-Based Evaluation as a standard metric alongside traditional accuracy. Reporting sensitivity scores can help researchers and practitioners identify brittle models.
Dynamic Benchmarks: To mitigate future data leakage, the authors propose Dynamic Benchmarks (time-stamped, procedurally generated, or human-curated) that are continuously updated to stay ahead of model training cutoffs.
Architectural Guidance: The findings imply that simply scaling model size is insufficient. Future model development should focus on architectural choices and training methodologies that specifically enhance generalization in difficult domains like test generation and complex repair.

Conclusion

This paper provides a critical lens through which to view Code LLM performance. By quantifying the "memorization advantage" via perturbation sensitivity, the authors reveal that generalization capabilities are highly task-dependent and model-specific. The work calls for a shift in the research community toward more robust, perturbation-aware evaluation frameworks to ensure that Code LLMs are truly learning to code rather than merely reciting training data.