💻 computer science

Evaluating LLMs on Java Code Snippet Adaptation Using a Mutation-Injection Framework

This paper proposes a mutation-injection framework to evaluate large language models on instruction-free Java code snippet adaptation, investigating how performance varies across different adaptation types, complexity levels, and context granularities using a dataset derived from open-source repositories with strong test coverage.

Original authors: Ali Aman, Muhammad Asaduzzaman, Shaowei Wang, Chanchal K. Roy

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ali Aman, Muhammad Asaduzzaman, Shaowei Wang, Chanchal K. Roy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef who finds a delicious recipe for "Spaghetti Carbonara" in an old cookbook. You want to make it for your friends, but there's a problem: your friends are allergic to eggs, and you only have a tiny pot, not a big one. You can't just copy the recipe exactly; you have to adapt it. You need to swap the eggs for something else, change the cooking time, and maybe skip a step.

In the world of computer programming, developers do the exact same thing. They find a piece of code (a "snippet") that works for someone else, copy it, and then have to tweak it to fit their own project.

This paper is a plan to test how well Artificial Intelligence (AI) can do this "recipe adaptation" job on its own, without being told exactly what to change.

Here is a breakdown of the paper's plan using simple analogies:

1. The Problem: The "Silent Chef" Test

Currently, when we ask AI to fix code, we usually give it a very specific list of instructions, like: "Change the variable name from 'x' to 'y' and swap the library."

But in the real world, developers don't write lists. They just say, "Here is the code I copied; make it work in my new project." The AI has to look at the code and the new environment and figure out the changes itself. The authors want to know: Can the AI figure out the changes without a step-by-step manual?

2. The Solution: The "Mutation Machine"

To test this fairly, the authors can't just use random code from the internet because they wouldn't know exactly what changes the AI should have made.

Instead, they are building a "Mutation Machine" (a framework they call Mutation-Injection). Here is how it works:

Step 1: They start with a perfect, working piece of code that has already been tested (like a perfect recipe).
Step 2: They use a robot to intentionally "break" or "mess up" the code in specific, controlled ways. For example, they might rename a variable, change a number, or swap a tool for a different one.
Step 3: They give this "broken" code to the AI and say, "Fix this so it works again."
Step 4: If the AI fixes it and the code passes all its tests, the AI gets a point.

Because they created the "breaks" themselves, they know exactly what the AI was supposed to do. It's like a teacher who writes a math problem with a known mistake, gives it to a student, and checks if the student found and fixed that specific mistake.

3. The Three Big Questions (The "Menu")

The researchers are testing the AI on three main things:

Question 1: Which "ingredients" are hardest to swap?
Some changes are easy, like renaming a variable (changing "flour" to "sugar"). Others are hard, like changing the entire cooking method (switching from baking to frying). They want to see which types of changes stump the AI the most.
Question 2: Does it get harder if you break more things at once?
If the AI can fix one broken part, can it fix three broken parts at the same time? They are testing if the difficulty adds up linearly or if the AI gets completely confused when multiple things go wrong.
Question 3: How much "context" does the AI need?
Imagine you are fixing a recipe.
- Scenario A: You only see the recipe card.
- Scenario B: You see the recipe card and the list of ingredients in your pantry.
- Scenario C: You see the recipe card, the pantry, and the entire kitchen layout.
  The researchers want to know: Does the AI need to see the whole "kitchen" (the surrounding code) to do a good job, or is the recipe card enough?

4. The Rules of the Game

The Language: They are testing this using Java, a very common programming language.
The "No-Help" Rule: The AI is not allowed to be told what to change. It only gets a generic prompt: "Adapt this snippet to fit the context."
The Safety Net: They are using real, open-source projects that have strong "test suites." Think of these tests as a safety inspector. If the AI changes the code and the safety inspector says, "This still works perfectly," the AI passes.

5. Why This Matters

Right now, we don't really know how good AI is at this specific "copy-paste-and-fix" skill. Existing tests are either too easy, too vague, or only look at whole functions (like a whole meal) rather than small fragments (like a single ingredient).

This study aims to build a massive, fair playground where they can measure exactly where AI succeeds and where it fails when adapting code. If they find that AI is terrible at "swapping tools" but great at "renaming ingredients," future tools can be built to help developers specifically with the hard parts.

In short: The authors are building a controlled "obstacle course" for AI to see how well it can fix broken code on its own, without being given a cheat sheet.

Technical Summary: Evaluating LLMs on Java Code Snippet Adaptation Using a Mutation-Injection Framework

Problem Statement

Developers frequently reuse code by copying fragments from existing repositories or online sources and adapting them to fit new contexts. This adaptation process involves diverse changes such as renaming variables, swapping APIs, adding error handling, and modifying control flow. In real-world scenarios, developers do not provide explicit step-by-step edit instructions; they expect tools to infer necessary changes based on the copied fragment and its surrounding context.

Current benchmarks for evaluating Large Language Models (LLMs) on code adaptation suffer from three primary limitations:

Reliance on Explicit Instructions: Many existing evaluations provide explicit edit guidance, failing to test the model's ability to infer changes autonomously.
Narrow Scope: Benchmarks often cover only specific change types (e.g., variable wiring) or operate exclusively at the function level, ignoring the finer granularity of code fragments (e.g., loop bodies, conditional branches) where copy-paste operations typically occur.
Lack of Controlled Ground Truth: Tasks derived from real commits often entangle multiple change types, making it difficult to isolate which specific adaptation types cause model failures.

Consequently, it remains unknown how well LLMs can perform instruction-free adaptation of code fragments when required changes are varied, controlled, and presented at different levels of complexity and context.

Methodology

The authors propose a Mutation-Injection Framework to construct a controlled, scalable dataset for evaluating instruction-free code adaptation. The methodology proceeds as follows:

1. Dataset Construction

Source: Mature open-source Java repositories (minimum 500 stars, 70% line coverage via JaCoCo, Maven buildable) spanning at least five distinct application domains.
Fragment Extraction: Using the Spoon framework, the authors extract syntactically complete, contiguous code fragments (3–20 statements) that are covered by test cases and contain at least two non-keyword identifiers.
Seed Selection: Fragments serve as "ground-truth seeds" representing the target state.

2. Mutation Injection (Reverse Adaptation)

Instead of asking models to adapt code to a target, the framework starts with the target (the seed) and applies adaptation operators in reverse to create a mutated fragment ( $F'$ ).

Taxonomy: A curated set of nine mutation operators derived from empirical studies of developer behavior (e.g., Zhang et al., Roy et al.). These include:
- Identifier-level: Variable renaming, constant updates, identifier resolution.
- Statement-level: Guard insertion, type changes, try/catch edits.
- Structural/Semantic: Control flow modification, API substitution, logic customization.
Validation: Mutations are validated to ensure the resulting code compiles and, crucially, fails at least one existing test case in the repository. This ensures the mutation introduces a semantic defect that the LLM must fix.
Complexity Levels: Tasks are categorized by complexity ( $L1$ to $L4$ ) based on the number and combination of injected mutations.

3. Task Formulation

LLMs are presented with the mutated fragment ( $F'$ ) and a prompt instructing them to "adapt the snippet to fit the surrounding context." No explicit edit lists or step-by-step guidance are provided.

Context Granularity: The study evaluates three context levels:
- C1: No context (fragment only).
- C2: Method context (enclosing method body).
- C3: File context (full source file).

4. Evaluation Metrics

Functional Correctness: The LLM's output is re-inserted into the original codebase location, and the repository's test suite is executed. Success is measured via pass@1 and pass@5 (where any of 5 generated outputs passes).
Adaptation-Level Correctness: A fine-grained inspection checks if specific injected mutations were correctly reversed, utilizing JSON metadata recorded during injection.
Statistical Analysis: Mixed-effects logistic regression is used to analyze interactions between model, adaptation type, context, and complexity.

Key Contributions

Instruction-Free Evaluation Paradigm: The study shifts focus from explicit edit guidance to autonomous inference, better reflecting real-world code reuse scenarios.
Fragment-Level Granularity: Unlike function-level benchmarks, this framework targets sub-method units (loops, conditionals), aligning with how developers actually copy-paste code in IDEs.
Controlled Ground Truth via Mutation Injection: By applying operators in reverse, the authors know exactly which changes are required. This allows for precise attribution of model failures to specific adaptation types, a capability lacking in benchmarks built from entangled real-world commits.
Scalable, Reusable Framework: The taxonomy is operationalized as programmable mutation operators, enabling future researchers to evaluate new tools at scale without manual annotation.

Research Questions

The study is structured around three exploratory questions:

RQ1 (Adaptation Type Difficulty): Which categories of adaptation are hardest for LLMs to perform without explicit instructions?
RQ2 (Adaptation Complexity): How does performance scale with the number and combination of injected changes? Does combining easy changes create emergent difficulty?
RQ3 (Context Granularity): How much surrounding context is required for successful adaptation, and does this requirement vary by adaptation type?

Significance and Claims

The paper positions itself as a Registered Report, emphasizing the rigor of its experimental design and the pre-registered nature of its analysis plan.

Addressing the Gap: The authors claim that existing benchmarks fail to simultaneously offer instruction-free tasks, fragment-level granularity, controlled change types, and scalability. Their framework is designed to fill this combined gap.
Practical Impact: The results aim to guide the development of future code-reuse tools and IDE plugins by identifying specific adaptation types where current LLMs struggle and determining optimal context windows for different change types.
Reproducibility: The authors commit to making the full dataset, mutation framework, and evaluation scripts publicly available to support replication and extension.

The paper does not claim to have solved the problem of code adaptation but rather proposes a systematic method to measure the current state of LLMs in this domain, providing a baseline for future improvements. It acknowledges limitations, such as the restriction to Java/Maven and the potential divergence between synthetic mutations and natural developer adaptations, noting that the framework is designed to be extensible to other languages and build systems in future work.