🤖 machine learning

Learning from 53.6K Real-World Developer Edits of AI-Generated Code

This paper introduces DECODE, a dataset of 53,600 real-world in-IDE edits of AI-generated code from over 1,000 developers, which reveals that most modifications occur within 15 minutes of acceptance and demonstrates that fine-tuning smaller models on this data significantly outperforms frontier LLMs in predicting code edits.

Original authors: Jenny T. Liang, Mihika Bairathi, Wayne Chi, Ameet Talwalkar, Nishant Subramani, Valerie Chen

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jenny T. Liang, Mihika Bairathi, Wayne Chi, Ameet Talwalkar, Nishant Subramani, Valerie Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to cook. You show it a recipe book, and it tries to make a dish. Sometimes, the robot gets it perfect. But often, it adds too much salt, forgets the garlic, or uses a spoon instead of a whisk. In the world of computer science, this robot is an "AI programming assistant," and the dishes are lines of code. For a long time, scientists tried to teach these robots by showing them finished recipes (called "Git commits")—the final, perfect dishes that made it into cookbooks. But this is like trying to learn how to cook by only looking at the food on the plate, ignoring the messy kitchen, the burnt toast, and the chef's frantic corrections.

The real magic (and the real mess) happens in the middle. When a human chef tastes a soup and says, "No, that needs more pepper," or "Actually, throw that away and start over," that is a "code edit." Until now, we didn't have a good way to watch the chef do this. We only saw the final result. This paper dives into that messy, beautiful kitchen. It asks: How do humans actually fix AI mistakes? Do they tweak a little bit, or do they trash the whole thing? And can we teach the robot to predict these fixes before they even happen?

The Great Code Detective: DECODE

The researchers behind this study realized that to make AI better at writing code, they needed to stop looking at the finished product and start watching the process. They created a massive new dataset called DECODE (Developer Edits of Code Dataset). Think of DECODE as a giant, high-definition security camera system installed in the IDEs (the digital workspaces) of over 1,000 real developers. Instead of just saving the final code, this system recorded every single tap, delete, and rewrite a developer made after the AI suggested a piece of code.

They collected 53,600 of these real-world editing sessions, mostly in Python, TypeScript, and JavaScript. It's like having a library of 53,600 stories about how humans and robots argue, compromise, and collaborate to build software.

What They Found: The 15-Minute Rule and the "Trash Can"

When the team looked at the data, they found some surprising patterns about how humans treat AI suggestions.

First, they discovered a strict 15-minute rule. Most of the editing happens very quickly. If a developer is going to change the AI's code, they usually do it within the first 15 minutes of accepting the suggestion. After that, the code is usually left alone.

Second, they found that humans are surprisingly harsh. In about 31% of the cases, the developer didn't just tweak the code; they deleted the AI's entire suggestion and started from scratch. It's like the AI handed you a sandwich, and you immediately threw it in the trash because the bread was slightly wrong. The data suggests that if an AI suggestion doesn't fit the developer's intent perfectly right away, it's often doomed to be abandoned.

However, when humans do keep the code, they don't change much. The data shows that for the code that survives, developers usually keep about 63% of the original AI suggestion. They tend to add very little of their own new code (only about 20% of the final code is new stuff added by the human). This means developers rely heavily on the AI, but they are very picky about the initial quality.

The researchers also noticed a specific order to how people fix things. It's like a checklist:

The "Is this even usable?" check: First, they decide if they want to keep the code at all. If not, they delete it.
The "Fix the bugs" phase: Next, they fix syntax errors or readability issues.
The "Make it mine" phase: Then, they rename variables or change numbers to fit their specific needs.
The "Change the plan" phase: Finally, if they are still working on it, they might change what the code actually does.

Teaching the Robot to Read Minds

The second big part of the paper was a test: Can we teach a smaller, open-source AI model to predict these edits better than the giant, super-smart "frontier" models?

Usually, the biggest, most expensive AI models are considered the best at everything. But here, the researchers took a smaller model (one with 3 billion parameters, which is tiny compared to the giants) and trained it specifically on the DECODE dataset. They taught it to look at an AI suggestion and guess: "Will the human delete this? Will they tweak it? Or will they leave it alone?"

The result was a shock. The small model, after learning from the real-world editing data, became significantly better at predicting edits than the massive, state-of-the-art models that had never seen this specific data.

For guessing what the final code would look like, the small model improved its accuracy by 0.17 (a huge jump in this field).
For guessing if the code would be changed at all, it also improved by 0.17.

This suggests that having the right "experience" (real-world editing data) is more important than just having a bigger brain. It's like a young chef who has watched thousands of cooking shows (the data) might know how to fix a dish better than a famous chef who has only ever cooked perfect meals in a lab.

The Takeaway: It's About the Human, Not Just the Code

The paper concludes that we need to stop treating AI code generation like a math problem where the only goal is "correctness." The real goal is "alignment." The code needs to fit the human's intent, or the human will just throw it away.

The authors suggest that future AI assistants shouldn't just try to write the perfect code on the first try. Instead, they should be trained to understand the process of editing. They need to learn that humans often delete things quickly, that they fix errors before changing features, and that they are more likely to keep code that is easy to tweak.

By using datasets like DECODE, we can build AI that doesn't just write code, but writes code that humans actually want to keep. It's a shift from asking "Is this code correct?" to asking "Is this code something a human can easily make their own?" And the data suggests that if we teach our AI to understand the messy, 15-minute editing window, we might finally get a robot that cooks a meal we're happy to eat.

Technical Summary: Learning from 53.6K Real-World Developer Edits of AI-Generated Code

Problem Statement

Despite the widespread adoption of AI programming assistants, generated code is frequently imperfect, containing bugs or misaligning with specific coding styles. Consequently, developers must manually modify AI-generated code, either through direct editing or by re-prompting the model. Current Large Language Models (LLMs) are primarily trained on publicly available Git data (e.g., commits and pull requests) to learn code editing. However, Git data represents only the final, successful state of code, lacking the intermediate steps, temporal ordering, and context of the actual editing process. This gap limits the ability of LLMs to predict or model how developers interact with and modify AI completions in real-time. Furthermore, existing datasets often rely on synthetic data, laboratory settings, or instructed edits, failing to capture the nuances of real-world, manual code edits of AI-generated content.

Methodology

1. Dataset Construction: DECODE

To address the lack of realistic editing data, the authors introduce DECODE (Developer Edits of Code Dataset), a collection of 53.6K real-world, in-IDE code edits of AI-generated code.

Source: Data was collected from a Visual Studio Code (VS Code) extension used by over 1,000 developers. The extension captures AI completions and snapshots of the working file every time a developer pauses for one second.
Scope: The dataset covers Python, TypeScript, and JavaScript, spanning 20 different LLMs and diverse development contexts (e.g., web development, data analysis, machine learning).
Extraction Pipeline:
1. Identification: Matching code files containing accepted AI completions ( $y_0$ ) with subsequent snapshots ( $y_t$ ).
2. Extraction: Using git diff --histogram to isolate specific lines added, removed, or modified relative to the original completion. This reduced the raw data from 6.5M lines to 772.2K lines directly relevant to edits.
3. Cleaning: Validating trajectories using CodeBERTScore (threshold 0.68) to ensure relevance and removing malformed data.
4. Privacy: Applying PII redaction pipelines (Starcoder/Stack and OpenAI Privacy Filter) to remove personally identifiable information.
Statistics: The final dataset comprises 5,831 edit trajectories from 1,141 developers, with a median trajectory duration of ~50 minutes and a median of 329 Levenshtein edits per trajectory.

2. Analysis of Developer Behavior

The authors utilized DECODE to analyze when, why, and how developers edit AI code using metrics such as the proportion of remaining AI code, developer-added code, and total edit volume.

Temporal Patterns: 50% of edits occur within the first 50 minutes of acceptance, with a steep drop-off after 15 minutes.
Edit Types: Edits were classified into four categories: Customizing code (10%), Improving quality (14%), Changing functionality (56%), and Removing completions (9%).
Outcomes: 31% of trajectories result in the complete removal of the AI completion. The distribution of remaining code is bi-modal: developers either keep the code with few modifications or discard it entirely.

3. Benchmarking LLMs

The study evaluated the ability of LLMs to predict code edits on two tasks:

Classification: Predicting whether an AI completion will be deleted, unmodified, or modified.
Generation: Predicting the final state of the code ( $y_t$ ) given the initial AI completion ( $y_0$ ).
Models Tested: Frontier models (e.g., GPT-5.2, Claude Sonnet 4.6) and open-source models (e.g., Qwen2.5-Coder, Llama3.2) were tested in few-shot settings and after fine-tuning on DECODE using LoRA.

Key Results

1. Insights into Developer Behavior

Rapid Abandonment: Developers often abandon AI completions quickly if they do not align with their intent. 23% of trajectories involve immediate removal, often followed by the developer writing their own implementation.
Low Addition Rate: Developers rarely add significant new code to AI completions; 20% of the final code is developer-added, and 36% of trajectories contain less than 5% new developer code.
Sequential Editing: A common pattern emerges where developers first decide whether to keep the code (removal), then fix errors (quality), adapt to needs (customization), and finally modify functionality.

2. Performance Improvements via Fine-Tuning

Superiority of Fine-Tuned Small Models: Fine-tuning open-source models (3B and 7B parameters) on DECODE significantly outperformed frontier models in both classification and generation tasks.
- Classification: Fine-tuned 3B models achieved an F1 score of 0.42–0.45, surpassing frontier models like Claude Sonnet 4.6 (0.37) and GPT-5.2 (0.32).
- Generation: Fine-tuned models improved Levenshtein similarity by +0.15 to +0.22 over their base versions and significantly outperformed frontier models (e.g., +0.17 improvement over frontier baselines).
Data Quantity: Performance improved as models were exposed to more edit steps during training (up to 4 steps), suggesting that capturing the sequential nature of edits is crucial.
No Degradation: Fine-tuning on DECODE did not degrade code generation capabilities on standard benchmarks (HumanEval, MBPP); in some cases (Qwen2.5-Coder-3B), performance slightly improved.

Significance and Claims

The paper argues that developer-centric machine learning approaches are necessary for the next generation of AI programming assistants. The key takeaways are:

Alignment over Correctness: Standard benchmarks focus on code correctness (pass@k), but real-world utility depends on how well code aligns with developer intent and context. Developers often discard "correct" code if it is not easily adaptable.
Editability as a Metric: The ability of AI-generated code to be easily edited (editability) is a critical, yet under-measured, factor. Future models should be able to detect generations that are likely to be discarded.
Temporal Modeling: LLMs must capture the sequential and temporal nature of developer edits (e.g., removals happen early, functionality changes happen later). Training on raw Git commits fails to capture this dynamic process.
Data Quality: Real-world interaction data (like DECODE) is superior to synthetic or Git-derived data for training models to handle manual code edits. Even small models fine-tuned on high-quality, real-world edit data can outperform much larger frontier models.

The authors conclude that future AI programming assistants require new modeling and evaluation paradigms that explicitly account for developer interaction, personalization, and the iterative nature of code editing.