Diffs vs. Whole Files: An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models
This paper empirically demonstrates that for Flutter/Dart code editing, direct full-file generation by large language models substantially outperforms iterative diff-based generation across all metrics, with the latter only proving competitive for short, spatially localized edits such as refactoring and error-handling tasks.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
When a computer program needs to fix a mistake in a piece of code, there are two main ways a smart machine can do the job. The first way is to rewrite the entire file from the beginning, producing a fresh, complete version of the document. The second way is to act like a human editor, making a series of small, specific changes—finding a sentence and swapping it for a new one, or deleting a line and inserting another—until the job is done. This second method, often called a "diff" or a patch, is popular in the software world because it seems more efficient; it generates less text and mimics how people actually work. It feels intuitive to think that making small, targeted edits is better than rewriting everything. However, whether this intuition holds true when teaching artificial intelligence to write code has remained an open question.
A recent study set out to settle this debate by pitting these two approaches against each other in a controlled experiment. Researchers trained two different computer models to fix code in a specific programming language used for building mobile apps. They taught one set of models to rewrite entire files in a single go, and they taught another set to perform the same tasks by issuing a sequence of small, step-by-step edits. They then tested both sets of models on nearly 1,800 different coding tasks to see which method produced better results. The findings were clear and somewhat surprising: the models that rewrote the entire file consistently outperformed the models that tried to make small, iterative changes. This advantage held true across every measure of success, from whether the code actually worked to how closely it matched the correct answer.
The researchers discovered that the failure of the step-by-step approach was not usually due to the models running out of time or getting stuck in a loop. In fact, most of the time, the models using the step-by-step method successfully completed their list of instructions. The problem was that the final result was often subtly broken. A large share of these failures occurred because the input code contained two or more identical or near-identical spans of text, and the model's disambiguation heuristic could not determine which specific one to replace. This single issue alone accounted for roughly half to two-thirds of the step-mode failures across the two architectures tested. Because the model could not accurately target the correct section, its small changes would accidentally edit the wrong part of the code or break parts that were previously working. These errors were often silent, meaning the code would still run but would not do what the user intended.
Even when the researchers filtered out the obvious failures and looked only at the tasks where both methods produced code that the computer could successfully compile, the direct rewriting method still produced higher-quality results. An independent artificial intelligence judge, which evaluated the code without knowing which method created it, rated the direct generation outputs as more correct and better written. The study ruled out the idea that the step-by-step models were simply undertrained or that the task was too difficult for them to handle in pieces. The gap in performance persisted even when the researchers accounted for these factors, suggesting that the method of generating the code itself was the primary cause of the difference.
However, the story is not entirely one-sided. The researchers found that the step-by-step approach did have a specific niche where it could compete. It performed well only when the required change was very small and localized to a tiny part of the file. For example, when the task involved fixing a single error or refactoring a short, isolated block of code, the step-by-step models were nearly as good as the ones that rewrote the whole file. But as soon as the task required a longer chain of changes or edits spread across different parts of the file, the step-by-step method quickly fell behind. The researchers concluded that the success of an editing strategy depends on the "locality" of the task: if the change is small and self-contained, a patch might work, but for anything more complex, rewriting the whole file is the safer and more reliable choice.
This discovery challenges the common assumption that making small, targeted edits is always the most efficient path for artificial intelligence. While the step-by-step method saves on the amount of text the computer needs to generate, it introduces a higher risk of subtle errors that accumulate over time. The study suggests that for building reliable code-editing tools, the best approach is not to force the model to always use one method or the other, but to recognize the nature of the task. When the change is broad or complex, the model should be allowed to rewrite the entire file to ensure accuracy. Only when the change is small and confined to a specific spot should the system rely on a sequence of small edits. This insight helps clarify how to design better tools for developers, ensuring that the artificial intelligence they use produces code that is not just efficient to generate, but correct and robust in practice.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.