Technical Summary: To Add Is Machine, To Delete Is Human

Problem Statement

While Large Language Models (LLMs) increasingly function as coding agents capable of resolving issues and generating pull requests, evidence suggests their patches often degrade code maintainability. A specific, systematic failure mode identified in this work is deletion avoidance: the tendency of models to retain code that an intended edit requires removing. Instead of performing a subtractive edit, models often preserve the obsolete logic and route execution around it using conditional guards or fallbacks. This behavior, termed Guard-and-Go, allows patches to pass existing test suites (which rarely verify code removal) while leaving codebases bloated and harder to maintain. The paper posits that current evaluation benchmarks, such as SWE-bench Verified, may overstate model readiness for production because they fail to penalize these non-removal strategies.

Methodology

The study employs a multi-faceted approach combining empirical analysis of existing benchmarks, the creation of a specialized diagnostic benchmark, and a controlled post-training intervention.

1. Empirical Analysis of SWE-bench Verified

The authors analyzed the five leading submissions to the SWE-bench Verified leaderboard (GLM-4.6, GPT-5, Kimi-K2, Opus-4.5, and Salesforce SAGE) using a consistent OpenHands scaffold to control for agent variation.

• Metric: Deletion recall was calculated by comparing model patches against developer patches, measuring the share of developer-deleted lines that the model also removed.
• Localization Check: The study distinguished between failing to find the code (localization error) and failing to remove it (execution error) by checking file, scope (function/class/module), and exact line modification.
• Strategy Classification: An LLM-based classifier categorized passing patches into "Delete-and-Replace," "Guard-and-Go" (retaining logic with a guard), or "Non-reference alternative."

2. Deletion-Sensitive Retrofitting

To determine if passing tests actually validate code removal, the authors retrofitted 34 deletion-heavy tasks from SWE-bench Verified with new FAIL_TO_PASS (F2P) tests. These tests explicitly fail if the targeted code remains in the file, isolating the removal requirement from other behavioral specifications.

3. CanItDelete Benchmark

To isolate deletion behavior from confounding factors like localization difficulty or the need for additive code, the authors curated CanItDelete.

• Construction: 200 tasks mined from real commits where the entire required edit was deletion (no additions). Tasks were selected from the top 100 starred Python and JavaScript repositories, ranked by structural complexity (pre-edit size, deleted lines, and deletion hunks).
• Evaluation: A deterministic, occurrence-aware evaluator scores outputs based on whether the target is fully absent, unrelated structure is preserved, and no behavior-affecting additions are introduced.
• Diagnostic Ladder: The benchmark utilizes four cumulative modes to diagnose failure points:
  1. Vanilla: Standard developer-style request.
  2. Explicit Deletion: Instruction forbids guards or fallbacks.
  3. Region Pointer: Identifies relevant functions/regions.
  4. Exact Lines: Supplies the exact spans to remove.

4. Post-Training Intervention (Proof of Concept)

A 7B parameter model was augmented with a deletion-focused post-training mixture.

• Data: 12,821 deletion examples (10,000 file-level and 2,821 repository-level) were added to the training set. These examples contributed 112.1M tokens to the 15.9B-token mixture, constituting approximately 0.7% of the total tokens.
• Evaluation: The model was tested on CanItDelete, SWE-bench Verified, CanItEdit, and EditBench to measure targeted improvement and potential regressions.

Key Results

1. Prevalence of Deletion Avoidance

• Low Recall: Even on the 197 tasks solved by all five models, deletion recall ranged from 65.2% to 71.7%. Models left 28.3% to 34.8% of required deletions in place.
• Localization vs. Execution: Models successfully located the correct file for >92% of deletions and the enclosing scope for ~70%, yet removed the exact line in only 44.6% to 51.6% of cases. The gap is primarily an execution failure, not a localization failure.
• Guard-and-Go Dominance: Among the 1,703 passing task–model pairs analyzed, 494 (29.0%) followed the Guard-and-Go pattern, where the model retains the developer-removed logic and adds a conditional bypass. These patches were 61.1% larger than the developer's patch on average.

2. Evaluation Gap

When 34 deletion-heavy tasks were re-evaluated with deletion-sensitive checks:

• Resolution rates for four frontier models dropped from 63.2% to 41.9% (a 21.3 percentage point decline).
• 33.7% of patches that passed the original test suites retained the validated deletion target.

3. CanItDelete Findings

• Performance Variance: On the 200 deletion-only tasks, success rates ranged from 79.0% (Claude Opus 4.8) down to 18.0% (smaller open models).
• Failure Modes: Incomplete deletion (leaving required code) accounted for 69.8% of failures across models. However, as models improved at finding targets, a second failure mode emerged: over-deletion (removing code beyond the target boundary).
• Impact of Guidance: Providing exact deletion spans (the "Exact lines" mode) nearly eliminated incomplete deletion for four of five models but did not fix over-deletion. For example, GPT-5.6 Sol's success rose to 80.5%, but it still failed 16.5% of tasks by deleting beyond the specified spans. This indicates a lack of control over the deletion boundary rather than a lack of capability to find the code.

4. Post-Training Intervention

Adding deletion supervision to the 7B model's post-training mixture:

• Reduced incomplete deletion on CanItDelete by 13.9 percentage points (from 80.4% to 66.5% failure rate).
• Improved SWE-bench Verified by 5.3 points and CanItEdit by 1.4 points.
• Did not introduce significant regressions in other benchmarks.
• Trade-off: The intervention increased the rate of over-deletion (complete deletion with invalid edits), suggesting that while the model learned to delete, it still struggles with where to stop.

Significance and Claims

The paper claims that deletion avoidance is a systematic, undertrained behavior in current LLMs rather than an intrinsic limitation. The authors argue that:

1. Current Benchmarks are Insufficient: Standard test-passing metrics mask the "Guard-and-Go" pattern, leading to an overestimation of model readiness for code maintenance.
2. Control is the Bottleneck: Models possess the capability to locate code to be deleted but lack the control to execute precise, bounded removals without over-editing or adding guards.
3. Deletion is Learnable: The proof-of-concept intervention demonstrates that targeted post-training can reduce deletion avoidance and improve broader code-editing performance, suggesting the deficit is due to data representation (additive bias in training data) rather than model architecture.

The authors remain modest about the scope, noting that the intervention was tested on a single 7B model and that the "over-deletion" trade-off indicates that deletion completion and boundary control are distinct training objectives that require further research to resolve simultaneously at scale.