1. Problem Statement

Automated Code Revision (ACR) tools, typically based on Large Language Models (LLMs) and Transformer architectures, aim to generate code fixes based on reviewer feedback. While these tools show promise in generating correct revisions for specific inputs, their real-world utility is threatened by a lack of consistency.

• The Core Issue: ACR tools often fail to produce semantically equivalent revisions when presented with structurally different but behaviorally identical code variants (e.g., renaming variables, reordering independent statements, or wrapping code in try-catch blocks).
• The Risk: This sensitivity to "semantics-preserving perturbations" (SPP) suggests that models rely on surface-level syntactic patterns rather than deep semantic understanding. If a tool works on one version of a bug but fails on a slightly refactored version of the same bug, it cannot be trusted in dynamic development environments.
• Research Gap: Previous evaluations focused on accuracy (Exact Match, BLEU) on static datasets. There was a lack of systematic investigation into how robust these tools are against controlled, semantics-preserving structural changes.

2. Methodology

The authors designed a rigorous framework to evaluate the consistency of state-of-the-art ACR tools against SPPs.

A. Dataset and Tools

• Dataset: A subset of 2,032 real-world Java method review instances from GitHub (derived from the Tufano et al. dataset). Each instance includes original code ($c_i$), a review comment ($r_i$), and the human-written ground-truth revision ($\hat{c}_i$).
• Models Evaluated: Five transformer-based ACR tools representing different scales and architectures:
  1. Fine-tuned T5-small.
  2. LoRA-tuned LLaMA-7B.
  3. LLaMA 3.3-70B.
  4. ChatGPT-3.5 Turbo.
  5. DeepSeek V3.
• Baseline: The study first established a baseline by identifying instances where each model successfully generated an Exact Match (EXM) on the original, unperturbed code.

B. Semantics-Preserving Perturbations (SPP)

The authors developed an automated framework using JavaParser to generate over 10,000 perturbed variants. They defined 9 types of SPPs categorized into three programming concepts:

1. Control Flow:
   • If-else swap: Negating conditions and swapping blocks.
   • Dead Exception insertion: Adding unreachable exception throwing code.
   • Dead variable assignment: Adding unreachable variable reassignments.
   • Try-catch wrapper: Wrapping the method body in a try-catch that rethrows exceptions.
   • Independent line swap: Swapping adjacent independent statements.
2. Data Flow:
   • Return via variable: Converting direct returns to intermediate variable assignments.
   • Def-use break: Breaking variable definition-use chains by inserting a new variable with a random name.
3. Identifier Naming:
   • Random variable names: Replacing variables with random 5-letter strings.
   • Shuffle variable names: Randomly reassigning local variable names within the method.

• Crucial Step: The perturbation was applied not only to the original code but also to the ground-truth revision to create a structurally aligned reference ($\hat{c}^{(k)}_i$). This ensures the evaluation measures if the model can apply the fix within the new structure, rather than reverting the structure to the training distribution.

C. Evaluation Metrics

To measure consistency, the authors used four metrics comparing the model's output ($\tilde{c}^{(k)}_i$) against the perturbed ground truth ($\hat{c}^{(k)}_i$):

1. Exact Match (EXM): Strict string equality.
2. CodeBLEU: Semantic and syntactic similarity.
3. Edit Match (EM): Whether the model included all necessary edits, even if it added extra ones.
4. Relative Edit Error (REE): The proportional difference in the magnitude of edits compared to the human revision.

D. Statistical Analysis

• RQ2 (Feature Analysis): A multi-level mixed-effects logistic regression was used to determine which perturbation features (e.g., distance from the review target, token edit count, input length) predict model failure.
• RQ3 (Mitigation): Three input representation strategies were tested to see if they could guide model attention:
  1. Code Repetition (CR): Repeating the target code span inside the review comment.
  2. Inline Commenting (IC): Embedding the review comment as a code comment.
  3. Chain-of-Thought (CoT): Prompting the model to explain the fix before generating code.

3. Key Contributions

1. Systematic Perturbation Framework: The creation of a robust, automated pipeline to generate and evaluate 9 distinct types of semantics-preserving perturbations on real-world Java code.
2. Comprehensive Consistency Benchmark: The first large-scale evaluation of 5 diverse ACR tools (from small fine-tuned models to massive foundation models) specifically measuring their sensitivity to structural variations.
3. Identification of Critical Failure Modes: Empirical evidence showing that ACR tools are highly sensitive to perturbations occurring inside or near the code region targeted by the review comment.
4. Mitigation Analysis: A critical investigation into whether common NLP prompting techniques (CoT, repetition) improve code revision consistency, finding that they often degrade performance.

4. Key Results

A. Significant Consistency Degradation (RQ1)

• ACR tools are highly sensitive to SPPs. The ability to generate correct revisions dropped by up to 45.3% (observed in LoRA-tuned LLaMA) when presented with semantically equivalent variants.
• Control Flow and Data Flow perturbations caused the most severe drops in performance. Specifically, If-else swap ($p_1$) and Def-use break ($p_7$) were the most disruptive.
• Dead code insertions (e.g., unreachable if blocks) had the least impact, suggesting models can ignore semantically inert noise better than structural logic changes.

B. Factors Influencing Inconsistency (RQ2)

• Proximity is Key: The regression analysis revealed that the distance between the perturbation and the review-targeted code span is the strongest predictor of failure.
  • Perturbations inside the targeted span reduced the odds of an Exact Match by ~50%.
  • Perturbations farther away were less disruptive.
• Token Edits: Higher token edit counts (structural divergence) correlated with lower consistency.
• Model Variance: Larger foundation models (LLaMA 3.3-70B, DeepSeek V3) showed lower Relative Edit Error (less over-editing) but still suffered significant Exact Match drops on complex structural changes.

C. Ineffectiveness of Mitigation Strategies (RQ3)

• Counter-intuitive Finding: The proposed mitigation strategies (Code Repetition, Inline Comments, CoT) failed to improve consistency and often worsened it.
  • CoT: Showed minor, inconsistent gains for LLaMA models but degraded performance for others.
  • Code Repetition & Inline Comments: Generally reduced performance, with some models (e.g., T5) seeing drops of up to 55.6% in specific perturbation types.
• Conclusion: Adding textual noise or forcing reasoning steps appears to distract the model's attention or disrupt the token alignment between the review comment and the code, rather than helping it.

5. Significance and Implications

• Reliability Gap: The study highlights a critical gap between high benchmark accuracy and real-world robustness. A tool that works on a specific code snippet may fail immediately upon minor refactoring, making it unreliable for automated integration.
• Attention Mechanism Limitations: The results suggest that current Transformer-based ACR tools rely heavily on surface-level syntax and local patterns rather than true semantic reasoning. They struggle to decouple the "fix" from the "structure."
• Guidance for Developers:
  • Developers should not rely on ACR tools for code that has undergone recent structural refactoring without re-evaluation.
  • Standard NLP prompting techniques (like CoT) are not directly transferable to code revision tasks and may be harmful.
• Future Research Directions:
  • Evaluation frameworks must include perturbation-based testing alongside accuracy metrics.
  • New architectural or training interventions are needed to improve semantic reasoning and robustness to structural noise.
  • Tools should expose "consistency scores" or robustness indicators to end-users to manage expectations.

In summary, the paper concludes that current Automated Code Revision tools are sensitive, not consistent. They are brittle against semantics-preserving structural changes, particularly those near the target of the review, and current "fixes" via prompting strategies are insufficient to solve this fundamental limitation.