Technical Summary: HEJ-Robust

Problem Statement

Recent Large Language Models (LLMs) have demonstrated strong performance in Automated Program Repair (APR) on standard benchmarks. However, existing benchmarks, such as Defects4J and HumanEval-Java-Bug, typically evaluate models on a single canonical form of buggy code. These benchmarks fail to reflect the syntactic variations common in real-world software, leaving the robustness of LLM-based repair models largely unexamined. While neural code models are known to be sensitive to semantics-preserving transformations in other tasks, robustness evaluation for function-level, human-crafted APR benchmarks remains an open gap.

Methodology

The authors introduce HEJ-Robust, a robustness benchmark designed to evaluate the consistency of APR models under syntactic perturbations.

Dataset Construction

• Base Dataset: The benchmark is built upon HumanEval-Java-Bug, a dataset of 164 manually curated Java programs with injected bugs, annotated buggy-line locations, executable test cases, and human-written patches.
• Transformations: The authors applied eight semantics-preserving code transformations to the base dataset, resulting in 1,450 transformed instances. These transformations reflect common real-world coding practices:
  1. Local Variable Renaming: Renames local variables using naturalness-aware substitution (via CodeBERT/GraphCodeBERT) rather than random strings.
  2. Method Renaming: Renames method names.
  3. Parameter Renaming: Renames method parameters.
  4. Log Statement Insertion: Adds System.out.println statements (does not affect test outcomes as tests validate return values).
  5. Try-Catch Insertion: Adds try-catch blocks at applicable locations.
  6. Boolean Exchange: Flips boolean initialization values and negates subsequent uses to maintain semantic equivalence.
  7. Loop Exchange: Converts for loops to while loops and vice versa.
  8. Condition Reordering: Swaps operands in == and != expressions.
• Annotation: After transformation, buggy-line locations were manually re-annotated. Instances where the transformation directly modified the buggy line were removed to avoid ambiguity.

Experimental Setup

• Models: Five fine-tuned LLMs were evaluated: PLBART (base and large) and CodeT5 (small, base, and large). These were selected as they are the only publicly available fine-tuned APR models evaluated on HumanEval-Java-Bug.
• Metrics:
  • Pass@10: The primary metric for functional correctness (a bug is fixed if at least one of 10 generated patches passes all test cases).
  • CodeBLEU: Reported as a reference control metric for code similarity.
• Protocol: Models were evaluated without modification to their original training or decoding configurations.

Key Results

The evaluation reveals significant robustness gaps in current LLM-based repair models:

• Performance Degradation: Across most transformations, model performance (Pass@10) dropped significantly. The most severe degradation occurred under Local Variable Renaming, where performance decreased by 50.5% to 57.31% across models. This suggests models rely heavily on specific identifier patterns learned during training.
• Variable Impact:
  • Parameter Renaming and Log Statement Insertion caused smaller drops, indicating that method-level context or appended logging is less disruptive.
  • Structural transformations (e.g., Loop Exchange) caused moderate drops.
  • Boolean Exchange results were excluded from headline claims due to the small sample size (7 instances), which made statistical significance unattainable.
• Model Size Correlation: Robustness does not correlate with model size. In several cases, larger models (e.g., CodeT5-large, PLBART-large) degraded more than their smaller counterparts.
• Metric Reliability: CodeBLEU scores remained largely stable across transformations, even when Pass@10 dropped drastically. This confirms that CodeBLEU is an unreliable indicator of functional robustness for APR, as models can produce syntactically similar but functionally incorrect patches.

Key Contributions

1. HEJ-Robust Benchmark: The introduction of a transformation-based robustness benchmark built on HumanEval-Java-Bug, covering eight semantics-preserving transformations and providing function-level, test-executable evaluation.
2. Systematic Evaluation: A comprehensive assessment of five fine-tuned LLMs under semantics-preserving transformations, demonstrating that current models lack robustness to minor syntactic variations.
3. Open Resources: The public release of the benchmark, code, and artifacts to facilitate future research on robust and reliable automated program repair.

Significance and Claims

The paper claims that current LLM-based repair models are not robust to minor syntactic variations, a critical limitation for real-world deployment where code styles and structures vary. The authors emphasize that:

• Existing benchmarks ignore syntactic diversity, potentially inflating perceived model capabilities.
• Functional metrics (Pass@k) are essential for robustness evaluation, whereas similarity metrics (CodeBLEU) fail to capture functional degradation.
• The benchmark is complementary to Defects4J-TRANS (which targets project-level bugs); HEJ-Robust specifically targets function-level, human-crafted bugs.

The authors conclude that while their findings highlight substantial robustness gaps, they are modest about generalization, noting that the benchmark is limited to isolated function-level programs and specific model families (PLBART and CodeT5). Future work is suggested to explore richer transformation spaces, project-level dependencies, and instruction-tuned or agent-based LLMs.