Byam: Fixing Breaking Dependency Updates with Large Language Models

1. Problem Statement

Modern software development relies heavily on third-party libraries (dependencies). When these libraries are updated, they often introduce breaking changes (e.g., method signature modifications, deprecations, or removals) that cause client applications to fail compilation.

• The Challenge: Identifying the root cause of these breaks and manually updating client code is tedious, time-consuming, and error-prone.
• The Gap: Existing automated solutions often rely on predefined rules or static analysis heuristics, which struggle with the diversity of real-world breaking changes and lack the contextual understanding required for complex repairs.
• Focus: This paper specifically targets compilation errors caused by breaking dependency updates in Java projects, as these are the most prevalent form of breakage.

2. Methodology: The Byam Pipeline

The authors propose Byam, an automated pipeline that leverages Large Language Models (LLMs) to generate code patches for fixing breaking updates. The process consists of three distinct steps:

Step 1: Extraction of Failed Build Information

Byam analyzes the build output log after a dependency update to isolate the failure.

• Error Localization: Identifies the specific files and lines causing compilation errors.
• API Differencing: Uses the tool japicmp to generate an API Difference (APIDiff) report. This report details structural changes between the old and new dependency versions (e.g., removed methods, changed signatures).
• Context Aggregation: Compiles the client code, the specific error message, the erroneous line, and the APIDiff.

Step 2: LLM Prompting and Repair Generation

Byam queries an LLM using engineered prompts. The study evaluates eight prompt configurations (P1–P8) to determine the optimal information mix:

• Baseline: Client code + Error message.
• Enhancements:
  • Erroneous Line: Explicitly highlighting the specific line of code causing the failure.
  • APIDiff: Providing the structural differences between dependency versions.
  • Chain-of-Thought (CoT): Instructing the LLM to reason step-by-step (analyze error, compare APIs, plan fix) before generating code.
• Strategy: The system iterates through files, generating a fixed version of the code for each compilation error.

Step 3: Validation

The generated patches are integrated into the client project, and the build is re-executed (typically mvn test).

• Success Criteria: The build must compile successfully and pass all existing tests.
• Granularity Analysis: The system tracks results at three levels:
  1. Build Level: Is the entire project fixed?
  2. File Level: How many individual files were successfully repaired?
  3. Error Level: How many specific compilation errors were resolved vs. introduced?

3. Experimental Setup

• Dataset: The BUMP dataset, containing 103 real-world breaking dependency updates in Java projects (filtered from 571 total updates to exclude Java version incompatibilities and configuration-only failures).
• Models Evaluated: Five LLMs were tested:
  1. Google Gemini-2.0 Flash
  2. OpenAI GPT-4o-mini
  3. OpenAI o3-mini
  4. Alibaba Qwen2.5-32b-instruct (Open Source)
  5. DeepSeek V3
• Metrics:
  • Build Success Rate (BSR): Percentage of projects fully fixed.
  • File Fix Success Rate (FFSR): Percentage of erroneous files fixed in failing builds.
  • Compilation Error Fix Rate (CEFR): Percentage of individual errors fixed.
  • Relative Error Fixed (REF): A metric balancing fixed errors against new errors introduced by the LLM.

4. Key Results

The study demonstrates that LLMs can effectively automate the repair of breaking dependency updates, with performance heavily dependent on the model and prompt design.

• Best Model: OpenAI o3-mini consistently outperformed other models.
  • Build Success: Achieved a 27% success rate (28 out of 103 builds fully fixed) using the most advanced prompt (P8).
  • Partial Repair: Even in builds that did not fully recover, o3-mini fixed 41% of erroneous files and 78% of individual compilation errors.
• Prompt Engineering Impact:
  • The combination of APIDiff, Erroneous Line, and Chain-of-Thought (CoT) reasoning (Prompt P8) yielded the best results.
  • Providing the specific erroneous line and API differences significantly improved performance across most models.
  • CoT reasoning was particularly beneficial for reasoning-capable models like o3-mini but sometimes detrimental to others (e.g., Gemini-2.0 Flash).
• Error Introduction:
  • o3-mini showed high stability, rarely introducing new errors (Median REF of ~80-93%).
  • Smaller models (e.g., Qwen2.5-32b) struggled, often introducing more errors than they fixed (Median REF of 0%).
• Comparison to Prior Work: Byam outperformed a recent agent-based approach (Frunkte & Krinke, 2025) on the same dataset, achieving 27% build success vs. 20%, highlighting the value of rich contextual prompting over iterative agent loops in a single-shot setting.

5. Key Contributions

1. Byam Framework: A novel pipeline for fixing breaking dependency updates that integrates build logs, API differencing, and advanced prompting strategies.
2. Large-Scale Empirical Study: Evaluation of five diverse LLMs across 103 real-world Java projects with eight distinct prompt configurations.
3. Granular Analysis: A multi-level evaluation (Build, File, Error) that reveals LLMs are highly effective at partial repairs, even when full build recovery is not achieved.
4. Replication Package: Full open-source availability of the dataset, code, and experimental results to ensure reproducibility.

6. Significance and Implications

• Developer Productivity: Byam significantly reduces the manual effort required to maintain up-to-date dependencies, a critical task for security and stability.
• Prompt Design Insights: The study proves that context is king. Simply asking an LLM to "fix the code" is insufficient; providing specific error lines, API diffs, and reasoning instructions is crucial for success.
• Future Directions: The authors suggest that while compilation errors are largely solvable, future work must address semantic failures (where code compiles but logic fails due to behavioral changes in the library) and integrate these tools into CI/CD pipelines (e.g., as PR bots).
• Industry Relevance: The findings suggest that current reasoning-oriented LLMs are approaching the capability to handle complex software maintenance tasks autonomously, provided they are guided with precise technical context.