Technical Summary: Agentic Generation of AST Transformation Rules for Fixing Breaking Updates

Problem Statement

Modern software systems rely heavily on third-party libraries managed via package managers (e.g., Maven, npm). These libraries evolve continuously, often introducing breaking dependency updates that remove methods, alter signatures, or relocate types, causing client projects to fail compilation. While tools like Dependabot and Renovate automate the version update step, they do not repair the resulting compilation failures.

Existing automated repair approaches typically generate project-specific code patches. When the same library update breaks multiple projects, these approaches must repair each project independently, generating non-reusable patches. This limits scalability, as the underlying repair logic (the conceptual fix for the API change) is identical across all affected clients, yet current tools fail to capture and transfer this logic.

Methodology: The BIGBAG Framework

The authors present BIGBAG, an agentic framework designed to generate fixing transformations: structured, standalone, executable programs that encode repair logic at the API level. Unlike project-specific patches, a single fixing transformation derived from one client can be applied to any other client broken by the same dependency update.

Core Workflow

BIGBAG operates through a four-step agentic workflow:

1. Input Context Assembly: The agent receives:
   • The client project with a broken build (identified via mvn compile logs).
   • Documentation for the selected AST transformation engine (Spoon or JavaParser).
   • A transformation template (a Maven project skeleton).
   • The new dependency's API specification and Javadoc (to identify correct replacement signatures).
2. Transformation Generation and Application: The agent executes a generate-apply-verify loop:
   • It analyzes compiler errors and source files to locate incompatible API usages.
   • It generates a standalone Java program (the transformation) that traverses the client's Abstract Syntax Tree (AST), identifies the affected nodes, and rewrites them to conform to the new API.
   • The agent compiles and executes the transformation against the client. If the build fails, the agent uses the feedback to revise the transformation.
3. Verification: BIGBAG verifies that the generated transformation, when applied in isolation, resolves the compilation failure. This ensures the transformation itself is the cause of the fix, not incidental edits made by the agent.
4. Generalization (Transfer): Verified transformations are applied to other client projects affected by the same breaking update to assess cross-project transferability.

Technical Components

• Coding Agents: The system utilizes frontier Large Language Models (LLMs) integrated with coding-agent scaffolds (OpenCode, GeminiCLI) that have file-system and shell-execution capabilities.
• AST Transformation Engines: The study compares two engines:
  • Spoon: Provides a fully resolved type model and a Processor<T> abstraction, offering semantic richness but requiring more scaffolding.
  • JavaParser: Offers a lighter Visitor API with fewer semantic guarantees but simpler syntax for structural changes.
• Models: Four models were evaluated: GPT-5.4-mini, Qwen3-30B, DeepSeek-v3.2, and Gemini-3.1-Pro.

Experimental Evaluation

The authors evaluated BIGBAG on 157 real-world breaking dependency updates from the BUMP benchmark, covering 69 client projects and 70 third-party libraries. The updates spanned PATCH, MINOR, and MAJOR semantic versioning types.

Key Results

The evaluation focused on three Research Questions (RQs):

RQ1: Transformation Well-Formedness

• Metric: Compilable Rule Rate (CRR) – the percentage of executions producing a valid, compilable AST transformation.
• Findings: The best configuration (OpenCode/DeepSeek-v3.2/JavaParser) achieved a CRR of 94.3%. The lowest was 50.3%.
• Engine Impact: The choice of AST engine was a primary determinant of success. JavaParser consistently yielded higher CRRs than Spoon (up to 17.8% difference) due to its simpler API, which reduced "Transformation Bypass" (where agents avoid using the AST library).

RQ2: Transformation Effectiveness

• Metric: Fix Rate (FR) – the percentage of compilable transformations that successfully repair the client project (build passes and tests succeed).
• Findings: The best configuration (GeminiCLI/Gemini-3.1-Pro/JavaParser) achieved a Fix Rate of 78.6%.
• Failure Modes: The dominant failure mode was Client Compilation Failure (48–88% of failures), where the transformation compiled but failed to fix the underlying code (e.g., targeting the wrong constructs).
• Observation: While Spoon offered semantic depth, its complexity led to more runtime errors (Rule Execution Errors). JavaParser's simplicity correlated with higher repair success.

RQ3: Transformation Generalizability

• Metric: Cross-Project Fix Rate – the percentage of external projects fixed by a transformation generated for a seed project.
• Findings: The overall cross-project fix rate was 33.3% (43 out of 129 verified targets).
• Uniformity: Transfer success was highly dependent on usage uniformity. For breaking updates where all clients invoked the affected API element uniformly (e.g., simple package renames), the fix rate exceeded 80%. Failures often occurred because the agent learned only the usage patterns visible in the single seed project, missing variations in other clients.

Key Contributions

1. BIGBAG Framework: The first approach to generate reusable, executable AST transformations for fixing breaking dependency updates, moving beyond project-specific patches.
2. Empirical Study: A systematic evaluation across 157 breaking updates, eight (model, engine) configurations, and three semantic versioning levels, demonstrating the feasibility of agentic repair.
3. Failure Characterization: A detailed analysis showing that:
   • AST Engine Choice is a first-order determinant of success (JavaParser outperformed Spoon in this context).
   • Usage Uniformity is the critical factor for cross-project transferability.
   • Compilation Success is correlated with the complexity of the transformation engine's API.

Significance and Claims

The paper claims that agentic generation of reusable fixing transformations is a viable approach to scalable repair of breaking updates.

• Scalability: By generating a single transformation that applies to all affected clients, BIGBAG addresses the scalability bottleneck of repairing dependency updates in large ecosystems.
• Engine Sensitivity: The work provides the first evidence that the choice of AST transformation engine significantly impacts repair success, suggesting that simpler APIs may be more effective for LLM-based synthesis in this domain.
• Limitations: The authors modestly note that transferability is limited by the "single-project seed" approach. If a client uses the broken API in a way not seen in the seed project, the transformation may fail. They propose future work involving multi-project seeding to capture broader usage patterns.

The study concludes that while current agents can reliably generate compilable transformations and fix a majority of updates within a single project, achieving high cross-project transferability requires addressing the diversity of API usage patterns across the ecosystem.