MigrateLib: a tool for end-to-end Python library migration

1. Problem Statement

Library migration involves replacing a software library with an alternative that provides similar functionality. This process is notoriously time-consuming and error-prone because it requires:

• Identifying all usages of the source library.
• Mapping equivalent APIs between the source and target libraries.
• Performing complex code transformations (e.g., changing function signatures, handling asynchronous operations, refactoring control flow).

Existing automated tools generally stop at the API mapping stage (identifying which function replaces which) but fail to perform the actual code transformation in real-world projects. Furthermore, few tools support arbitrary library pairs or handle the full end-to-end process, including context-dependent changes like asynchronous transformations.

2. Methodology

The authors propose a three-phase approach to investigate the feasibility of using Large Language Models (LLMs) for migration and to build a practical tool.

Phase 1: Empirical Study of LLM Capabilities

• Dataset: Used PyMigBench, a benchmark of 314 real-world Python library migrations containing 2,910 code changes.
• Models Evaluated: Llama 3.1-70B, GPT-4o-mini, and GPT-4o.
• Evaluation Metrics:
  • Code Change Correctness: Comparing LLM-generated changes against developer ground truth using AST matching and manual review.
  • Test-Based Correctness: Running unit tests on the migrated code to ensure behavioral preservation.
• Key Findings:
  • GPT-4o performed best, achieving 57% fully correct migrations and 94% correct code changes.
  • LLMs excel at high-cardinality changes (e.g., one-to-many mappings) and handling different API styles (e.g., argparse to click).
  • Challenges Identified:
    1. File Identification: LLMs cannot identify which files need migration without external help.
    2. Incomplete Output: LLMs often skip unchanged code blocks, returning partial files.
    3. Transitive Dependencies: If a migrated function becomes asynchronous, all its callers must also be updated, which LLMs often miss.
    4. Version Compatibility: LLMs tend to use the latest library APIs, which may be incompatible with the project's specific Python or library versions.

Phase 2: Building MigrateLib

Based on Phase 1 findings, the authors developed MigrateLib, a Command Line Interface (CLI) tool that combines LLMs with static and dynamic analysis.

Workflow:

1. Preparation (Static/Dynamic Analysis):
   • Sets up a virtual environment with specific library versions.
   • Identifies files requiring migration using static analysis (parsing import statements) and dynamic analysis (building a runtime call graph via test profiling) to catch indirect usages.
2. LLM Migration Step:
   • Prompts the LLM (GPT-4o) with specific instructions, including source/target library versions, to generate migrated code.
3. Post-Processing Step 1: Merge Skipped Code:
   • Uses difflib to detect code blocks the LLM skipped (unchanged code).
   • Merges these blocks back into the migrated file to ensure file completeness.
4. Post-Processing Step 2: Infer Async Transformation:
   • Detects if a function changed from synchronous to asynchronous.
   • Traverses the call graph to find all transitive callers and automatically adds async/await keywords and pytest.mark.asyncio decorators where necessary.
5. Verification:
   • Runs the test suite after each step to verify correctness.

Phase 3: Large-Scale Evaluation

• Setup: Evaluated on 717 simulated migrations across 175 real-world Python projects (not from the original benchmark).
• Selection Criteria: Projects were filtered for high test coverage (>80%), active maintenance, and the presence of requirements files to ensure reliable testing.
• Metric: "Fully Correct" means 100% of tests that passed before migration still pass after.

3. Key Contributions

1. Empirical Study: A systematic analysis of LLM capabilities in library migration, highlighting that while LLMs are strong (57% fully correct), they struggle with context (file identification, transitive async changes) and completeness.
2. MigrateLib Tool: An open-source, end-to-end CLI tool that automates the full migration lifecycle. It uniquely integrates:
   • Dynamic call graph analysis for file discovery.
   • Post-processing to fix skipped code and infer asynchronous transformations.
   • Version-aware prompting.
3. Evaluation Framework: A novel "simulated migration" evaluation setup that allows for large-scale testing without requiring ground-truth developer changes, relying instead on test suite preservation.

4. Results

• Automation Success: MigrateLib successfully completed 32% of the 717 migrations with 100% correctness (zero manual intervention).
• Partial Success: For the remaining migrations, the tool improved the median test correctness from 65% (LLM only) to 86% after post-processing steps.
• Manual Effort Reduction: For migrations requiring manual fixes, MigrateLib performed a median of 73% of the necessary code changes. Developers only needed to manually edit 27% of the changes.
• Failure Analysis: The primary reasons for incomplete migrations were:
  • Incorrect API usage (arguments, imports).
  • Complex async structural changes (e.g., converting class constructors).
  • Missing context (e.g., dependent libraries or configuration changes not visible in the single file).
  • Test expectations needing updates due to behavioral differences.

5. Significance

• Practical Impact: MigrateLib demonstrates that end-to-end library migration is feasible for a significant portion of real-world projects, drastically reducing the manual effort required (from 100% to ~27% for partial cases).
• Methodological Advance: It moves beyond simple API mapping to a holistic solution that addresses the "last mile" problems of code transformation, such as transitive async updates and file completeness.
• Scalability: The tool supports arbitrary library pairs, making it applicable to a wide range of migration scenarios, not just pre-defined pairs.
• Future Direction: The paper suggests that combining LLMs with static/dynamic analysis and iterative feedback (using test errors to re-prompt) is a viable path toward fully autonomous software maintenance tasks.