💻 computer science

Prefactory: Automated Discovery and Application of Library-Adoption Refactorings

This paper introduces Prefactory, an automated system that synthesizes executable search heuristics using LLMs to efficiently detect and apply library-adoption refactorings in Python code, achieving significantly higher detection and validation rates than existing baselines on a new benchmark of real-world projects.

Original authors: Islem Bouzenia, Michael Pradel

Published 2026-07-21

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Islem Bouzenia, Michael Pradel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian tasked with organizing a massive, chaotic library where every book is written by a different author using their own unique style. Some authors reinvent the wheel, writing long, complicated instructions for things that already have a perfect, pre-made tool in the library's back room. In the world of computer programming, this is called "refactoring." It's the art of cleaning up messy, hand-written code and swapping it for clean, efficient calls to existing, well-tested libraries. Think of it like realizing you've been hand-knitting a sweater when you could just buy a perfect one from a store. The problem is that finding these opportunities is like searching for a needle in a haystack, especially when the needle doesn't look like a needle at all. The hand-written code might not even mention the library's name; it just does the same thing in a clunky way. For years, computers have struggled to spot these hidden chances to upgrade code because they are too rigid, and asking a super-smart AI to read every single file in a project is too expensive and slow.

Enter Prefactory, a new tool designed to solve this puzzle. Instead of asking a super-intelligent AI to read the entire library of code from scratch every time (which is like hiring a detective to read every book in the library one by one), Prefactory teaches the AI to build a set of custom "metal detectors" first. These detectors are like specialized metal detectors tuned to find specific shapes and sounds of buried treasure. Once the AI builds these detectors, they can scan the entire codebase in the blink of an eye, finding the messy, hand-written code that matches the pattern of a library function. When a match is found, the AI steps in one last time to perform the swap and double-check that the new code works exactly like the old one.

The researchers tested this idea on a massive collection of 100 real-world examples where developers had already successfully swapped hand-written code for library calls. They found that Prefactory was incredibly good at spotting these opportunities. It successfully identified the right files in 75 out of 100 cases and pinpointed the exact functions to fix in 56 cases. In comparison, the next best method, which relied on a standard AI agent reading through the code, only found the right files in 35 cases and the right functions in 32. Even better, Prefactory managed to produce 40 fully validated, working refactorings that passed all the safety tests.

What makes this approach so special is how it saves money and time. Scanning a project with Prefactory takes an average of just 1.3 seconds and costs less than $0.05 per instance to find and fix. The standard AI agent method, by contrast, costs about three times as much and takes much longer because it has to "think" through the whole project every single time. The paper suggests that by turning the AI's knowledge into reusable, executable detectors, we can make code cleanup fast, cheap, and reliable. However, the authors also note that the system isn't perfect yet; it sometimes struggles with very large, complex libraries where the "hand-written" code looks very different from the library's standard patterns, and it can't fix code that involves complex, multi-file architectural changes. But for the vast majority of everyday code cleanup tasks, this new "detector-building" approach seems to be a game-changer.

Technical Summary: Prefactory – Automated Discovery and Application of Library-Adoption Refactorings

Problem Statement

Library-adoption refactoring involves replacing hand-written code implementations with calls to existing library APIs. This practice reduces code size, improves idiomatic quality, and leverages well-tested implementations. However, automating this process is challenging because:

Lack of Explicit Cues: The original hand-written code often does not mention the target library or the specific API, making it difficult for tools relying on import statements or direct references to find opportunities.
Syntactic Variability: Developers implement the same functionality in diverse ways (different variable names, control flows, or helper functions), meaning fixed syntactic patterns or linter rules often fail to generalize.
Limitations of Existing Approaches:
- Static Tools: Linters and modernizers (e.g., Ruff, Pylint) are effective only for manually specified, precise patterns.
- LLM Agents: While Large Language Models (LLMs) can generalize to varied patterns, using them to scan entire codebases via repeated prompting is costly, difficult to reproduce, and hard to control systematically at scale.

The core bottleneck is detection: finding relevant code regions without explicit library mentions and generalizing beyond fixed templates, followed by generating a behavior-preserving replacement.

Methodology: The Prefactory Pipeline

Prefactory is an automated, four-stage pipeline designed to convert LLM semantic knowledge into deterministic, reusable search mechanisms. It operates on a target project and a target library name.

Stage 1: Detector Generation

Instead of prompting an LLM to scan code directly, Prefactory uses an LLM to synthesize executable detectors.

Input: Target library metadata (from PyPI, documentation, and mined examples of past refactorings) and project vocabulary (frequent names from the target project).
Output: Two families of detectors:
1. Lexical Detectors: Regular expressions matching domain terms, API-related names, and textual patterns associated with hand-written implementations.
2. Structural Detectors: AST (Abstract Syntax Tree) matchers that identify algorithmic shapes (e.g., loops over values, arithmetic index computation, accumulation into data structures) regardless of specific variable names.
Validation: Generated detectors are filtered to ensure they compile, do not crash, and do not produce overly broad matches (e.g., matching >20% of files).

Stage 2: Project Scanning

This stage is deterministic and LLM-free.

Prefactory filters out non-source files (e.g., vendored code, tests, documentation).
The validated lexical and structural detectors are executed against the remaining source files.
Matches are recorded, identifying candidate files and functions, along with the specific detector matches that triggered the flag.

Stage 3: Candidate Ranking and Filtering

To manage LLM costs, Prefactory ranks candidates and selects a small subset for rewriting.

Ranking Metric: A tuple $(D(c), B(c), -S(c))$ $(D (c), B (c), - S (c))$ is computed for each candidate $c$ $c$ :
- $D(c)$ : Detector-match strength (weighted sum of matches, prioritizing structural matches).
- $B(c)$ : Density of basic Python implementation patterns (loops, arithmetic, branches) indicating hand-written logic.
- $S(c)$ : Candidate size (smaller is preferred).
Selection: The top 5 files are selected, and within those, the top 10 functions are chosen for the next stage.

Stage 4: Refactoring Generation and Validation

Generation: An LLM is prompted with the candidate function, surrounding context, target library API hints, and the specific detector matches. It attempts to rewrite the function using the target library.
Relevance Filtering: AST-based checks ensure the edit actually calls the target library and is not a cosmetic change.
Validation:
1. Differential Tests: An LLM generates a standalone test script comparing the original and refactored functions on generated inputs (using the original as the oracle). If the test fails, the LLM is asked to repair the edit (up to 3 attempts).
2. Project Tests: If differential tests pass, the refactored code is run against the project's existing test suite to ensure no regressions.

Key Contributions

PrefactoryBench: A benchmark of 100 real-world library-adoption refactorings from 61 open-source Python projects and 18 libraries. It includes reference execution environments, project tests, and held-out tests to evaluate both detection and transformation.
Prefactory System: An automated approach that synthesizes reusable, executable detectors (lexical and structural) from LLM knowledge, enabling efficient, deterministic scanning without repeated LLM calls during the search phase.
Empirical Evidence: A controlled comparison demonstrating that detector synthesis outperforms direct LLM prompting, agentic search, and rule-based tools in both detection accuracy and cost-efficiency.

Results

Evaluated on PrefactoryBench, Prefactory achieved the following:

Detection: Detected 75 instances at the file level and 56 at the function level.
Refactoring: Produced 40 validated refactorings (passing both differential and project tests) from the 56 detected functions.
Comparison: Outperformed the strongest baseline, Codex CLI (which detected 35 files and 32 functions and produced 22 validated refactorings), as well as linters, vanilla LLM prompting, and Semgrep+LLM.
Efficiency:
- Scan Time: Average of 1.3 seconds per project (LLM-free).
- Cost: End-to-end cost is below $0.05 per instance, roughly one-third the cost of the Codex CLI baseline.
- Reusability: Detectors are generated once per project-library pair and reused for scanning, making repeated scans inexpensive.
Real-World Application: In a study on current versions of five benchmark projects, Prefactory surfaced actionable opportunities, with two submitted pull requests accepted by maintainers.

Significance and Claims

The paper claims that detector synthesis is a practical and scalable workflow for library-adoption refactoring. By turning the LLM's semantic knowledge into deterministic search mechanisms, Prefactory avoids the high cost and reproducibility issues of direct agentic search.

The authors note that the approach is most effective when library opportunities expose both vocabulary and recognizable implementation structures. Limitations remain for large, heterogeneous libraries (e.g., numpy, pandas) where opportunities span many API families, and for code involving complex I/O or distributed behavior where differential validation is difficult. The work demonstrates that capturing LLM knowledge in reusable, executable detectors is more effective and cheaper than applying LLM reasoning through repeated, broad queries.