Technical Summary: Testing Framework Migration with Large Language Models

Problem Statement

Python developers predominantly rely on two testing frameworks: unittest (part of the standard library) and Pytest (a third-party framework). While Pytest offers significant advantages, including simpler assertions, reusable fixtures, and better interoperability, migrating existing test suites from unittest to Pytest remains a manual, time-consuming, and error-prone process. Although Pytest can execute unittest tests to allow for gradual migration, the complexity of test suites often leads to migrations taking months or years, or remaining incomplete. Furthermore, maintaining dual frameworks increases maintenance overhead. Despite the proliferation of Large Language Models (LLMs) in software engineering tasks such as code generation and bug fixing, no prior study has explored their capability to automate the migration of test frameworks.

Methodology

The authors conducted an empirical study to evaluate the capability of LLMs to automate the migration of Python tests from unittest to Pytest. The study design involved the following steps:

1. Dataset Construction: The researchers analyzed the top 100 most-starred Python projects on GitHub. Using a detection tool, they identified 690 migration commits across 37 projects. From these, they manually curated a dataset of 923 isolated migrations (changes strictly related to framework migration without unrelated code modifications).
2. Selection for Execution: To ensure validity, the authors filtered the dataset to include only migrations that were still present and executable in the projects' latest versions as of September 2025. This resulted in a final set of 40 isolated migrations from seven projects (including ansible, httpie, redis-py, and apache/airflow). These were classified as either "simple" (30 cases, involving direct assertion/class changes) or "complex" (10 cases, involving fixtures, mocks, and structural adaptations).
3. LLM Configuration: Two state-of-the-art models were evaluated: GPT-4o and Claude Sonnet 4.
   • Prompting Strategies: Three strategies were tested: Zero-shot, One-shot, and Chain-of-Thought (CoT).
   • Temperature Settings: Two settings were used: 0.0 (deterministic) and 1.0 (neutral).
   • Total Variants: This configuration generated 480 migration attempts (40 migrations × 2 models × 3 strategies × 2 temperatures).
4. Evaluation: The LLM-generated migrations were executed within the original project environments. The authors compared the results against the developer-migrated baseline using two metrics:
   • Correctness: Whether the test executed successfully (PASSED vs. FAILED).
   • Coverage: Whether the code coverage remained identical to the original test.
   • Error Analysis: Categorization of failure types (e.g., AssertionError, TypeError, SyntaxError, missing fixtures).

Key Contributions

The paper makes two primary contributions:

1. First Empirical Study on Test Framework Migration: It provides the first investigation into using LLMs to automate the migration of test frameworks, specifically from unittest to Pytest.
2. Comprehensive Dataset and Analysis: The authors introduce the TestMigrationsInPy dataset, containing 923 real-world migrations, and provide a detailed analysis of LLM performance across different models, prompting strategies, and migration complexities.

Results

Correctness (RQ1)

• Overall Success Rate: LLMs achieved an overall correctness rate of 48.54% (233 out of 480 migrations passed). Conversely, 51.5% failed.
• Model Performance: Claude Sonnet 4 slightly outperformed GPT-4o (51.66% vs. 45.41% success rates).
• Prompt Strategy: Chain-of-Thought yielded the best overall correctness (34.76%). One-shot prompting generally resulted in lower success rates, potentially due to representational bias or over-specialization to the provided example.
• Temperature: Temperature settings (0.0 vs. 1.0) had a negligible impact on correctness for both models.
• Difficulty: Simple migrations had significantly higher success rates than complex ones. While both models struggled with complex scenarios (involving fixtures and mocks), Claude Sonnet 4 performed better in complex cases compared to GPT-4o.

Coverage (RQ2)

• For all successful migrations, test coverage remained exactly the same as the original developer-migrated tests. This indicates that when LLMs succeeded, they preserved the original test behavior and intent without altering runtime coverage.

Error Analysis (RQ3)

• The 247 failures were categorized into six main types:
  • TypeError: The most frequent error (156 cases), often caused by missing dependencies or incorrect fixture substitutions (e.g., in redis/redis-py).
  • SyntaxError: Malformed code blocks (e.g., unbalanced try/except), notably in apache/airflow.
  • AssertionError: Subtle changes in values or formatting.
  • Structural Mismatch: Issues with test discovery or class-based dependencies.
  • Missing Fixtures: Failure to reference necessary fixtures or variables.
  • Signature Drift: Unintended modifications to function parameters.
• Failures were primarily attributed to the models' inability to handle context-sensitive instructions, such as fixture relationships and external dependencies, rather than simple syntax translation.

Migration Properties (RQ4)

• Structural Differences: Claude Sonnet 4 exhibited a conservative migration style, preserving class-based structures (70.83% of cases) and legacy unittest references. GPT-4o favored more aggressive transformations, converting class-based tests to function-based ones (only 45% preserved classes).
• Redundancy: There was a high degree of code duplication across configurations. Claude Sonnet 4 showed high convergence (up to 92.50% duplication in Zero-shot), while GPT-4o was more diverse. This suggests that prompt design influences convergence more than temperature.
• Idiomatic Usage: Both models rarely utilized advanced Pytest features (e.g., parameterization) unless explicitly prompted, preferring direct one-to-one syntax translations over more idiomatic, maintainable solutions.

Significance and Implications

The paper concludes that while LLMs can accelerate test migration, they are not yet a fully autonomous solution for complex scenarios.

• Feasibility: LLMs can successfully migrate simple, syntactic changes (assertions, class-to-function conversion) with high reliability.
• Limitations: The models struggle with architectural coherence, fixture adaptation, and dependency management. They tend to preserve legacy patterns (in the case of Claude) or fail to apply advanced framework features (in both models) without explicit guidance.
• Practical Guidance: The study suggests that for practitioners, Chain-of-Thought prompting is the most effective strategy, and temperature tuning offers little benefit for this specific task. The results highlight the need for future approaches that incorporate detailed, fine-grained context about the test environment to help LLMs understand relationships between fixtures and external components.
• Future Work: The authors propose investigating coding agents for autonomous testing, contribution studies (submitting PRs to open-source projects), and deeper quality assessments of LLM-migrated tests against industry-grade repositories.