Technical Summary: TAM-Eval

Problem Statement

While Large Language Models (LLMs) have demonstrated significant promise in software engineering tasks such as code generation and isolated test creation, their application to unit test maintenance remains underexplored. Current research largely focuses on generating single test functions or predicting oracles, neglecting the complex, iterative lifecycle of maintaining entire test suites as production code evolves. In practice, failing to maintain tests leads to outdated suites, broken pipelines, and undetected regressions. Existing evaluation benchmarks often lack repository-level context, rely on function-level granularity, or fail to simulate realistic maintenance workflows involving creation, repair, and updating of test files.

Methodology

The authors introduce TAM-Eval (Test Automated Maintenance Evaluation), a framework and benchmark designed to evaluate LLM performance across three core test maintenance scenarios: creation, repair, and updating.

Benchmark Construction

The dataset comprises 1,539 validated scenarios derived from Python, Java, and Go open-source repositories. The construction process involved a rigorous four-stage filtering pipeline to ensure high quality and minimize data contamination:

1. Repository Selection: Filtered for recent updates (post-2020/2025), permissive licenses, and contributor activity.
2. Execution-based Filtering: Ensured projects were buildable and runnable within 30 seconds. Tests were executed twice to eliminate flaky tests, and pairs were retained only if the original suite achieved >40% line coverage.
3. Content-based Filtering: Excluded samples with trivial functions, excessive comments (>70%), auto-generated code, or insufficient test case counts. Balanced sampling was applied to prevent over-representation of large repositories.
4. Task Creation:
   • Test Creation: Simulated by clearing test files, adding coverage to partially tested code, or recovering removed test cases.
   • Test Repair: Introduced synthetic defects (syntax errors, execution-targeted faults, coverage-targeted breakdowns, and efficiency-targeted breakdowns) using tree-sitter heuristics and LLM prompting.
   • Test Updating: Simulated by reverting test files to previous commits while keeping the production code current, creating a mismatch that requires adaptation.

Evaluation Framework

TAM-Eval employs a reference-free, fully automated evaluation pipeline running in sandboxed Docker environments. It does not rely on ground-truth test code for scoring but instead uses execution-based metrics:

• Pass Rate: The ratio of successful test executions to total attempts.
• $\Delta$TestCov: The relative improvement in line-level code coverage.
• $\Delta$MutCov: The relative improvement in mutation score (the ability of tests to "kill" injected mutants).

The framework supports iterative recovery, allowing models up to three attempts with feedback (e.g., compiler errors, stack traces) to refine their outputs.

Key Contributions

1. Holistic Maintenance Scenarios: Unlike prior work limited to function-level generation, TAM-Eval operates at the test file level, addressing the complete cycle of generation, repair, and update within a unified context.
2. Curated Real-world Dataset: A high-quality dataset of 1,539 samples from actively maintained repositories, filtered to ensure buildability, stability, and relevance.
3. Unified, Feedback-Aware Evaluation: A system-agnostic protocol using a single prompt template and iterative execution feedback to simulate practical developer workflows.
4. Extensible Open-Source Suite: A modular framework supporting multiple languages (Python, Java, Go) and tools (e.g., coverage.py, JaCoCo, mutpy, PIT), released to facilitate reproducibility.
5. Empirical Analysis: A comprehensive evaluation of state-of-the-art (SOTA) LLMs, revealing performance gaps and specific failure modes.

Experimental Results

The authors evaluated several SOTA models, including GPT-5, GPT-OSS-120B, Qwen3 Coder 480B A35B, and others.

• Overall Performance: Even the best-performing models exhibit limited capabilities in realistic test maintenance. GPT-5 achieved the highest Pass Rate (42.37%) and the largest gains in coverage ($\Delta$TestCov: +20.8%, $\Delta$MutCov: +11.7%). However, most models showed marginal improvements in mutation coverage, rarely exceeding 12 percentage points.
• Iterative Improvement: Performance generally improved with multiple attempts and feedback, highlighting the importance of automated verifier signals (compilers, test runners). However, initial attempts without feedback often resulted in low pass rates.
• Language Variance: Go emerged as the most suitable language for current LLMs, likely due to its concise syntax and strict typing, yielding the highest Pass Rates and coverage gains. Java showed lower coverage improvements despite reasonable pass rates, suggesting that test executability does not always translate to meaningful coverage in verbose, statically-typed languages.
• Task Difficulty: Update tasks proved the most challenging, with reduced mutation coverage gains compared to creation and repair tasks.
• Failure Modes: The dominant failure mode across all models was execution errors (runtime/compilation failures), accounting for over 60% of invalid outputs. Syntax errors were relatively rare, indicating models have mastered surface-level formatting but struggle with semantic correctness and environment integration.

Significance and Claims

The paper claims that TAM-Eval provides the first comprehensive, automated benchmark for evaluating LLMs in the context of unit test maintenance rather than just generation. The authors emphasize that while LLMs show potential, current state-of-the-art models are not yet reliable for autonomous test maintenance, particularly in complex update scenarios.

The work highlights a critical need for:

• Developing more context-aware methods that better understand repository-level dependencies.
• Distilling supervision from verifiers such as compilers, mutation testing, and coverage signals to guide model training.
• Moving beyond isolated test generation to address the full lifecycle of test suite evolution.

The authors release TAM-Eval as an open-source framework to support future research in automated software engineering, noting its utility as a validation set for training reward models in downstream tasks.