PITMuS: A Tool for Automated Bug Dataset Generation via Source-Level Mutant Reconstruction

Technical Summary: PITMuS

Problem Statement
The rise of LLM-based software engineering has created a critical demand for executable, context-rich bug artifacts. These artifacts—specifically paired correct and buggy code, methods under test (MUTs), documentation, and metadata—are essential for training and evaluating automated bug localization, repair, and test generation techniques. While curated benchmarks like Defects4J exist, they are static, limited in scope, and increasingly vulnerable to data contamination as code models are trained on large public corpora.

A complementary strategy involves generating fresh, cutoff-aware datasets by selecting real system versions and injecting controlled bugs via mutation testing. In the Java ecosystem, PIT is the state-of-the-practice mutation testing tool. However, PIT operates at the bytecode level for speed and practicality, reporting mutants via XML metadata (mutator type, source file, line number, description) rather than providing the actual mutated source code. This design creates a significant gap: users cannot easily inspect, replay, compile, or reuse PIT mutants as structured source-level dataset records. Furthermore, reconstructing the exact source edit from a line number and mutation description is non-trivial, as a single source line may contain multiple candidate operators (e.g., multiple additions), leading to ambiguity that bytecode-level information resolves but source-level tools often miss.

Methodology
PITMuS is a tool designed to bridge this gap by reconstructing source-level mutants from standard PIT reports without replacing the underlying mutation engine. The workflow integrates three primary inputs:

1. PIT XML Report: Identifies reported mutants.
2. Compiled .class Files: Provide debug information and bytecode-level details.
3. Original Java Source Files: Serve as the base for reconstruction.

The tool operates in two distinct phases:

• Phase 1: Automated Dataset Generation
  PITMuS parses the XML report to extract mutation details. To resolve ambiguities where a line number and description do not uniquely identify a source token, the tool employs a hybrid approach:

  • It uses javalang to parse the original source code and identify candidate tokens.
  • If multiple candidates exist on a line, it utilizes bytecode information from the compiled class file to pinpoint the exact target.
  • Once the specific token is identified, the mutation is applied to reconstruct the mutated statement.
  • The tool then extracts the enclosing method, any preceding Javadoc, and relevant metadata (mutant ID, file path, bytecode index) to form a structured record: <original_method, mutated_method, documentation, metadata>.

• Phase 2: Source-Level Mutant Injection
  This phase materializes the reconstructed mutants as actual Java source files. PITMuS copies the original source file, replaces the target statement with the reconstructed mutated statement while preserving indentation, and validates the result using javalang tokenization. Injection can be performed at four granularities:

  • Mutation-level: A specific mutant by ID.
  • Statement-level: All mutants associated with a specific statement.
  • Class-level: All mutants for a selected class.
  • System-level: All mutants across the entire system.

Key Contributions
The paper presents the following contributions:

1. PITMuS Tool: A system that reconstructs source-level mutants from standard PIT reports, enabling the creation of structured datasets containing buggy/fixed code pairs and documentation context.
2. Dual Utilities: The provision of tools for both source-level mutant injection (creating runnable source files) and method-level dataset construction (generating structured records for training/evaluation).
3. Empirical Evaluation: A comprehensive evaluation on eight open-source Java systems, demonstrating the tool's ability to process and reconstruct a large volume of mutations.

Results
The evaluation was conducted on eight diverse open-source Java systems (including commons-lang3, bcel, and jsoup), covering domains from bytecode engineering to HTML parsing.

• Scale: PITMuS processed 69,229 PIT mutations across 1,913 source files.
• Reconstruction Rate: The tool successfully reconstructed 69,198 aligned original–mutant method pairs, achieving a 99.96% success rate.
• Documentation Context: 45,913 (66.35%) of the reconstructed pairs included non-empty Javadoc, preserving documentation context crucial for LLM training.
• Operator Performance: Six mutation operators (VoidMethodCall, NullReturns, TrueReturns, FalseReturns, Increments, InvertNegatives) achieved 100% reconstruction. The remaining operators maintained rates above 99.69%. The highest failure rate was observed in RemoveConditionals (14 failures out of 35,739).
• Failure Analysis: The 31 unreconstructed mutations (failure rate < 0.05%) were attributed to source-position mismatches. These occurred either when multiple bytecode-level mutants mapped to the same source edit (leading to duplicate avoidance) or when mutations occurred in multi-line expressions where the reported line number differed from the operator's actual line.

Significance
The paper claims that PITMuS addresses a critical need for researchers and developers who require aligned original–mutant code pairs and documentation context without manual processing. By converting standard, report-level mutation artifacts into usable source-level examples, PITMuS:

• Lowers the effort required to build mutation-based datasets.
• Makes PIT mutants accessible to users needing source-level representations for empirical studies, model training, and tool evaluation.
• Provides a scalable, cutoff-aware strategy for generating fresh datasets, mitigating the contamination risks associated with static benchmarks in the LLM era.

The authors emphasize that while evaluated on specific systems, the tool is applicable to any Java system where PIT can be integrated. The code and artifacts are released as open source under the Apache 2.0 license.