TestMap: Evidence Infrastructure for Foundation-Model-Assisted Test Generation

Technical Summary: TestMap – Evidence Infrastructure for Foundation-Model-Assisted Test Generation

Problem Statement
Foundation models (FMs) are increasingly capable of generating plausible unit tests; however, determining their correctness, utility, maintainability, and integration value remains a significant challenge. The core problem is a "recursive validation" issue: generated tests are both code artifacts and validation artifacts. They must be validated before they can be trusted to validate the system under test. Currently, the validation process is fragmented across disparate tools (build systems, test runners, coverage tools, mutation tools, static analyzers), often reporting only aggregate outcomes rather than connecting evidence to individual test candidates. Furthermore, existing approaches frequently discard failed or intermediate candidates, losing valuable data regarding model limitations, missing context, or potential defects in the system under test.

Methodology: The TestMap Infrastructure
The paper presents TestMap, an open-source infrastructure prototype designed to automate evidence-backed FM-assisted test generation and evaluation, specifically targeting C#/.NET repositories. TestMap treats generated tests as "candidates" rather than final answers, recording their full lifecycle.

The system operates through a structured pipeline:

1. Repository Ingestion & Static Analysis: TestMap clones a target repository, uses Roslyn APIs to parse the solution, and performs static analysis to map source code, dependencies, and test structures. It collects code metrics and detects test smells (using a customized version of xNose).
2. Baseline Execution: Before generation, the existing test suite is executed to establish a baseline for coverage, mutation scores, and execution outcomes.
3. Target Selection & Context Construction: Generation targets (e.g., source methods) are selected based on configurable criteria such as uncovered lines, surviving mutants, or related tests. An "evidence package" is constructed, including the target code, related tests, coverage gaps, and framework conventions.
4. Model-Guided Generation: TestMap employs a decomposed generation pipeline (inspired by RUG) rather than single-prompt generation. It breaks the task into planning steps (scenario planning, method naming, arrangement, input planning, action planning, assertion planning) followed by final synthesis. It supports different context modes (e.g., NoHistory vs. ChainedHistory) and pass@k budgets.
5. Validation & Repair: Generated candidates undergo pre-execution validation (syntax extraction, insertion, compiler diagnostics) and post-execution validation (build, run, coverage, mutation). Failed candidates are not discarded; they are preserved with diagnostic data. If enabled, a repair loop uses compiler errors, execution logs, and assertion failures to prompt the model for revisions.
6. Evidence Persistence: All artifacts—including prompts, model responses, build logs, execution results, coverage deltas, mutation outcomes, and repair histories—are persisted in a SQLite database. This allows for granular analysis of individual candidates and aggregated strategy comparisons.

Key Contributions
The paper makes the following contributions:

1. Problem Framing: It frames FM-assisted test generation as a recursive validation problem where generated tests must be validated before serving as evidence.
2. TestMap Prototype: It introduces an open-source evidence infrastructure that automates repository analysis, candidate validation, evidence collection, repair tracking, and experiment logging.
3. Lifecycle Preservation: The system preserves the full lifecycle of candidates, including failed, repaired, low-impact, and evidence-positive outcomes, enabling the study of intermediate results.
4. Repository-Specific Experiments: It facilitates experiments that compare models, prompts, generation strategies, and repair workflows on concrete codebases, moving beyond standardized benchmarks.
5. Evidence Model: It defines a structured evidence model linking candidates to repository context, generation parameters, execution outcomes, and quality indicators.

Results and Findings
The paper does not present a large-scale empirical study with quantitative performance metrics (e.g., "TestMap improved coverage by X%"). Instead, the "results" are the demonstration of the infrastructure's capability to:

• Successfully ingest and analyze real C#/.NET repositories.
• Execute the full generation-to-validation pipeline, including handling build failures and execution errors.
• Persist detailed histories of candidates, distinguishing between ValidationFailed, Validated, ValidatedLowImpact, and ValidatedEvidencePositive outcomes.
• Support repository-specific comparisons of generation strategies (e.g., naive prompting vs. metrics-driven prompting).

The authors note that the infrastructure reveals that generated tests often require repair and that failed candidates provide critical evidence regarding oracle quality, context gaps, and toolchain inefficiencies.

Significance and Claims
The paper claims that TestMap addresses a critical infrastructure gap in software engineering. Its significance lies in shifting the focus from merely producing executable tests to making generated tests observable, repeatable, and comparable.

• Beyond Benchmarks: TestMap complements standardized benchmarks by providing evidence on how FM-assisted generation behaves under the actual constraints of a specific repository (build systems, dependencies, existing test styles).
• Evidence for Decision Making: By preserving failed and repaired candidates, the tool supports developer review and empirical analysis, helping to distinguish between model limitations, toolchain issues, and actual system defects.
• Future Direction: The authors conclude that the future of FM-assisted testing requires infrastructure that not only generates tests but provides the evidence necessary to decide if those tests are useful, maintainable, and trustworthy. TestMap is presented as a step toward this evidence infrastructure, acknowledging open challenges such as assertion quality, flakiness, metric attribution, and developer acceptance.

The paper maintains a modest tone, explicitly stating that the current prototype does not directly study developer decisions or measure improvements in trust calibration, but rather provides the data infrastructure necessary for such future studies.