Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents

1. Problem Statement

Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating patches. A common behavior in these workflows is for agents to write new tests on the fly (agent-written tests) rather than relying solely on existing human-written test suites.

The core problem addressed is the unclear value of this behavior. While writing tests is a standard software engineering practice, it consumes significant interaction budget (API calls, tokens, and context window).

• Observation: High-performing models like GPT-5.2 achieve top-tier resolution rates (71.8%) while writing almost no new tests (0.6% of tasks), whereas other top models like Claude Opus 4.5 write tests frequently (83% of tasks) with only a marginal performance gain (74.4%).
• Hypothesis: Agents may be mimicking a learned software development routine without the tests actually contributing to the final patch's correctness. If true, this represents a substantial waste of resources.

2. Methodology

The study employs a quantitative analysis of agent trajectories on the SWE-bench Verified benchmark using the mini-SWE-agent framework. This framework provides a "light scaffold" where testing is optional (no hard-coded test planners or enforced stages), allowing the study to isolate the model's intrinsic tendencies.

Experimental Setup:

• Models: Six top-performing LLMs (Claude Opus 4.5, Gemini 3 Pro, GPT-5.2, Kimi K2 Thinking, MiniMax M2, DeepSeek v3.2 Reasoner).
• Data: 500 tasks per model, analyzing interaction logs (trajectories) to extract agent-written test artifacts.
• Analysis Approach: The study is structured around three Research Questions (RQs):
  1. RQ1 (Behavior): What testing behaviors emerge? (Frequency, timing, execution intensity).
  2. RQ2 (Content): What feedback signals do these tests provide? (Assertions vs. value-revealing prints).
  3. RQ3 (Intervention): How does prompting agents to write (or not write) tests affect outcomes and costs?

RQ3 Intervention Design:

• Encourage: Modified prompts for low-test models (GPT-5.2, Gemini 3 Pro) to explicitly instruct them to write new test files.
• Discourage: Modified prompts for high-test models (Kimi K2, DeepSeek v3.2) to explicitly forbid writing new test files.
• Metrics: Task resolution success, API calls, and token usage (input/output).

3. Key Contributions

1. Behavioral Analysis: A comprehensive characterization of agent-written testing as a model-dependent process style rather than a universal driver of success.
2. Feedback Signal Analysis: A novel classification of test content, distinguishing between verification-oriented assertions and observational value-revealing prints.
3. Prompt-Intervention Study: A controlled experiment demonstrating that altering test-writing behavior via prompts has minimal impact on task resolution but significant impact on efficiency costs.

4. Key Results

RQ1: Testing Behaviors

• Frequency: Test writing is highly model-dependent. GPT-5.2 writes almost no tests, while MiniMax M2 and Kimi K2 write tests in >97% of tasks.
• Correlation with Success: There is no strong correlation between test-writing frequency and task resolution. Unresolved tasks often write more tests than resolved ones, suggesting that struggling agents may over-test without success.
• Timing: Test writing typically occurs late in the trajectory (spanning 0.4 to 0.8 of the interaction steps), often serving as a final check rather than an iterative debugging loop.

RQ2: Feedback Signals

• Observational Dominance: When agents write tests, the feedback is overwhelmingly observational.
  • Prints vs. Assertions: Value-revealing print statements vastly outnumber assert statements (e.g., ~83% prints vs. ~17% assertions for Claude Opus).
  • Print Categories: Most prints are used for value/content inspection (checking specific runtime values) rather than structural summaries or exception handling.
  • Assertion Types: When assertions are used, they are mostly exact-value checks (C4) or property checks (C2), with very few relational or range-based constraints.
• Implication: Agents use tests primarily as debugging probes to inspect state, not as rigorous oracles to validate correctness.

RQ3: Intervention Outcomes & Costs

• Outcome Stability: Prompt-induced changes in test-writing behavior (encouraging or discouraging) resulted in statistically insignificant changes in task resolution rates for all models.
  • Example: Encouraging GPT-5.2 to write tests increased test creation but did not improve the 71.8% success rate.
  • Example: Discouraging Kimi K2 from writing tests reduced test creation by ~68% but only dropped the success rate by ~2.6%.
• Efficiency Impact: While outcomes remained stable, costs changed drastically.
  • Discouraging Tests: Reduced API calls by ~35% and input tokens by ~49% for high-test models.
  • Encouraging Tests: Increased output tokens by ~20% for GPT-5.2 without any success gain.

5. Significance and Implications

• Rethinking "Testing Helps": In high-autonomy agent settings, the intuition that "more testing equals better results" is flawed. Current agent-written tests often act as a habitual routine that consumes budget without providing actionable validation signals.
• Process vs. Outcome: The primary value of agent-written tests in current models is process modification (changing the interaction pattern and cost), not outcome improvement.
• Practical Recommendations:
  • For Practitioners: Do not simply force agents to write more tests. Instead, focus on targeted testing (e.g., generating minimal regression tests) or converting observational prints into assertions to improve signal quality.
  • For Researchers: Future benchmarks should move beyond binary success/failure metrics. They must account for process efficiency and the quality of feedback signals (assertions vs. prints) to understand agent behavior.
  • For Benchmarks: Final resolution rates on SWE-bench may be too coarse; they should be interpreted alongside process-sensitive metrics (token usage, test creation rates) to avoid misleading conclusions about model capabilities.

Conclusion: The paper concludes that agent-written tests currently function more as a costly, model-dependent habit than a reliable validation mechanism. Improving agent performance requires better oracles and actionable feedback, not merely increasing the volume of test generation.