🤖 AI

TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis

This paper introduces TDAD, an open-source tool that leverages graph-based impact analysis to provide AI coding agents with targeted test context, significantly reducing code regressions and improving issue-resolution rates compared to both baseline approaches and procedural TDD instructions.

Original authors: Pepe Alonso, Sergio Yovine, Victor A. Braberman

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pepe Alonso, Sergio Yovine, Victor A. Braberman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a very talented, but slightly reckless, robot assistant to fix a leak in your house. The robot is great at finding the leak and patching it (solving the problem). However, because the robot doesn't fully understand how the pipes connect to the rest of the house, it often accidentally bursts a pipe in the kitchen while fixing the bathroom.

In the world of software, this is called a regression. The robot fixes the bug you asked for, but breaks something else that was working fine.

This paper introduces a new tool called TDAD (Test-Driven Agentic Development) to stop these accidents. Here is the story of how it works, explained simply.

The Problem: The "Fix-It-All" Robot

Currently, AI coding agents are like that robot. We measure their success by asking: "Did they fix the bug?" If yes, we give them a gold star. We rarely ask: "Did they break anything else?"

The authors found that these AI agents are actually quite good at fixing bugs, but they are terrible at knowing what not to touch. When they try to fix a problem, they often break 6 or 7 other things in the process. This is why many AI fixes get rejected by human bosses in real companies.

The Old Way: "Just Be Careful!"

Researchers tried a common-sense approach first. They told the AI: "Hey, remember Test-Driven Development! Write a test first, then fix the code, then check the test."

They thought this would make the AI more careful. It backfired.

The Analogy: Imagine telling a nervous driver, "Remember the rules of the road! Look left, look right, check mirrors, signal, then drive."
The Result: The driver got so overwhelmed by the long list of instructions that they forgot how to actually drive and crashed more often.
The Paper's Finding: Giving the AI a long list of procedures (how to do things) actually made it break more code because it used up its brainpower (memory) on the instructions instead of the actual problem.

The New Way: TDAD (The "Map" Strategy)

Instead of telling the AI how to think, the authors gave it a map.

Building the Map: Before the AI starts working, TDAD scans the entire codebase and draws a dependency map. It connects every piece of code to the specific tests that check if that code works.
- Analogy: It's like a master electrician's blueprint that shows exactly which light switches control which lights in the house.
The Handoff: When the AI is asked to fix a leak in the bathroom, TDAD hands it a sticky note that says: "You changed the bathroom pipe. You must check these 3 specific switches to make sure you didn't turn off the kitchen lights."
The Result: The AI doesn't need to guess. It has a short, clear list of exactly what to verify.

The Results: A Miracle of Precision

The team tested this on a famous benchmark (SWE-bench) using open-source AI models running on regular computers (not supercomputers).

The "No Map" Group: The AI broke about 6% of the working code.
The "Long Instructions" Group: The AI broke nearly 10% of the code (the instructions confused it).
The "TDAD Map" Group: The AI broke only 1.8% of the code.

That is a 70% reduction in accidents!

Furthermore, in a second experiment with a different AI, giving it this map actually helped it solve more problems (from 24% to 32%) because the AI wasn't wasting energy guessing what to check.

The "Self-Improving" Twist

The coolest part? The authors built a loop where the AI could improve the TDAD tool itself.

The AI looked at its own instructions and realized, "Hey, these instructions are too long and confusing. Let's make them shorter."
It simplified the instructions from 107 lines down to just 20 lines.
Result: The success rate jumped from 12% to 60%. The AI realized that less talk, more data was the secret sauce.

The Big Lesson: Context > Instructions

The paper teaches us a valuable lesson about talking to AI:

Don't give the AI a long lecture on how to do its job (procedures).
Do give the AI the specific, relevant facts it needs to do its job (context).

If you want an AI to be a good coder, don't tell it to "be careful." Give it a map of the danger zones so it can navigate safely on its own.

Summary

TDAD is a tool that builds a "dependency map" for AI coders. Instead of lecturing the AI on how to write code, it simply points out exactly which tests to run to ensure nothing breaks. This simple change reduced code breakage by 70% and proved that for AI, good information is better than long instructions.

1. Problem Statement

AI coding agents (LLM-based) have demonstrated high success rates in resolving software issues (e.g., on SWE-bench), but they frequently introduce regressions—breaking tests that previously passed.

The Gap: Current benchmarks focus almost exclusively on resolution rate (did the agent fix the issue?), ignoring regression rate (did the agent break existing functionality?).
The Consequence: Real-world maintainers often reject agent-generated pull requests due to CI/CD failures and regressions.
The Challenge: Agents lack dependency awareness. They cannot efficiently determine which tests to run to verify a change without running the entire suite (too slow) or guessing (missing indirect dependencies). Traditional Regression Test Selection (RTS) methods are designed for CI pipelines (post-commit), not for agentic workflows where self-correction must happen before submission within a limited context window.

2. Methodology: TDAD (Test-Driven Agentic Development)

TDAD is an open-source tool that equips AI agents with structural knowledge via graph-based impact analysis. Instead of teaching agents how to perform Test-Driven Development (TDD), it tells them which tests to check.

Core Architecture

Offline Indexing (Stage 1):
- Parses a Python repository to build a Code–Test Dependency Graph.
- Nodes: Files, Functions, Classes, Tests.
- Edges: CONTAINS, CALLS, IMPORTS, TESTS (linking tests to code), INHERITS.
- Uses static analysis (AST parsing) to resolve relationships without runtime execution.
Impact Analysis (Stage 2):
- Given a set of changed files, the system traverses the graph to identify "at-risk" tests.
- Scoring Strategy: Combines four strategies with weighted confidence:
  - Direct: Tests directly linked to changed code (Weight: 0.95).
  - Transitive: Tests linked via 1–3 call-chain hops (Weight: 0.70).
  - Coverage: File-level dependencies (Weight: 0.80).
  - Imports: Files importing the changed module (Weight: 0.50).
- Outputs a lightweight, static text file (test_map.txt) mapping source files to relevant tests.
Agent Integration (The "Skill"):
- The agent receives a concise 20-line skill definition (SKILL.md) and the test_map.txt.
- Workflow: Fix the bug $\rightarrow$ grep the test map for related tests $\rightarrow$ Run tests $\rightarrow$ Self-correct if failures occur.
- Constraint: Requires no external databases (like Neo4j) or APIs at runtime; uses standard tools (grep, pytest).

3. Key Contributions

TDAD Tool: An open-source, pip-installable tool for graph-based test impact analysis that reduces regressions without requiring heavy infrastructure.
Benchmark Methodology: Proposes elevating regression rate to a first-class metric alongside resolution rate, arguing that resolution without regression safety is insufficient.
The "TDD Prompting Paradox": Empirical evidence showing that providing procedural TDD instructions (e.g., "write tests first") without specific test context actually increases regressions, especially in smaller models.
Autonomous Self-Improvement: A loop where an agent iteratively refines TDAD's own configuration (e.g., simplifying prompts, improving heuristics), significantly boosting performance.
Open Data: All code, logs, and 28 experiment records are publicly available under the MIT license.

4. Experimental Results

The evaluation was conducted on SWE-bench Verified using two distinct phases and models.

Phase 1: Regression Reduction (100 Instances)

Model: Qwen3-Coder 30B (4-bit quantized, consumer hardware).
Baselines: Vanilla prompt vs. TDD procedural prompt vs. TDAD (Graph + TDD).
Key Findings:
- Regression Reduction: TDAD reduced test-level regressions by 70% (from 6.08% to 1.82%).
- Failure Count: Reduced P2P (Pass-to-Pass) failures from 562 (Vanilla) to 155 (TDAD).
- The Paradox: A "TDD Prompt" (procedural instructions only) increased regressions to 9.94% (worse than vanilla). This occurred because verbose instructions crowded out context tokens, and the agent attempted ambitious fixes without knowing which tests to verify.
- Resolution: TDAD maintained a similar resolution rate (29% vs. 31% vanilla) but with significantly higher safety.

Phase 2: Generalization as an Agent Skill (25 Instances)

Model: Qwen3.5-35B-A3B + OpenCode agent framework.
Setup: TDAD deployed as a reusable skill with a NetworkX backend.
Key Findings:
- Resolution Improvement: Resolution rate increased from 24% to 32%.
- Generation Improvement: The rate of generating non-empty patches rose from 40% to 68%.
- Mechanism: The test map provided structural context that helped the agent navigate the codebase and generate valid patches, not just verify them.

Auto-Improvement Loop

An agent iteratively modified TDAD's source code over 15 iterations.
Result: Resolution jumped from 12% to 60% with 0% regression.
Key Insight: The most impactful change was simplifying the SKILL.md from 107 lines of detailed instructions to 20 lines of concise guidance, proving that smaller models benefit more from context density than procedural verbosity.

5. Significance and Implications

Context > Procedure: For AI coding agents, especially smaller models, providing targeted structural context (which tests to run) is far more effective than prescribing procedural workflows (how to write tests).
Redefining Success: The paper argues that the AI coding community must stop rewarding agents solely for fixing issues if they break other parts of the system. A composite metric (Resolution - $\alpha$ $\times$ Regression) is proposed.
Practical Deployment: TDAD demonstrates that complex dependency analysis can be distilled into lightweight, static artifacts (text files) that fit within LLM context windows, making advanced regression safety feasible for consumer-grade hardware.
Future Direction: The work suggests a shift from "agentic reasoning" (trying to reason through the whole codebase) to "agentic tooling" (using external, pre-computed maps to guide reasoning).

Conclusion

TDAD successfully bridges the gap between static code analysis and agentic development. By replacing verbose procedural instructions with precise, graph-derived test impact maps, it drastically reduces code regressions while maintaining or improving issue resolution rates. The findings highlight a critical design principle for AI agents: surfaces of relevant context outperform instructions on process.