Human-AI Collaboration for Scaling Agile Regression Testing: An Agentic-AI Teammate from Manual to Automated Testing

This paper presents a retrieval-augmented, multi-agent AI system developed in partnership with Hacon that accelerates agile regression testing by automatically generating executable scripts from validated specifications, thereby significantly increasing throughput and reducing manual effort while highlighting the continued necessity of human oversight and clear requirements for quality assurance.

The Gist

Imagine you are running a busy restaurant kitchen. You have a head chef (the Human) and a very fast, incredibly eager sous-chef who is an AI (the AI Teammate).

The Problem: The "Recipe" Bottleneck

In this restaurant, the head chef is great at writing down new recipes (called Test Specifications) for every new dish they want to serve. However, turning those written recipes into actual, cooked dishes (called Automated Test Scripts) takes a lot of time and effort.

• The Reality: The head chef writes recipes faster than the kitchen can cook them.
• The Result: The kitchen gets backed up. They have to cook many dishes by hand (manual testing) every night, which is slow, tiring, and leaves less time to try out new ideas. They are stuck in a "manual-to-automated" gap.

The Solution: The AI Sous-Chef

The researchers built an Agentic AI Teammate. Think of this AI not as a robot that replaces the chef, but as a super-fast sous-chef who works while the head chef is sleeping or doing other tasks.

Here is how this AI teammate works:

1. The Silent Partner: The AI doesn't interrupt the chef. Instead, it waits for the chef to drop a new recipe into a specific folder.
2. The Memory Bank (RAG): Before cooking, the AI looks at a massive library of old recipes and how they were successfully cooked in the past. It uses this memory to guess how to cook the new dish.
3. The "First Draft" Meal: The AI cooks a "first draft" of the dish. It's not perfect yet, but it's a solid start. It also brings a "tasting report" (a summary of what it did and any errors it found).
4. The Human Taste-Test: The head chef comes in, tastes the AI's dish, and says, "Hey, you forgot to chop the onions, and you used the wrong spice. But the sauce is perfect!" The chef fixes the small mistakes and serves the final dish.

How They Work Together (Collaboration)

The paper found that this partnership works best when everyone knows their role:

• The AI does the heavy lifting: It does the boring, repetitive work of writing the initial code (cooking the base). It saves the human from starting from a blank page.
• The Human does the thinking: The human checks if the dish actually tastes right (does it meet the customer's needs?) and ensures it follows the restaurant's style guide (clean code).

The Catch: The AI is very literal. If the recipe says "add salt," the AI might add a whole bucket of salt because it doesn't understand the unwritten rule that "a pinch is enough." The human has to teach the AI these unwritten rules over time.

What They Learned (The Results)

After testing this system in a real industrial kitchen (Hacon, a Siemens company), they found:

• Speed Boost: The AI generated about 30% to 50% of the final code perfectly. The human only had to fix the rest. This meant they could automate tests much faster than before.
• The "Junior Chef" Effect: The AI's first drafts were like the work of a talented but inexperienced junior chef. It was good enough to be useful, but it needed a senior chef to polish it.
• The Importance of Clear Recipes: If the human wrote a vague recipe (e.g., "make it taste good"), the AI got confused. If the recipe was super clear (e.g., "add 2g of salt"), the AI did a great job.
• Learning Together: The humans learned to write better recipes for the AI, and the AI learned to make fewer mistakes. They were "co-adapting."

The Big Takeaway

You can't just press a button and let the AI do everything. The secret to success is Human-AI Collaboration.

Think of it like a dance. The AI leads the first few steps, moving fast and covering a lot of ground. The human follows, then steps in to correct the rhythm and add the flair. If they dance together, they can move faster and better than either could alone.

In short: The AI is a powerful tool that clears the clutter, but the human is still the captain of the ship, steering the quality and ensuring the final result is perfect.

Technical Summary

Here is a detailed technical summary of the paper "Human–AI Collaboration for Scaling Agile Regression Testing: An Agentic-AI Teammate from Manual to Automated Testing."

1. Problem Statement

The paper addresses a critical bottleneck in large-scale agile software engineering, specifically within the context of Hacon (a Siemens company). Despite mature CI/CD pipelines and dedicated test engineering teams, there is a persistent gap between the volume of manual test specifications and the rate at which they can be converted into executable automated scripts.

• The Bottleneck: Manual test specifications are produced faster than they can be automated. While automated coverage grows by only 1–2% per release, manual tests account for 82–87% of the total, growing by 10–20% per release.
• The Consequence: This leads to increased manual testing effort, delayed feedback cycles, and a violation of core agile principles regarding rapid delivery and sustainable development.
• The Gap in Current Solutions: Existing Large Language Model (LLM) approaches often fail to address the nuanced requirements of industrial regression testing, such as maintainability, assertion specificity, alignment with project conventions, and the necessary human governance for quality assurance.

2. Methodology and System Design

The authors propose and evaluate an Agentic AI Teammate designed to function as a "silent collaborator" within agile workflows.

System Architecture: Hacon Test Automation Copilot

The system utilizes a Retrieval-Augmented Generation (RAG) approach combined with a bounded multi-agent workflow. It operates asynchronously to generate first-draft automation artifacts before sprints begin.

• Input: Validated test specifications (exported from Jira in JSON format) containing test data, CI configuration, ordered steps, and metadata.
• Agent Workflow:
  1. Orchestration Layer: Coordinates a bounded loop based on iteration limits and CI availability.
  2. Generator Agent: Uses RAG over historical specification–script pairs to produce candidate test scripts.
  3. Execution: Scripts are executed in a dedicated Jenkins environment.
  4. Evaluator Agent: Analyzes execution logs against a compliance matrix (syntactical correctness, executability, step coverage, semantic correctness).
  5. Reporter Agent: Consolidates results into structured Markdown summaries (one for managers, one for engineers) and execution traces in MLflow.
• Integration: The system integrates into the sprint workflow where engineers review the AI-generated drafts, deciding to accept, refactor, or rewrite them before merging into the regression suite.

Evaluation Strategy

The study employed a mixed-method analysis involving:

• Artifact Analysis: 61 test specifications (Xray format) covering six functional areas.
• Human Review: 5 test engineers independently reviewed and refactored 46 AI-generated scripts.
• Metrics:
  • Quantitative: Likert-scale surveys on code quality, readability, and readiness.
  • Qualitative: Semantic block-level reviews (comparing AI output vs. human-refactored versions) and version control (Git) analysis.
  • Note: Line-level diffs were discarded in favor of semantic analysis to better capture reasoning quality.

3. Key Contributions

The paper makes four primary contributions:

1. C1 (Characterization): An industry-verified characterization of the "manual-to-automated" gap in agile regression testing, quantifying the stagnation of automation coverage despite increased effort.
2. C2 (Architecture): A concrete Agentic AI architecture for test script generation that utilizes RAG and multi-agent orchestration to handle domain-specific constraints.
3. C3 (Evaluation): An early industrial evaluation distinguishing between productivity gains, output quality, and collaboration practices, moving beyond theoretical benchmarks.
4. C4 (Practical Implications): Actionable lessons for scaling regression automation and fostering effective Human–AI collaboration cultures in agile environments.

4. Key Results and Insights

The evaluation yielded significant findings regarding productivity and the nature of Human–AI collaboration:

Productivity Gains

• Code Retention: Approximately 30–50% of the AI-generated code per script remained unchanged after human review.
• Role Perception: Engineers viewed the AI output as comparable to a "junior tester's first draft." It significantly reduced initial authoring time but was not deployment-ready without review.
• Throughput: The system successfully increased test script throughput, helping to narrow the manual-to-automated gap.

Limitations and Challenges

• Literal Interpretation: The AI tended to interpret specifications too literally, failing to grasp implicit domain knowledge and informal conventions that human engineers intuitively understand.
• Coding Standards: AI-generated scripts often violated team coding standards (e.g., lack of separation between test data and logic, missing try-catch blocks) and occasionally "hallucinated" method or API names.
• Specification Dependency: Even high-quality (A–B rated) specifications were insufficient for the AI if they relied on tacit knowledge. The AI requires unambiguous, comprehensive instructions.

Collaboration Culture

• Co-Adaptation: A dynamic learning process emerged. Engineers adapted their review practices (e.g., critically reviewing Markdown reports) and refined their specification writing to better suit the AI.
• Governance: The "Human-in-the-Loop" model proved essential. The AI has bounded autonomy; it cannot introduce artifacts without human approval, ensuring accountability and traceability.

5. Significance and Conclusion

The paper demonstrates that Agentic AI is a viable solution for scaling regression testing in industrial settings, but it is not a "silver bullet" for full automation.

• Shift in Role: The AI acts as a force multiplier, offloading the repetitive, time-consuming task of initial script drafting, allowing human engineers to focus on higher-value tasks like semantic validation, architecture, and complex edge-case handling.
• Future Direction: The authors conclude that sustainable success depends on co-adaptation. Future work must focus on:
  1. Enhancing AI adaptation via specialized agents for deeper semantic evaluation.
  2. Improving human adaptation by standardizing test step wording and authoring guidelines.

Final Takeaway: The integration of agentic AI into agile teams requires a shift from "automation as a replacement" to "automation as a collaborative assistant," where clear specifications and rigorous human oversight remain the pillars of quality and maintainability.