HerAgent: Rethinking the Automated Environment Deployment via Hierarchical Test Pyramid

1. Problem Statement

Automated software environment deployment is critical for testing, debugging, and reproducing failures. However, current automated approaches face three primary limitations:

• Incomplete Success Criteria: Existing methods often define "success" loosely, such as successful dependency installation (pip install) or partial test execution (e.g., pytest collection). These signals do not guarantee that the application's main entry point can actually run.
• Lack of Holistic Understanding: Prior work typically reacts to individual compilation or test errors (local failure signals) rather than reasoning about the repository's global structure, dependency graphs, and execution workflows. This leads to brittle fixes and poor scalability on complex projects.
• Limited External Validity: Many approaches are designed for specific project structures (e.g., Python projects using pytest) and evaluated on small, homogeneous datasets, failing to generalize to diverse languages (especially C/C++) and heterogeneous build systems.

2. Methodology: HerAgent

The authors propose HerAgent, an automated framework that rethinks environment deployment through a Hierarchical Test Pyramid and a Script-Centric Repair Mechanism.

A. The Environment Maturity Hierarchy

Instead of a binary success/fail signal, HerAgent defines a three-level hierarchy of environment readiness, forming a "Test Pyramid":

1. Installability: The environment can successfully execute build or installation commands (e.g., mvn install, pip install). This confirms dependencies are present but does not prove the program runs.
2. Testability: The environment can execute test-oriented commands (e.g., unit tests, smoke tests, --version checks). This provides limited evidence of functionality but often relies on mocks or avoids full execution paths.
3. Runnability: The environment can successfully execute the project's main entry point or integration workflows (e.g., python main.py, CLI startup, end-to-end integration tests). This is the highest level of maturity, indicating the system works end-to-end under realistic conditions.

B. The HerAgent Workflow

HerAgent operates as a multi-agent system using a Knowledge Graph (built via Tree-sitter and Neo4J) to understand the codebase holistically. The workflow consists of three stages:

1. Initial BashFile Generation:

   • The agent retrieves and analyzes environment configuration files (e.g., requirements.txt, pom.xml) to generate an initial executable Bash script.
   • This script includes layers for OS abstraction, base environment preparation, and domain-specific build logic. It serves as the single source of truth for the environment state.

2. Test Pyramid Construction:

   • The agent mines the repository to extract, classify, and refine test commands into the three maturity levels (Installability, Testability, Runnability).
   • It uses a feedback loop to ensure the Test Pyramid is complete, filtering out noise and identifying missing common commands via web search if necessary.

3. Interactive Environment Deployment (The Core Loop):

   • Execution Loop: The agent executes the Bash script in a container. If it fails, it triggers repair.
   • Feedback Loop: Once the script runs, the agent executes commands from the Test Pyramid, starting from lower maturity levels and progressing upward.
   • Bash File Repair Mechanism: Unlike prior works that regenerate scripts or execute isolated commands, HerAgent treats the Bash File as a persistent state carrier.
     • When an error occurs, the agent analyzes the output and generates specific repair commands.
     • These commands are integrated into the Bash File (either by appending or replacing faulty sections) rather than executed in isolation.
     • This "Whole Script + Single Command" hybrid approach prevents context loss and ensures that environment state evolves monotonically.

3. Key Contributions

• Conceptual Novelty: Introduction of the Environment Maturity Hierarchy, the first unified framework defining success via three progressive levels (Installable, Testable, Runnable), shifting the goalpost from "tests pass" to "application runs."
• Approach Novelty: A holistic, script-centric repair strategy. By maintaining a global Bash File and using a knowledge graph for project understanding, HerAgent avoids the "local optima" trap of error-driven, reactive fixes.
• State-of-the-Art Performance: Comprehensive evaluation across four benchmarks (EnvBench, Repo2Run, ExecutionAgent, Installamatic) covering 14 programming languages.
• Artifact Availability: Full release of code, execution trajectories, and experimental results.

4. Experimental Results

HerAgent was evaluated against four related works (PIPER, ExecutionAgent, Repo2Run, Installamatic) and frontier LLMs (GPT-5, Qwen series).

• Overall Effectiveness: HerAgent achieved State-of-the-Art (SOTA) performance across all benchmarks.
  • On EnvBench-Python, it achieved a 4.7x improvement over the best baseline (42/96 vs. 9/96 successes).
  • On Repo2Run-Bench, it reached 110/122 successes, outperforming the previous best (109/122) and zero-shot baselines by 80+ projects.
  • On Installamatic-Bench, it achieved a 62% relative improvement (34/40 vs. 21/40).
• Unique Coverage: HerAgent uniquely resolved 11–30 instances per benchmark that no prior method could configure, demonstrating complementary capabilities.
• C/C++ Performance: On complex C/C++ projects (a known weak point for automated tools), HerAgent surpassed prior approaches by 66.7%. It was the only approach to consistently reach the Runnability state for 17 out of 24 C/C++ projects, whereas prior methods failed to configure them fully.
• Ablation Studies:
  • Interactive Feedback was identified as the most critical component; removing it caused catastrophic failure (0 successes).
  • A hybrid repair strategy (combining Single Command Repair for precision and Whole Script Repair for global consistency) was essential for reaching the Runnability state.

5. Significance

This paper fundamentally shifts the paradigm of automated environment setup:

1. Redefining Success: It argues that passing unit tests is insufficient for "User-Ready" environments. The focus must shift to Runnability (main entry point execution).
2. Scalability: By moving from reactive, error-driven debugging to proactive, holistic script maintenance, HerAgent scales effectively to complex, heterogeneous, and compiled language projects (C/C++).
3. Foundation for Agents: A functional, runnable environment is a prerequisite for autonomous coding agents. HerAgent provides the necessary infrastructure for agents to validate changes and observe real failures, moving beyond static analysis.

In summary, HerAgent demonstrates that treating environment deployment as a state-transition problem guided by a hierarchical test pyramid, and maintaining state via a persistent executable script, yields significantly more robust and complete environment configurations than current state-of-the-art methods.