Technical Summary: Cochise – A Reference Harness for Autonomous Penetration Testing

Problem Statement

Recent advancements in Large Language Model (LLM)-driven autonomous penetration testing have yielded promising results. However, existing systems (e.g., incalmo, cAI, pentestGPT) often combine complex architectural, prompting, and tool-integration choices. This complexity creates a "black box" effect, making it difficult to isolate specific contributions or determine what performance gains are derived from the agent's core logic versus its surrounding infrastructure. Furthermore, the lack of reusable execution and logging infrastructure hinders the comparison of models and architectures under a common protocol. Additionally, realistic security testbeds (like Game of Active Directory, or GOAD) require massive hardware footprints (48–64 GB RAM, ~190 GB storage), creating a barrier to entry for researchers who wish to verify or build upon existing work without provisioning these resources themselves.

Methodology

The authors present Cochise, a minimal, 597-line Python reference harness designed to standardize the experimental interface for autonomous penetration testing.

System Architecture

Cochise operates on a Planner–Executor architecture, decoupling long-term strategic planning from short-term tactical execution:

• Execution Host: Cochise connects to a Linux execution host (e.g., a Kali Linux VM) via SSH. This host has network reachability to the target environment (e.g., GOAD). The codebase is deliberately separated from the testbed to prevent the agent from accidentally destroying its own configuration or logs and to prevent lateral movement back to the agent.
• Planner Module: Acts as the strategic core, maintaining long-term state outside the LLM context. It utilizes a structured textual task representation, specifically a Pentest-Task-Tree (PTT) (a tree-structured Markdown list), and a knowledge component. The planner receives the current PTT, a summary of the last task, and the current worldview to select the next task.
• Executor Module: Functions as a ReAct-style (Reasoning and Acting) agent. It is instantiated per task, connects to the execution host via SSH, and issues commands to solve the specific task assigned by the planner.
  • Ephemeral State: The executor's state is discarded after task completion. This bounds context window costs, isolates LLM-context failures (e.g., malformed reasoning, prompt drift), and forces the planner to serve as the integration point for cross-task knowledge.
  • Self-Correction: Both modules implement the Reflexion pattern to detect and repair invalid command invocations.

Logging and Analysis

Every interaction (LLM API calls, planner decisions, SSH commands, and outputs) is serialized into a structured JSON trajectory log. This format captures:

• Event keys (e.g., strategy updates).
• Exact prompts and raw completions.
• Granular cost metrics (input/output/reasoning/cached tokens).
• Bash command strings and their stdout/stderr outputs.

To address the hardware barrier of live testbeds, the authors provide:

1. cochise-replay: For offline visualization of captured runs.
2. cochise-analyze-logs & cochise-analyze-graphs: Tools to parse JSON logs for cost, token, duration, and compromise analysis, exporting results to LaTeX tables or graphs.
3. Public Corpus: A dataset of JSON trajectory logs from GOAD runs, allowing researchers to study agent behavior without provisioning the heavy testbed.

Key Contributions

1. Minimal Reference Harness: A 597-line Python scaffold that fixes the execution interface, model abstraction, state handling, and trajectory format. It uses the LiteLLM framework for easy model swapping and a configurable scenario prompt for diverse use cases.
2. Architectural Separation: A clear separation between a long-term stateful planner and an ephemeral ReAct-style executor, designed to manage the unbounded action space and side-effecting nature of penetration testing.
3. Reproducibility Tooling: A suite of tools for offline replay and analysis, alongside a public corpus of trajectory logs, enabling research without the need for expensive hardware.
4. Comparative Baseline: A stable scaffold against which alternative agent designs can be ablated and compared.

Results

The authors evaluated Cochise against the Game of Active Directory (GOAD) testbed using two frontier LLMs: Gemini-3-Flash and Claude-4.7-Opus.

• Capability: Both models successfully compromised at least one of the three Active Directory domains in all runs. In 3 out of 5 runs, both models compromised all three domains.
• Efficiency: While both models succeeded, Gemini-3-Flash demonstrated a significantly lower cost profile, with an average cost per account compromised being 12x lower than Claude-4.7-Opus ($0.11 vs. $0.74).
• Complexity: In terms of code size, Cochise (597 core lines of code) is significantly more compact than existing prototypes, which range from 1,822 to 9,100 core lines of code (Table 1).

Significance and Claims

The paper explicitly positions Cochise not as a state-of-the-art penetration testing agent, but as reusable experimental infrastructure.

• Research Utility: Its primary value lies in enabling the comparison of models, agent architectures, and penetration-testing traces under a common protocol.
• Accessibility: By providing a minimal harness and a corpus of pre-generated logs, it lowers the barrier for researchers to study agentic behavior in adversarial environments without incurring token costs or hardware provisioning overhead.
• Ethical Stance: The authors acknowledge the dual-use nature of the research. They emphasize that Cochise is intended to improve reproducibility and comparability for defenders and researchers, rather than serving as an operational red-team assistant. The minimal nature of the code highlights issues inherent to the underlying LLMs rather than the harness itself.