Technical Summary: Fully Automated End-to-End Adversary Emulation from CTI Using LLMs

Problem Statement

Adversary emulation is a critical method for validating organizational defense systems by replicating specific Tactics, Techniques, and Procedures (TTPs) of threat groups. However, achieving fully automated, end-to-end adversary emulation remains a significant challenge. Existing approaches suffer from fragmentation:

• Manual Playbook Generation: Tools like MITRE Caldera and PurpleSharp automate execution but rely on security experts to manually author playbooks, creating a scalability bottleneck.
• Partial Automation: Recent systems like AURORA use Large Language Models (LLMs) to generate playbooks from Cyber Threat Intelligence (CTI) reports but still require manual intervention for payload serving and execution. Crucially, they lack mechanisms to diagnose or repair execution failures, often proceeding with invalid attack chains when commands fail due to environmental differences.

The core problem is the absence of a unified workflow that integrates playbook generation, execution, and failure recovery without human intervention, particularly when translating unstructured, natural-language CTI reports into executable attack scenarios.

Methodology

The authors propose a fully automated framework that transforms MITRE ATT&CK-aligned CTI reports into executable Caldera playbooks using LLMs. The system architecture consists of four functional layers:

1. Input Layer

• CTI Reports: The system ingests structured CTI reports (e.g., from KISA) containing TTPs, operational details (IPs, file paths), and mitigation strategies.
• Environment Specification: A document provided by a security operator detailing the target environment (OS versions, network topology, credentials). This allows the LLM to abstract environment-specific details from the CTI report and concretize them for the test environment.

2. LLM-Based Playbook Generation Layer

This layer transforms the input into a Caldera Adversary through three steps:

• Step 1 (Abstract Flow): The LLM extracts attack goals from the CTI text, identifying objectives, tactics, and methods to create an environment-independent abstract attack flow.
• Step 2 (Concrete Flow): The LLM combines the abstract flow with the environment specification to generate specific nodes. Each node contains an executable command, a MITRE ATT&CK Technique ID, and concrete parameters (IPs, paths). Logical dependencies are established to determine execution order.
• Step 3 (Caldera Conversion): Nodes are converted into Caldera Abilities (executable units) and assembled into an Adversary (playbook) in abilities.yml and adversaries.yml formats.

3. Execution Layer

The generated Adversary is uploaded to the MITRE Caldera platform. A Caldera Agent on the target host executes the Abilities and returns results (exit codes, stdout, stderr) to the system.

4. Failure Recovery Layer

This is a novel component designed to handle the heterogeneity of attack-simulation failures.

• Failure Classification: Execution errors are classified into four types using regex patterns derived from observed error logs:
  1. Syntax Error: Command parsing or escaping issues.
  2. Dependency Error: Privilege issues, authentication failures, or cascading failures from prior steps.
  3. Missing Environment: Absence of required resources (files, network endpoints).
  4. Timeout: Execution exceeding time limits.
• Recovery Strategy: For recoverable failures, the system constructs a new prompt for the LLM containing the original command, the specific failure type definition, accumulated revision history, and the environment specification. The LLM generates a revised command, which is re-executed. This loop continues up to three times or until success.

Key Contributions

1. Unified End-to-End Framework: The paper presents the first system to integrate playbook generation, execution, and failure recovery into a single automated workflow, eliminating the need for manual intervention at any stage.
2. Failure-Type-Aware Recovery Mechanism: Unlike general code correction, this mechanism is domain-specific, classifying failures into four distinct categories and applying tailored recovery strategies. It leverages accumulated revision history to prevent repeated errors.
3. Comprehensive LLM Evaluation: The study evaluates four commercial LLMs (Claude Sonnet 4.5, GPT-4o, Gemini 2.5 Pro, and Grok 4 Fast), demonstrating that different models exhibit distinct strengths in fidelity, chain length, and recovery capability.

Experimental Results

The framework was evaluated on 11 high-quality CTI reports (averaging 41.5 pages each) using a controlled Windows environment (PC1, PC2, PC3).

Performance Metrics (Best Model: Claude Sonnet 4.5)

• ATT&CK Conformance: 94.91% of generated Abilities corresponded to valid MITRE ATT&CK techniques.
• CTI Fidelity: Achieved a CTI F1-score of 60.50% (Precision: 73.95%, Recall: 52.48%).
• Execution Success:
  • Initial execution success rate: 69.63%.
  • Final execution success rate (after recovery): 84.22%.
  • Improvement: The failure recovery mechanism increased success rates by 14.59 percentage points on average across all models.
• Efficiency: Generating and revising a playbook took an average of 197.39 seconds (approx. 3.3 minutes) and cost $0.35 in LLM tokens per report.

Comparison with State-of-the-Art (AURORA)

Using 10 CTI reports from the AURORA dataset:

• Execution Success: The proposed system achieved a final success rate of 65.17%, surpassing AURORA's 60.72%.
• Fidelity: The system achieved a higher CTI F1-score (30.57%) compared to AURORA (26.07%).
• Recovery Impact: While AURORA had a higher initial success rate (60.72%) due to using a pre-built command library, the proposed system's initial rate was lower (32.90%) because it generates commands from scratch. However, the recovery mechanism boosted the proposed system's success rate by 32.27 percentage points, ultimately outperforming AURORA.

Significance and Claims

The authors claim this work represents a significant step toward practical, scalable adversary emulation. By automating the entire lifecycle from CTI report to successful attack execution, the system addresses the scalability bottleneck of manual playbook authoring.

The paper emphasizes that the failure recovery mechanism is the critical differentiator. In attack simulation, environmental nuances often cause commands to fail where they would succeed in a different context. The ability to automatically diagnose these failures (e.g., distinguishing between a syntax error and a missing environment variable) and revise the playbook without human intervention is essential for maintaining the validity of the attack chain.

The study concludes that while the current system assumes a pre-prepared environment, it successfully demonstrates that fully automated, end-to-end adversary emulation is feasible, offering substantial time and cost savings while maintaining high fidelity to threat intelligence. Future work is suggested to automate the environment provisioning and specification generation steps.