Technical Summary: COHORT – Collaborative Orchestration for Hardening via Offensive Replay on Emulated Topologies

1. Problem Statement

Mitigating a specific adversary in an enterprise network is traditionally a slow, expert-dependent process. It requires an analyst to derive a tailored mitigation, validate it without disrupting production, and verify it disrupts the specific attack. This workflow is constrained by two primary bottlenecks:

1. Expert Dependency: The process relies heavily on human judgment to synthesize new controls rather than relying on curated playbooks.
2. Production Risk: Experimental adjustments to firewall rules, routing policies, or host configurations cannot be safely exercised on live production networks due to the risk of breaking legitimate operations.

Existing tools (IDS/IPS, SIEM, SOAR) typically execute curated responses for known patterns or validate detection coverage via Breach-and-Attack Simulation (BAS) against unmodified networks. They do not autonomously synthesize new, deployable mitigations or validate them by re-executing the observed adversary against the modified network.

2. Methodology: The COHORT Framework

COHORT is the first end-to-end framework designed to autonomously generate, deploy, and validate deployable mitigations against a specific observed adversary. It operates entirely within a high-fidelity emulated environment to avoid production exposure.

Core Architecture

The framework utilizes a role-decomposed multi-agent Large Language Model (LLM) workflow running on a GNS3 emulator that boots real vendor firmware (FortiGate, Cisco IOS, Open vSwitch). The adversary is emulated using Caldera executing MITRE ATT&CK-aligned actions.

The workflow consists of the following specialized agents:

• Mitigation Suggester: Analyzes the adversary's operational report and network topology to propose reusable, enterprise-level control strategies (e.g., segmentation, routing constraints).
• Mitigation Implementer: Translates the selected strategy into concrete device configuration commands within a bounded command budget.
• Critic: Reviews the implementation before replay to identify configuration errors, policy violations, or deviations from intent. This triggers an iterative implementation–critique loop (up to $K$ iterations) to refine the configuration before it is accepted or discarded.
• Judge: Evaluates the outcome by replaying the same adversary against the mitigated network. It compares the Attack Step Success Rate (ASSR) of the mitigated run against the unmitigated baseline.
• Summarizer: Produces a human-readable artifact of the process.

Validation Mechanisms

Validation relies on offensive replay rather than abstract simulation or expert proxies. A mitigation is considered successful only if it meets two criteria:

1. Attack Disruption: It strictly reduces the Attack Step Success Rate (Mitigation Effectiveness, $ME > 0$).
2. Connectivity Preservation: It passes two independent probes from the compromised host: a LAN ping to an internal target and an HTTP request to an internet endpoint.

Additionally, the framework supports Cumulative Evaluation, where approved mitigations are stacked onto a persistent network state to measure compound effects and prevent regression.

3. Key Contributions

• End-to-End Autonomous Framework: COHORT addresses both the expert-knowledge bottleneck and the production-risk bottleneck. It autonomously generates and validates mitigations using real device commands in a high-fidelity emulator, removing the need for human synthesis during the generation phase.
• Multi-Agent Decomposition: The paper demonstrates that decomposing the workflow into specialized roles (Suggester, Implementer, Critic) significantly outperforms a single-agent baseline.
• Iterative Critique Loop: The inclusion of a Critic agent that refines configurations before replay is shown to be a critical component of the system's success.
• Cumulative Mitigation Evaluation: The framework introduces a mechanism to stack defenses on a persistent state, revealing compound effects that per-mitigation evaluation misses.

4. Experimental Results

The framework was evaluated across three enterprise topologies (Small, Medium, Large) and four attack scenarios (Ransomware, Lateral Movement, DNS Exfiltration, Data Theft), using GPT-5.4 mini as the underlying model.

• Mitigation Success Rate (MSR): The multi-agent framework achieved an overall MSR of 46.7%. This is a 4.4× improvement over a single-agent baseline (10.7%) using the same model and tool access.
  • Decomposition of Improvement: Role specialization (Multi-Agent without Critic) accounted for approximately 75% of the gain (38.6% MSR), while the iterative critique loop accounted for the remaining 25%.
• Cumulative Effectiveness: When approved mitigations were stacked, the system reached a plateau of approximately 52% cumulative attack-step reduction by the fourth defense.
• Topology Independence: No statistically significant difference in MSR was detected across small, medium, and large topologies.
• Mitigation Categories:
  • Host-centric controls (e.g., Host Hardening, Application Control) dominated in both volume and effectiveness (e.g., Application Control achieved 82.1% MSR).
  • Network-based filtering was the second most common category by volume but had a low success rate (5.8%). Failures were primarily due to rules being placed at the wrong network hop, evaluated in the wrong order, or failing to account for hidden intermediaries, rather than incorrect intent.

5. Significance and Scope

The paper claims significance as the first framework to close the loop between adversary emulation, real-device configuration generation, and offensive replay validation. By confining experimentation to a high-fidelity emulator, it enables the safe derivation of deployable mitigations that would otherwise be too risky to test on production networks.

Limitations and Scope:
The authors explicitly state that the results do not generalize to:

• Adaptive adversaries: The framework assumes a scripted adversary that does not re-plan in response to defenses.
• Windows endpoints: Evaluation was restricted to Linux containers due to emulation resource constraints.
• Vendor heterogeneity: Testing was limited to FortiGate, Cisco IOS, and Open vSwitch.
• Production deployment: The framework outputs validated candidates, but the final deployment remains a separate, human-driven step.

The paper concludes that while COHORT successfully automates the derivation and validation of mitigations for scripted adversaries, the system requires human review before deployment to ensure safety, particularly given the potential for the framework to surface new vulnerabilities (e.g., container escape weaknesses) during the hardening process.