ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation

1. Problem Statement

Cyber threat investigation is a complex, multi-step process requiring security analysts to sift through heterogeneous security logs, correlate events across multiple data sources, and reason through multi-hop chains of evidence to identify threats. While Large Language Models (LLMs) have shown promise in complex reasoning tasks, there is a lack of standardized benchmarks to evaluate LLM agents specifically on cyber threat investigation. Existing benchmarks (e.g., CTIBench, SECURE) primarily focus on knowledge memorization, text extraction, or multiple-choice Q&A, failing to test an agent's ability to interact with a database, discover schemas, and perform sequential reasoning to solve real-world security incidents.

2. Methodology

The authors propose ExCyTIn-Bench, a comprehensive benchmark designed to simulate real-world threat investigation scenarios. The methodology involves three core components:

A. Data Collection & Environment Construction

• Source: Data was collected from a fictional Microsoft Azure tenant ("Alpine Ski House") configured with realistic security monitors.
• Scale: The environment includes 57 distinct log tables (e.g., SigninLogs, EmailEvents, SecurityAlert, SecurityIncident) containing over 14 million rows in the full 44-day log stream.
• Incidents: The dataset covers 8 distinct, multi-stage attack chains based on real-world historical attacks (e.g., Manatee Tempest Ransomware, BEC/Account Takeover). Each incident spans 2 hours to 5 days and involves various MITRE ATT&CK techniques.
• Interface: The benchmark provides a MySQL Docker environment where agents interact via SQL queries. Agents must discover the schema (tables and columns) through exploration, as the full schema is not provided in the prompt.

B. Question Generation via Bipartite Graphs

To generate high-quality, non-repetitive, and grounded questions, the authors developed a principled graph-based generation pipeline:

1. Graph Construction: For each incident, a bipartite graph $G=(U, V, E)$ is constructed where $U$ represents Alert vertices and $V$ represents Entity vertices (e.g., users, IPs, hosts). Edges connect alerts to the entities they involve.
2. Path Selection: The system selects two distinct alerts ($u_s$ and $u_e$) as start and end points. It identifies the entities connected to these alerts to form a path.
3. QA Generation: An LLM is prompted to generate a question using the start alert and its connected entities as context, and the end alert's entity as the answer.
4. Solution Path: The shortest path between the start and end nodes in the graph serves as the ground-truth solution path, allowing for fine-grained evaluation of intermediate steps.

• Output: This process generated 7,542 questions, with 589 selected for the test set to ensure diversity and prevent path overlap between train/test sets.

C. Evaluation Framework

• Reward System: The benchmark uses a decayed reward mechanism (inspired by Reinforcement Learning).
  • Full Reward (1.0): Awarded for the correct final answer.
  • Partial Reward: If the final answer is incorrect, the system checks if the agent found intermediate nodes (IoCs) along the solution path. Rewards are assigned based on the progress made, with a decay factor ($\gamma=0.4$) applied to steps further from the final answer.
• Judging: Evaluation is performed using an LLM-as-a-judge (GPT-4o) to handle variations in answer formatting and to verify if the agent's reasoning aligns with the ground truth, rather than relying solely on strict string matching.

3. Key Contributions

1. First Threat Investigation Benchmark: ExCyTIn-Bench is the first benchmark specifically designed to evaluate LLM agents on the end-to-end task of cybersecurity threat investigation using real-world log data.
2. Novel QA Generation Strategy: The paper introduces a method to generate deterministic, non-repetitive questions from bipartite alert-entity graphs, ensuring questions are grounded in explicit data paths and have verifiable solution trajectories.
3. Fine-Grained Process Evaluation: Unlike binary pass/fail metrics, the benchmark provides step-wise rewards, enabling the evaluation of an agent's ability to extract intermediate Indicators of Compromise (IoCs) and its progress in complex search spaces.
4. Comprehensive Empirical Analysis: The authors provide a detailed analysis of model performance, behavior (schema discovery, query success), and efficiency (cost vs. reward) across a wide range of models.

4. Experimental Results

The authors evaluated numerous proprietary and open-source models (including GPT-4o, Claude-Opus-4.5, Llama-4, Qwen-3, and reasoning models like o1/o3) using various prompting strategies (ReAct, Expel, Best-of-N, Self-Reflection).

• Performance Ceiling: The task is highly challenging. The best-performing model, Claude-Opus-4.5, achieved a maximum average reward of 0.606, leaving significant room for improvement.
• Model Comparison:
  • Proprietary vs. Open Source: Top proprietary models (Claude-Opus-4.5, GPT-5.1) outperform open-source models, but large open-source models (e.g., Llama-4-Maverick-17B) are rapidly closing the gap, achieving competitive rewards (~0.29).
  • Reasoning Models: Models with explicit reasoning capabilities (e.g., o1, o3) show strong performance, particularly when combined with reflection strategies.
• Method Impact:
  • ReAct + Reflection: This combination yielded the highest performance gains across most models, significantly outperforming the base prompt.
  • Expel: The Expel method (distilling rules from training data) showed high efficiency (fewer turns) but higher cost due to prompt inflation.
• Efficiency: There is a strong positive correlation (Pearson $r=0.86$) between query success rates and final rewards. Smaller models often struggle with schema discovery, leading to more errors and lower rewards.
• Ablation Studies:
  • DB Scope: Removing alert logs (simulating zero-day attacks) caused a significant drop in performance, highlighting the critical role of detection alerts in investigation.
  • Time Window: Expanding the time window to a full 44-day history introduced noise, slightly reducing performance compared to per-incident slices.

5. Significance and Future Directions

• Advancing Autonomous Security: ExCyTIn-Bench provides a critical testbed for developing autonomous agents capable of reducing the burden on human SOC analysts by automating multi-step threat hunting.
• RL for Cybersecurity: The fine-grained reward structure makes the benchmark uniquely suitable for training agents via Reinforcement Learning (RL), allowing for precise credit assignment across investigation steps.
• Realistic Evaluation: By using real-world attack patterns and requiring schema discovery, the benchmark moves beyond "knowledge retrieval" to test genuine investigative reasoning and tool use.
• Open Source: The authors have open-sourced the benchmark, environment, and code (SecRL), fostering reproducibility and accelerating research in AI-driven cybersecurity defense.

In conclusion, ExCyTIn-Bench establishes a new standard for evaluating LLM agents in cybersecurity, demonstrating that while current models show promise, significant advancements in reasoning, schema understanding, and multi-hop planning are required to achieve fully autonomous threat investigation.