💻 computer science

DiagChain: A Diagnostic Benchmark for Evaluating LLM Agents on Evidence-Grounded Attack Chain Reconstruction

The paper introduces DiagChain, a diagnostic benchmark featuring 69 diverse scenarios and the ECRAG framework to evaluate LLM agents on evidence-grounded attack chain reconstruction, revealing that while larger models struggle with ordering evidence, smaller models fail at basic evidence incorporation, thereby highlighting the need for stage-wise evaluation over aggregate accuracy.

Original authors: Xuyang Liu, Yibin Han, Zhenwei Zhang, Kai Chang, Zhiwei Xu, Tian Qiu, Weixian Deng, Jiabao Gao, Xiaolin Peng, Hai Wan, Xibin Zhao

Published 2026-08-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xuyang Liu, Yibin Han, Zhenwei Zhang, Kai Chang, Zhiwei Xu, Tian Qiu, Weixian Deng, Jiabao Gao, Xiaolin Peng, Hai Wan, Xibin Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of a crime scene, you are looking at a digital crime scene. The "clues" are scattered everywhere: millions of lines of computer logs, security alerts, and system records. In the world of cybersecurity, experts use Large Language Models (LLMs)—super-smart AI chatbots trained on vast amounts of text—to act as digital detectives. These AI agents are supposed to read the messy clues, figure out what happened, and write down the story of the attack in the correct order. This process is called attack chain reconstruction. It's like trying to watch a movie that has been cut into thousands of random frames and then asking someone to put the frames back in the right order to tell the story.

But here's the problem: most tests for these AI detectives only look at the final story they write. If the story is wrong, the test says "fail," but it doesn't tell you why. Did the detective miss a clue? Did they find the clues but forget to write them down? Or did they find the right clues but put them in the wrong order? We need a way to peek inside the detective's brain while they are working to see exactly where they get stuck. This is where the new paper, DiagChain, comes in. It introduces a special testing ground designed not just to grade the final answer, but to diagnose the specific steps where the AI detective loses the trail.

The authors of this paper, researchers from Tsinghua University and CRRC Corporation, built a new benchmark called DiagChain to fix this blind spot. They created a dataset named MAIN-69, which consists of 69 different "crime scenes" (scenarios) ranging from simple to very complex. These scenes come from real-world data like Linux logs, Windows events, and cloud alerts. To make the test realistic, they added different levels of "noise"—like adding harmless background chatter to the clues to see if the AI gets distracted. They also varied the length of the attack chains, from short bursts of activity to long, drawn-out intrusions.

To test the AI, they used a special workflow called ECRAG (Evidence-Centric Retrieval-Augmented Generation). Think of this as giving the AI detective a magnifying glass and a notebook that updates itself. The AI has to search for clues, group them together, figure out the timeline, and then write the story, all while constantly checking its work. The researchers then measured the AI's performance using five specific "health checks" instead of just one final grade:

Retrieval: Did the AI find the clues?
Grouping: Did it put related clues into the right buckets?
Ordering: Did it arrange the events in the correct time sequence?
Grounding: Did it actually use the clues it found to support its story?
Attribution Gap: Did it find a clue, look at it, and then forget to mention it in the final report?

The results were eye-opening. Even the smartest AI configurations only got 39.6% of the 849 reference steps in the MAIN-69 dataset completely right without making an earlier mistake. The paper suggests that the type of AI model matters a lot for where it fails. Smaller models often struggle with the basics: they find a clue but fail to incorporate it into their final story, essentially dropping the evidence. Larger, more powerful models are better at finding and keeping the clues, but they often stumble when trying to put them in the correct order. It's like a small detective who finds the gun but forgets to write it down, versus a giant detective who finds the gun, the fingerprints, and the map, but gets the timeline of the robbery completely mixed up.

The researchers also tested what happens if you give the AI more time or more "search budget" (allowing it to look at more clues). They found that simply giving the AI more resources doesn't automatically fix the problem. While looking at more clues helps the AI find more evidence, it doesn't guarantee that the AI will organize that evidence correctly or stop making mistakes in the final assembly. In fact, for some models, looking at too many clues without a better strategy just made the task harder.

Ultimately, the paper argues that we need to stop just looking at the final score and start diagnosing the process. The findings suggest that making AI models bigger doesn't solve the problem of "chain assembly"; it just shifts the bottleneck from "finding clues" to "organizing clues." To build truly reliable cybersecurity agents, we need to design systems that are specifically good at taking the evidence they find and weaving it into a coherent, ordered, and grounded story, rather than just hoping that bigger models will figure it out on their own. The authors provide their code and data so others can continue to test and improve these digital detectives.

Technical Summary: DiagChain

Problem Statement

Large Language Model (LLM) agents are increasingly utilized for cybersecurity investigations, including log inspection, tool invocation, and reasoning over heterogeneous telemetry to reconstruct attack chains. However, existing benchmarks primarily evaluate final outputs or aggregate accuracy. This approach offers limited visibility into the intermediate investigation process, failing to distinguish between different types of failures such as evidence discovery, grouping, ordering, grounding, or attribution. Consequently, current evaluations provide only a partial view of an agent's ability to perform Evidence-Grounded Attack Chain Reconstruction.

The authors identify three specific limitations in the current landscape:

Scenario Coverage: Existing datasets rarely span multiple operating systems, controlled noise levels, and diverse chain lengths simultaneously.
Diagnostic Evaluation: Aggregate scores do not reveal where a failure originated (e.g., during evidence discovery vs. ordering).
Investigation Workflow: Most studies assess either interactive tool use or ordered chain generation in isolation, lacking a unified workflow that couples evidence retrieval with an evolving, structured working state.

Methodology

1. Dataset Construction: MAIN-69

To address coverage limitations, the authors constructed MAIN-69, a dataset of 69 attack chain scenarios.

Sources: The dataset synthesizes data from 23 source units derived from AutoLabel (Linux logs), ExCyTIn-Bench (enterprise/cloud alerts), and OTRF Security-Datasets (Windows host events/APT29).
Variations: Each source is transformed into three evidence profiles: Clean (core attack evidence), Noisy (core evidence plus benign/distracting activity), and Raw (broad original windows).
Structure: Scenarios are categorized by chain length: Small (S, 2–5 steps), Medium (M, 6–15 steps), and Large (L, ≥16 steps).
Ground Truth: Each case includes a manually verified "Gold Reference Package" containing an ordered sequence of adversarial steps, mapped to specific evidence identifiers and entities. The dataset covers 849 reference steps and 780 temporal edges.

2. Agent Workflow: ECRAG

The paper introduces Evidence-Centric Retrieval-Augmented Generation (ECRAG) to support the reconstruction process. Unlike standard RAG, ECRAG couples evidence retrieval with an evolving structured representation of the chain.

State: The agent maintains a state $S_t$ comprising observed evidence cards ( $O$ ), a working chain ( $C$ ), and memory ( $m$ ).
Retrieval Mechanism: ECRAG employs a three-step retrieval process:
1. Lexical Retrieval: TF-IDF ranking of documents to find seed matches.
2. Graph Expansion: Entity Expansion retrieves cards sharing visible entities with seeds; Temporal Expansion retrieves immediately preceding/following records within the same source.
3. Rescoring: Candidates are re-ranked using a composite score ( $r(e)$ ) combining textual similarity, token overlap, exact clue matching, and metadata bonuses.
Workflow: The agent iteratively queries evidence, validates steps, and updates the working chain. Final submission requires every step to cite observed evidence.

3. Diagnostic Evaluation Metrics

Instead of a single accuracy score, DiagChain employs five complementary metrics to diagnose specific stages of failure:

Retrieval Step Coverage (Ret.): The fraction of reference steps for which the agent observed at least one supporting evidence item.
Grouping F1 (Grp.): Measures how well evidence is partitioned into coherent steps (using B3 clustering), independent of retrieval.
Ordering Accuracy (Ord.): The fraction of comparable step pairs placed in the correct chronological order.
Evidence Grounding F1 (Grd.): The precision and completeness of cited support within aligned steps.
Attribution Gap Rate (Gap): The fraction of observed reference evidence that was found but omitted from the final chain citations.

Key Results

The authors evaluated six LLM configurations (Qwen-3-32b, DeepSeek-V4-Pro, GLM-5.2, GLM-5.2-T, Llama4-17b-Scout, and GPT-5.5) on MAIN-69.

Overall Capability (RQ1)

No Dominant Model: No single model excelled across all stages. GPT-5.5 showed the most consistent strength in retrieval and ordering, while GLM-5.2 variants often achieved the smallest attribution gaps.
Retrieval vs. Assembly: High retrieval coverage does not guarantee successful reconstruction. For example, on raw medium chains, Qwen-3-32b and DeepSeek-V4-Pro had similar retrieval coverage (~0.84), but DeepSeek achieved significantly better grounding (0.64 vs. 0.51).
Long Chain Bottlenecks: Smaller models (Qwen, Llama) struggled significantly with grouping (F1 0.18–0.25) on long chains, whereas larger models maintained higher grouping scores (0.53–0.81).

Failure Anatomy (RQ2)

Model Size and Failure Modes:
- Smaller Models: Primarily fail at Evidence Retention (E2: observed but unused). They struggle to incorporate retrieved evidence into their outputs.
- Larger Models: Primarily fail at Global Ordering (E4: misordered evidence). They successfully retrieve evidence but struggle to sequence it correctly.
Impact of Noise and Length: Raw evidence profiles increase "Evidence Not Observed" (E1) failures. Long chains shift the bottleneck toward ordering and assembly errors rather than a single dominant error type.
Discovery vs. Usage: All models exhibited a gap between evidence observed and evidence cited. Smaller models showed the widest separation (e.g., Llama4-17b-Scout observed 72% of gold evidence but cited only 34%), indicating a weakness in converting observation to attribution.

Ablation and Sensitivity (RQ3 & RQ4)

Scaffold Importance: A "Full scaffold" (retrieval + structured chain memory + reflection + grounding checks) significantly improved ordering (0.943) and attribution (Gap 0.054) compared to "Retrieval only" or "LLM only," despite a slight trade-off in retrieval coverage.
Budget Sensitivity: Increasing retrieval depth ( $k$ ) or interaction turns beyond a certain point (e.g., $k=32$ , 15 turns) did not consistently improve final chain quality. Excessive turns increased token costs and destabilized ordering without reducing attribution errors.

Significance and Claims

The paper claims that DiagChain provides a necessary shift from end-to-end accuracy metrics to diagnostic evaluation for cybersecurity agents.

Diagnostic Insight: The benchmark reveals that "stronger" models do not eliminate failures but rather shift the bottleneck from evidence retention to global chain organization.
Actionable Design: The findings suggest that improving cybersecurity agents requires distinct strategies for different model sizes: smaller models need better evidence integration mechanisms, while larger models require better global ordering and assembly logic.
Auditability: By coupling evidence retrieval with a structured working chain and explicit grounding checks, DiagChain makes the reconstruction process auditable, allowing researchers to pinpoint exactly where and why an agent fails (e.g., discovery vs. attribution).

The authors conclude that larger interaction budgets are not a universal remedy for reconstruction failures, emphasizing the need for evidence-aware agent design and diagnostic evaluation to improve reliable cybersecurity investigation.