Security researchers have long hoped that large language model agents could speed up one of the most tedious parts of incident response: piecing together how an attacker moved through a system, step by step, using scattered logs and telemetry. A new diagnostic benchmark called DiagChain puts that hope to the test, and the results suggest the technology still has a long way to go before it can reliably handle attack chain reconstruction on its own.
Developed by a team of researchers including Xuyang Liu, Yibin Han, Zhenwei Zhang, Kai Chang, Zhiwei Xu, Tian Qiu, Weixian Deng, Jiabao Gao, Xiaolin Peng, Hai Wan, and Xibin Zhao, DiagChain isn’t just another accuracy scoreboard. It’s built specifically to show where and why LLM agents fail when they try to reconstruct the ordered sequence of actions an attacker took, based on evidence pulled from system telemetry.
Summary
Key takeaways
- DiagChain is a diagnostic benchmark that evaluates LLM agents on evidence-grounded attack chain reconstruction, going beyond simple pass/fail accuracy.
- The benchmark’s MAIN-69 suite includes 69 scenarios spanning multiple operating systems, evidence noise levels, and chain lengths.
- A new method called ECRAG pairs evidence retrieval with an evolving structured representation of the chain being reconstructed.
- Across 6 different LLMs tested, the best configuration succeeded on just 39.6% of 849 reference steps.
- Smaller models struggle to use retrieved evidence at all, while larger models struggle mainly with putting that evidence in the correct order.
Introducing DiagChain: A New Benchmark for Attack Chain Reconstruction
DiagChain exists because most existing benchmarks only look at final outputs or overall accuracy scores, offering little insight into how mistakes actually form during an agent’s reasoning process. That’s a meaningful gap for cybersecurity teams trying to decide whether an AI agent is trustworthy enough to help triage a real intrusion.
Purpose and Scope of DiagChain
At its core, DiagChain is a diagnostic benchmark built for evidence-grounded cybersecurity tasks. Rather than simply grading whether an agent got the final attack narrative right or wrong, it evaluates each stage of the reconstruction process separately. This stage-wise approach lets researchers pinpoint exactly where an LLM agent’s reasoning breaks down, whether that’s during evidence gathering, evidence interpretation, or the ordering of events into a coherent chain.
Composition of MAIN-69 Scenario Suite
The benchmark’s centerpiece is MAIN-69, a suite of 69 scenarios designed to stress-test agents across a range of realistic conditions. These scenarios span multiple operating systems, vary in how much noise is mixed into the evidence, and differ in chain length, meaning some attack sequences are short while others require tracking a longer string of attacker actions. That variety is meant to expose whether an agent’s performance holds up when conditions get messier, not just when everything is clean and straightforward.
Methodological Innovations and Evaluation Metrics
Beyond the scenario suite, DiagChain introduces its own retrieval method and a five-part scoring system, both aimed at making failure diagnosis systematic rather than guesswork.
Evidence-Centric Retrieval-Augmented Generation (ECRAG)
One of the paper’s key contributions is ECRAG, short for Evidence-Centric Retrieval-Augmented Generation. Unlike standard retrieval-augmented generation setups, ECRAG couples the retrieval of evidence with an evolving structured representation of the chain the agent is trying to reconstruct. In practical terms, this means the system doesn’t just pull relevant evidence once and move on; it continuously updates its internal picture of the attack chain as new evidence comes in, which should, in theory, help agents keep track of complex, multi-step intrusions more coherently.
Five Complementary Metrics for Diagnostic Assessment
To make sense of where things go wrong, DiagChain relies on five complementary metrics, each targeting a distinct stage of the reconstruction process. Together, these metrics allow researchers to isolate specific failure points rather than lumping every mistake into one generic accuracy number. This is central to the benchmark’s value: a diagnostic assessment that separates “the agent didn’t find the right evidence” from “the agent found the evidence but put events in the wrong order” is far more useful for improving these systems than a single pass/fail score.
Performance Assessment of Large Language Models
So how did today’s models actually perform? Not especially well, according to the benchmark’s results.
Experimental Setup Using Six LLMs
The research team ran evaluations using six different LLMs against the MAIN-69 scenarios. This setup let them compare how models of varying capability handled the same evidence-grounded reconstruction tasks, under the same noise conditions and chain-length challenges, giving a consistent basis for comparing performance across the board.
Key Performance Results and Success Rates
The headline number is stark: even the best-performing configuration in the study succeeded on only 39.6% of the 849 reference steps included in MAIN-69. In other words, the strongest setup tested still got the sequence of attacker actions wrong more than 60% of the time when measured against the benchmark’s ground-truth steps. That’s a meaningful reality check for anyone hoping to hand attack chain reconstruction entirely over to an AI agent right now.
Insights on Model Size and Reconstruction Challenges
Why does this matter beyond the raw numbers? Because the failure patterns differ depending on model size, and that distinction points to two very different engineering problems that need solving.
Limitations of Smaller Models
According to the researchers’ analysis, smaller models struggle with something more basic than ordering events correctly: they have trouble incorporating retrieved evidence into their outputs at all. That suggests the bottleneck for smaller models sits earlier in the pipeline, at the point where evidence needs to actually inform the agent’s reasoning rather than being ignored or misapplied.
Challenges for Larger Models in Evidence Ordering
Larger models clear that first hurdle more successfully and are able to proceed further into the reconstruction process. But they run into a different wall: correctly ordering the evidence they’ve gathered becomes the main bottleneck. This is a subtler failure mode, since the model has the right pieces but struggles to arrange them into the accurate sequence of attacker actions, which is precisely the output that matters most for a real security investigation.
This split matters for anyone building or deploying LLM evaluation benchmark tools in security operations. It implies that scaling up model size alone won’t automatically solve attack chain reconstruction. The two failure modes call for different fixes, better evidence integration for smaller models and better sequencing or reasoning strategies for larger ones, rather than a one-size-fits-all improvement path.
The researchers frame these findings as validation for diagnostic evaluation over simple end-to-end accuracy scores. A single aggregate number might tell a security team that a model “gets it right 40% of the time,” but it wouldn’t tell them whether that failure comes from missed evidence, misread evidence, or scrambled sequencing. DiagChain’s stage-wise metrics are designed to close that gap, offering what the authors describe as actionable insights for improving evidence-grounded cybersecurity agents going forward.
FAQ
What is DiagChain designed to evaluate?
DiagChain is designed to evaluate large language model agents on evidence-grounded attack chain reconstruction through diagnostic, stage-wise assessment.
What content does the MAIN-69 scenario suite cover?
MAIN-69 includes 69 scenarios covering multiple operating systems, varying evidence noise levels, and different attack chain lengths.
How does the ECRAG methodology improve attack chain reconstruction?
ECRAG couples evidence retrieval with an evolving structured representation of the reconstructed chain to assist in chaining analysis.
What are the main performance challenges identified for LLMs in this benchmark?
Smaller models struggle to incorporate retrieved evidence, while larger models face challenges in correctly ordering evidence in reconstruction.
Article produced with the assistance of artificial intelligence and reviewed by the editorial team.
