💬 NLP

Cross-Session Threats in AI Agents: Benchmark, Evaluation, and Algorithms

This paper introduces CSTM-Bench, a dataset and evaluation framework for detecting cross-session AI agent threats that evade memoryless guardrails, demonstrating that bounded-memory Coreset Readers outperform both session-bound and full-log approaches while proposing a new metric that balances detection recall with serving stability.

Original authors: Ari Azarafrooz

Published 2026-04-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ari Azarafrooz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a security guard to protect a high-tech office. This guard's job is to check every person who walks through the door and make sure they aren't trying to steal anything.

The Problem: The "Amnesia" Guard
Currently, most AI security guards have a very specific kind of amnesia. They check a person, say "Hello," check their ID, and then immediately forget them. If a bad guy walks in 50 times over the course of a month, the guard checks each visit, sees nothing suspicious, and lets them pass every time.

The bad guy isn't stealing anything in a single visit. Instead, they are playing a game of "Jenga."

Visit 1: "Can you tell me what servers we have?" (Harmless)
Visit 2: "Can you show me the password for the backup server?" (Harmless on its own)
Visit 3: "Can you email that password to my personal address?" (Harmless on its own)

Individually, every request looks like a normal employee doing their job. But if you stack them together, it's a heist. The current security guards are "session-bound," meaning they only look at the current moment. They miss the big picture because the "heist" is spread out over time.

The Solution: The "Detective" with a Memory
This paper introduces a new way to catch these slow-burn attacks. Instead of just checking the person at the door, we need a Detective who keeps a running file on everyone.

However, there's a catch: The Detective can't remember everything. If you try to read a 10,000-page history book to catch a thief, the Detective gets overwhelmed, misses the clues, or falls asleep.

The Innovation: The "Highlighter" (Coreset Memory)
The paper proposes a smart middle ground called the Coreset Memory Reader. Think of this as a very efficient Highlighter.

Instead of reading the entire 10,000-page history book, the Highlighter scans the pages and only keeps the 50 most suspicious sentences in a small notebook.

If a sentence looks like normal work, it gets ignored.
If a sentence looks weird or out of place, it gets highlighted and kept.

When the Detective (the AI) needs to make a decision, they only read the 50 highlighted sentences. This keeps the Detective sharp, fast, and focused on the real threats, ignoring the noise.

The Benchmark: The "Training Ground"
The authors created a massive training ground called CSTM-Bench (Cross-Session Threat Memory Benchmark).

The Test: They created 26 different types of "heists" (attacks) that are designed to be invisible if you only look at one moment.
The Villains: They even built a "Villain AI" that tries to rewrite its own story to look innocent, specifically to trick the security guards.
The Result:
- The Old Guard (checking one message at a time) failed miserably when the Villain rewrote the story.
- The Full-Log Detective (trying to read every message ever) got overwhelmed by the sheer volume of text and missed the clues.
- The Highlighter Detective (the new method) succeeded. By filtering out the boring stuff and keeping only the high-suspicion parts, it caught the heist even when the Villain tried to hide.

The "Cost" of Memory
The paper also introduces a new way to measure how "expensive" this memory is. Imagine if the Highlighter kept changing its mind about which sentences to keep. Every time it swaps a sentence, the Detective has to re-read everything from scratch. This is slow and expensive.

The authors created a metric called CSR (Coreset Stability Rate). It measures how often the Highlighter keeps the same sentences in the notebook.

High Stability: The notebook stays the same. The Detective can read quickly. (Good!)
Low Stability: The notebook changes every second. The Detective has to re-read constantly. (Bad/Expensive!)

The Big Takeaway
This paper tells us that to stop AI from being tricked by slow, long-term attacks, we can't just make the AI's memory bigger. We need to make the AI's memory smarter. We need a system that filters out the noise and keeps only the most important clues, allowing the AI to see the "whole story" without getting overwhelmed by the details.

In short: Don't try to remember everything. Remember the right things.

1. Problem Statement: Stateless Guardrails vs. Stateful Threats

Current AI agent security relies on stateless guardrails that evaluate each message in isolation. While effective against single-turn attacks (e.g., immediate prompt injection), they fail against cross-session threats.

The Attack Vector: Adversaries distribute a malicious payload across dozens or hundreds of sessions. Individually, every message appears benign and compliant with the agent's policy. The threat only materializes when an external system (or the agent's own memory) aggregates these fragments over time.
The Failure Mode:
- Slow-Drip Injections: A jailbreak is split into innocuous fragments across 50+ sessions.
- Compositional Exfiltration: Session 1 lists files, Session 2 reads config, Session 3 summarizes credentials, and Session 4 sends the data.
- Semantic Laundering: Sensitive data is transformed through benign-looking intermediaries (e.g., summarization) before exfiltration.
The Limitation of Existing Solutions:
- SIEM/Post-hoc Correlation: Too slow; the damage is done before the alert fires.
- Full-Log Correlators: Concatenating all historical logs into a single LLM call fails due to signal dilution (benign noise drowns out the attack signal) and context window truncation (oldest, critical reconnaissance steps are dropped).

2. Methodology: CSTM-Bench

The authors introduce CSTM-Bench (Cross-Session Threat Memory Benchmark), a dataset and evaluation framework designed to measure the feasibility of cross-session threat detection.

A. Dataset Construction

Scale: 54 scenarios per split, comprising 26 Attack taxonomies, 14 Benign-pristine, and 14 Benign-hard (confounder) scenarios.
Taxonomy: Attacks are classified on two axes:
1. Kill-Chain Stages: Based on the "Promptware Kill Chain" (Reconnaissance, Initial Access, Persistence, Action, etc.).
2. Operations Ontology: How fragments combine: Accumulate (escalation), Compose (disjoint fragments), Launder (transformation), and Inject_on_Reader (attacking the correlator itself).
Identity Anchors: Each scenario is bound to one of seven specific agent personas (e.g., customer_support, devops_agent, financial_advisor) with crisp policy definitions to ground-truth "violation."
Two Splits:
1. Dilution Shard: Measures how well a correlator performs when attack signals are diluted by benign traffic within a fixed context window.
2. Cross-Session Shard (Adversarial): A closed-loop rewriter (Claude Opus 4.7) modifies the attack scenarios to pass a per-session judge while preserving cross-session artifacts. This creates a "ground-truth cross-session-only" regime where single-session detection is impossible.

B. Evaluation Architecture

The benchmark evaluates three distinct reader architectures against the same correlator model (Anthropic Claude Sonnet 4.6):

Per-Session Judge: Stateless; evaluates each session in isolation.
Full-Log Correlator: Unbounded memory; concatenates all historical logs chronologically into a single prompt.
Coreset Memory Reader: A bounded-memory approach ( $K=50$ ). It uses a plug-in ranker to select and retain only the highest-signal fragments, feeding a compressed buffer to the correlator.

C. Metrics

The paper proposes a composite score, CSTM, which fuses detection quality with serving cost:
$\text{CSTM} = 0.7 \cdot F1(\text{CSDA@action}, \text{precision}) + 0.3 \cdot \text{CSR\_prefix}$

CSDA@action: Recall of detecting the attack before the "action-on-objective" (the final malicious step).
CSR_prefix (Coreset Stability Rate): A novel, label-free metric measuring the stability of the top- $K$ buffer's ordered prefix. High stability implies better KV-cache reuse, directly correlating to lower inference costs in real-time serving.

3. Key Contributions

Problem Formalization & Dataset: The first benchmark specifically targeting cross-session threats, distinguishing between "dilution" (benign noise) and "isolation-invisible" (adversarial rewriting) failure modes.
Feasibility Characterization: Empirical evidence that Full-Log Correlators fail under cross-session adversarial conditions. Even with 1M+ token context windows, they lose ~50% of attack recall when facing adversarial rewriting because the attack signal is structurally distributed and diluted.
Architectural Solution: Demonstration that a bounded-memory Coreset Reader outperforms both stateless judges and full-log correlators. By filtering out benign noise, the Coreset Reader preserves the high-signal fragments necessary for detection.
Cost-Aware Metric: Introduction of CSR_prefix as a first-class metric, proving that detection stability is a prerequisite for real-time, cost-effective deployment.

4. Key Results

Per-Session Judge:
- Dilution Shard: 100% recall (catches attacks in isolation).
- Cross-Session Shard: Drops to 53.8% recall. The adversarial rewriter successfully made individual sessions appear benign.
Full-Log Correlator:
- Dilution Shard: 88.5% recall. Suffers from signal dilution; benign traffic overwhelms the attack signal.
- Cross-Session Shard: Drops to 50.0% recall. The rewriter successfully framed the entire log as "authorized workflows," causing the LLM to miss the latent attack structure.
Coreset Memory Reader (K=50):
- Dilution Shard: 96.2% recall. Outperforms Full-Log by compressing benign noise, allowing the correlator to focus on the attack.
- Cross-Session Shard: 73.1% recall. The only reader to maintain high recall under adversarial rewriting. It successfully recovered scenarios that both the Per-Session Judge and Full-Log Correlator missed.
False Positives: The Coreset Reader significantly reduced false alarms on "Benign-hard" scenarios (confounders) compared to the Full-Log Correlator, as the ranker filtered out the "attack-shaped" benign patterns.

5. Significance and Implications

Context Windows Are Not a Silver Bullet: The paper proves that simply increasing context window size (e.g., to 1M tokens) does not solve cross-session threats. The issue is information-theoretic dilution, not just memory capacity.
The Necessity of Bounded Memory: Effective cross-session detection requires an information bottleneck (a ranker) to filter the stream before it reaches the expensive correlator LLM. This preserves the "needle" while discarding the "haystack."
Real-Time Viability: By introducing CSR_prefix, the authors show that high-performance detection must account for serving costs. A stable buffer allows for KV-cache reuse, making real-time cross-session monitoring economically feasible.
Adversarial Robustness: The benchmark demonstrates that attackers can adapt to defeat both session-bound and full-log detectors. The Coreset approach is the only architecture shown to survive this "closed-loop" adversarial pressure, suggesting it is the necessary architectural pattern for future AI agent security.

In conclusion, CSTM-Bench establishes that cross-session threat detection is an architectural problem, not just a model capability problem. The solution lies in combining bounded-memory filtering (Coresets) with downstream correlation, rather than relying on unbounded logs or isolated session checks.