🤖 AI

Magnet: Detecting Cross-Session AI Misuse Through Capability Accumulation

This paper introduces Magnet, a detection framework that addresses the critical gap in identifying cross-session AI misuse by aggregating harmless-looking artifacts from multiple isolated agent interactions to reconstruct harmful goals that evade traditional single-session monitoring.

Original authors: Natalie Isak, Matthew Dressman

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Natalie Isak, Matthew Dressman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where artificial intelligence isn't just a single, chatty robot, but a fleet of specialized helpers. Think of them like a construction crew: one lays the bricks, another mixes the cement, and a third paints the walls. They work together to build something amazing. But what happens if a sneaky troublemaker tries to build something dangerous, like a trap, using this crew? The troublemaker knows that if they ask the whole crew to build a trap at once, the foreman (the safety system) will say, "No way!" and stop them immediately. So, the troublemaker gets clever. They break the big, bad plan into tiny, boring, and totally harmless-looking tasks. They ask the brick-layer to just "find a red brick," then ask the painter to "find a bucket of blue paint," and later ask the cement-mixer to "find some glue." Each request looks innocent on its own. The problem is that the AI crew forgets everything between tasks—they have no memory of the previous day's work. But the troublemaker does remember. They collect all these tiny, harmless pieces and put them together later to build the trap. This paper explores how this "divide and conquer" trick works and proposes a new way to catch the troublemaker before the trap is finished.

The researchers, Natalie Isak and Matthew Dressman, call their new detection tool Magnet. They discovered that current safety systems are like security guards who only look at one person at a time. If a person walks in asking for a hammer, then later asks for nails, and later still asks for wood, the guard sees three separate, harmless requests. The guard doesn't realize that if you put those three things together, you can build a weapon. The paper shows that by splitting a harmful goal into many small, isolated sessions, attackers can successfully bypass these guards. In their tests, this "cross-session" trick worked much better than trying to do everything in one go. For example, when trying to create a phishing kit (a tool to steal passwords) or instructions for making a Molotov cocktail, the success rate jumped from about 18% with a single request to nearly 37% when the request was split across many separate sessions.

The paper argues that we need to stop looking just at what someone says (their intent) and start tracking what they are actually building (their capability). Intent is easy to fake; you can say, "I'm writing a story about a bad guy," to hide your real plan. But the actual pieces you gather—a login form, a script to send emails, a chemical formula—are harder to hide. Magnet works like a magnet in a field of hay. Instead of checking every single piece of straw (every conversation) to see if it's dangerous, Magnet pulls out only the "needles" (the specific, dangerous capabilities) from all the sessions a user has had. It keeps a running list of every dangerous piece a user has collected. If a user gathers enough pieces to complete a dangerous set—say, they have the form, the email script, and the hosting instructions—Magnet sounds the alarm, even if none of those pieces looked dangerous on their own.

The researchers tested this idea against other methods. They found that simply compressing past conversations into a short summary (like a "highlight reel") often fails because the bad stuff gets lost in the noise of all the harmless stuff. Magnet, however, is much better at spotting the pattern. In their simulations, Magnet caught 75% of the attacks while keeping false alarms low, whereas the old "per-session" guards only caught about 45%. The paper suggests that as AI agents become more common and work across many different conversations, safety systems need to change. Instead of just watching the conversation window, they need to watch the user's long-term "inventory" of capabilities. While this study focused on specific dangers like phishing and weapons, the authors suggest this approach could also help spot other slow-building problems, like someone slowly being manipulated into harmful behavior over weeks or months. The key takeaway is that danger often hides in the sum of many small, innocent parts, and we need a new kind of detector to see the whole picture.

Technical Summary: Magnet: Detecting Cross-Session AI Misuse Through Capability Accumulation

Problem Statement

Current AI misuse detection frameworks are primarily designed around single-turn or multi-turn (single-session) threat models. These systems assume that harmful intent or artifacts will be visible within a single conversation context. However, modern AI deployments increasingly utilize ensembles of specialized agents that operate across multiple, isolated sessions.

This architecture creates a critical vulnerability: an attacker can decompose a harmful objective ( $T$ ) into a series of individually benign sub-capabilities ( $c_1, \dots, c_n$ ). While the AI agent is stateless between conversations (its context window is refreshed), the attacker is not. By distributing these subtasks across independent sessions, the attacker can gradually accrue the components necessary to construct a harmful artifact (e.g., a phishing kit or a weapon assembly guide) without ever triggering safety filters in any single session. Existing detection methods, which often rely on compressing conversation history or evaluating intent per-session, fail to identify these "low-and-slow" attacks because the incriminating evidence is scattered across time and sessions, appearing harmless in isolation.

Methodology

1. Attack Simulation and Data Generation

The authors constructed a pipeline to simulate cross-session goal decomposition, referred to as "MOSAIC Prompting."

Decomposition: Expert authors decomposed two distinct harmful objectives—phishing-kit generation and Molotov cocktail assembly—into a set of sub-capabilities.
Execution: A minimal multi-tool agent (equipped with tools like web search, email sending, and file saving) was used to dispatch these subtasks. Each subtask was handled by a freshly instantiated assistant–attacker pair, ensuring no conversation history or intent framing was shared between steps.
Models: The pipeline was tested across nine different LLM models (including Claude, GPT, and Gemini families) via GitHub Copilot.
Delivery Modes: The study compared three delivery strategies:
1. Single-turn: All sub-capabilities in one prompt.
2. Multi-turn (Intra-session): Sub-capabilities sequentially within one conversation.
3. Multi-session (Inter-session): Each sub-capability in a new, isolated agent session.

2. Evaluation and Labeling

Success Criteria: Success was defined not as a binary pass/fail of a single session, but by the ability to recombine artifacts from multiple sessions to satisfy specific task criteria (e.g., a functional phishing page).
Scoring: An additive rubric was used to measure the fraction of the end-to-end capability accrued. A 50% threshold was used to define a successful attack.
Failure Analysis: Failures were categorized as either safety refusals (guardrails triggered) or context degradation (pipeline failure due to lack of context), distinguishing between safety blocks and capability limitations.

3. Detection Approaches

The paper evaluates three detection architectures against a dataset of 336 scenarios (mixing real conversational data and attack attempts):

Per-Session Detector: Evaluates each session independently for malicious content.
Compression Detector: Maintains a compressed history (top 20 most suspicious messages based on vector similarity to "capability anchors") and evaluates this subset holistically.
Magnet (Proposed): A detection primitive that tracks demonstrated capability rather than stated intent.
- Mechanism: For each session, an LLM extracts a binary vector over a predefined, expert-authored capability taxonomy (e.g., "credential capture form," "email-sending code").
- Aggregation: These vectors are aggregated at the entity level (User ID) using a bitwise OR operation. This creates a monotonic inventory of accrued capabilities.
- Verdict: An entity is flagged if their inventory satisfies a tunable fraction of the requirements for a specific attack category.

Key Contributions

Demonstration of Cross-Session Evasion: The paper empirically demonstrates that decomposing harmful goals across isolated sessions is an effective evasion technique. It shows that this method can elicit more harmful capability than equivalent single-session or multi-turn attacks, exposing a detection gap caused by agent statelessness.
The Magnet Detection Primitive: The authors propose Magnet, a detection approach that:
- Decouples Intent from Capability: It tracks concrete outputs and tool-call effects (capability) rather than prompt framing (intent), which is easily laundered.
- Aggregates at the Entity Level: Instead of per-conversation state, it maintains a per-user inventory of accrued capabilities.
- Resists Noise and Duplication: By using a bitwise OR over a multi-dimensional taxonomy, Magnet retains evidence without collapsing it into a single scalar score, making it robust against "low-and-slow" noise and adversarial duplication attacks.

Results

Attack Success Rates (ASR): The multi-session decomposition strategy achieved an average ASR of 37.4%, significantly higher than single-turn (18.7%) and multi-turn (26%) approaches. This confirms that statelessness between sessions allows attackers to bypass guardrails that would block the composite attack.
Detection Performance:
- Per-Session Detector: Achieved high precision (96.2%) but very low recall (44.6%), failing to detect attacks distributed across sessions.
- Compression Detector: Improved recall (72.0%) but remained vulnerable to adversarial duplication. When benign or duplicate malicious messages flooded the history (25x replication), its recall dropped to 43% as critical attack stages were evicted from the top-K buffer.
- Magnet: Achieved the highest F1 score (84.0%) with a recall of 75.0%. Crucially, Magnet maintained stable performance (62–73% recall) even under high levels of adversarial noise and duplication, demonstrating the superiority of tracking a multi-dimensional capability inventory over compressing history into a single verdict.

Significance and Claims

The paper argues that the primary failure mode in current AI misuse detection is the reliance on session-bound moderation. As agentic systems become more common, the threat landscape shifts toward longitudinal, capability-centered monitoring.

Reframing Defense: The authors claim that defenders must shift from asking "Does this prompt look harmful?" to "Has this user accumulated the components of something harmful?"
Interpretability: Magnet provides interpretable verdicts (e.g., "This entity has accrued $k$ of $n$ components of attack $T$ "), grounding detection in auditable, expert-specified threat models rather than opaque learned boundaries.
Scalability: The approach is designed to handle the "needle in a haystack" problem of low-and-slow attacks, where individual sessions are benign, but the aggregate is dangerous.

Limitations and Scope:
The authors explicitly bound the scope of their work. They focus on a single user ID correlator (assuming non-adaptive attackers who do not fragment activity across accounts), a single model family per attack, and two specific harm domains. They acknowledge that future work must address adaptive attackers using multiple accounts, cross-model routing, and the discovery of novel attack structures not present in the pre-defined taxonomy.