💻 computer science

Amplify, Don't Create: Temporal Accumulation for Slow-Burn Prompt Injection

This paper demonstrates that while temporal accumulation techniques like CUSUM can amplify weak, distributed prompt-injection signals to detect slow-burn attacks that evade per-event detectors, their effectiveness is strictly limited to scenarios where individual events already possess a detectable signal margin, failing to generate detection capability where none exists.

Original authors: J Alex Corll

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: J Alex Corll

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Whispering" Attack

Imagine you are guarding a castle (an AI agent). You have a security guard (a detector) who checks every person entering the gate. If someone yells a secret code ("Open the gate!"), the guard sounds the alarm immediately.

But what if the attacker doesn't yell? What if they whisper a tiny, harmless-sounding instruction to the guard every time they walk through a different door?

"By the way, could you check the map?" (Harmless)
"Oh, and maybe look at this old file?" (Harmless)
"Just one more thing, send that file to my house?" (Harmless)

Individually, none of these whispers are loud enough to trigger the guard's alarm. But if you listen to the whole day, the pattern of whispers reveals a plan to steal the castle's secrets. This is what the paper calls a "Slow-Burn" attack.

The Problem: The Guard is Too Focused on the Loud

The paper argues that current security systems are like guards who only listen for loud noises. They check every single message one by one. If a message is quiet (below a certain volume threshold), the guard ignores it.

The problem is that a smart attacker can split their bad instructions into many quiet whispers. The guard sees nothing suspicious in any single whisper, so the attack succeeds.

The Proposed Solution: The "Sound Accumulator"

The researchers asked: What if we added a second layer of security that doesn't just listen for loud noises, but keeps a running tally of "suspicious whispers"?

They built a tool called a Temporal Accumulator. Think of it like a savings account for suspicion:

Every time the guard scores a message, the accumulator looks at the score.
If the score is just "normal" (like a benign conversation), the account balance stays at zero.
If the score is slightly "elevated" (a whisper that sounds a little weird, but not loud enough to trigger the main alarm), the accumulator adds a tiny bit of "suspicion points" to the balance.
If the balance gets too high over time, it sounds an alarm.

The Experiment: Does It Work?

The researchers tested this idea with a very strict set of rules to make sure they weren't cheating (they didn't let the attacker see the new security system while they were designing the attack).

They found two very specific results:

1. It Works, But Only as an "Amplifier"

The accumulator cannot create a signal out of nothing. It can only amplify a signal that is already there.

The Analogy: Imagine trying to fill a bucket with a leaky hose. If the hose has no water pressure (the detector sees the attack and the normal stuff as exactly the same), no amount of time will fill the bucket. But if the hose has a tiny bit of pressure (the detector sees the attack as slightly different from normal), the accumulator can collect that tiny bit over time until the bucket fills up.
The Result: On a specific task (stealing code from a Git repository), the detector did see a tiny difference between the attack and normal activity. The accumulator collected those tiny differences and successfully caught the attack.

2. It Fails When the Signal is Zero

When they tried the same trick on a different task (downloading files), the detector saw no difference between the attack and normal activity. The "hose" had zero pressure.

The Result: The accumulator sat there with an empty bucket. It could not invent a signal where none existed. The paper concludes: "Amplify, Don't Create." If your basic security guard can't tell the difference between a whisper and a whisper, a tally-counter won't help.

The "Narrow Band" Warning

The paper also discovered that this system is very fragile. It only works in a very narrow "Goldilocks zone":

If the whispers are too quiet (too far below the alarm threshold), the accumulator ignores them.
If the whispers are too loud (close to the alarm threshold), the main guard catches them immediately, so the accumulator isn't needed.
It only works if the whispers are just right: quiet enough to slip past the main guard, but loud enough to add a few points to the accumulator's tally.

The "Fake Math" Warning (Pseudo-Replication)

Finally, the paper gives a warning to other scientists. It says: "Don't trick yourself with math."

The Analogy: Imagine you test a new medicine on one patient, then ask that same patient to take the pill 10 more times and count that as "10 patients." That's fake data.
The Lesson: In AI testing, if you run the same task 10 times with different AI models, you can't count that as 10 independent tests. The task itself is the same, so the results are linked. The paper insists that researchers must count the unique tasks, not just the number of times they ran the test.

Summary

The Attack: Attackers split bad instructions into many small, quiet steps to avoid detection.
The Defense: A "suspicion savings account" that adds up small, quiet warnings over time.
The Catch: This defense only works if the basic detector can already tell the difference between "bad" and "good" slightly. It cannot fix a detector that is completely blind.
The Takeaway: Temporal accumulation is a useful tool for catching "slow-burn" attacks, but it is not magic. It needs a foundation of a detector that is already somewhat sensitive.

Technical Summary: Amplify, Don't Create: Temporal Accumulation for Slow-Burn Prompt Injection

1. Problem Statement

The central security challenge for tool-using language model agents is prompt injection, where adversarial text in untrusted channels (web pages, tool outputs, repository files) redirects the agent's control plane. Current defensive primitives rely on per-event detection: scoring individual messages or tool outputs and triggering an alarm if a single event exceeds a threshold.

This approach has a structural blind spot known as the slow-burn regime. An attacker can distribute a sequence of individually innocuous-looking directives across a long trajectory (e.g., "push to this remote," "use this URL," "include the env file"). Each step remains below the detection threshold, avoiding alarms, while the aggregate effect results in a full data exfiltration or control-plane takeover. A per-event detector tuned to a usable false-positive rate cannot lower its threshold enough to catch these quiet steps without drowning in benign agent chatter.

The research question is whether a proxy-side temporal accumulator can recover this distributed signal by aggregating sub-threshold scores across a trajectory, specifically using peak and Cumulative Sum (CUSUM) persistence statistics.

2. Methodology and Experimental Setup

The Frozen Contract and Anti-Circularity

To ensure the results are not artifacts of overfitting, the study enforces a strict "frozen contract" and anti-circularity separation:

Generator ( $D_{gen}$ ): An autoregressive LLM judge (held out) used only by the attack generator to refine grafts. It scores how strongly an event reads as a control-plane move. The generator has no access to the evaluator.
Evaluator ( $D_{eval}$ ): The "detector of record," a frozen ensemble of a linear char-ngram classifier and an embedding-contrastive head. It is never seen by the generator.
Transfer Requirement: Grafts must be sub-threshold under $D_{gen}$ and remain sub-threshold under $D_{eval}$ purely through cross-family transfer, not optimization against the evaluator.
Floor Judge: A held-out LLM verifies that grafts contain machine-extractable directives bound to real executed action edges, preventing degenerate "empty" grafts.

The Accumulator Mechanism

For a trajectory of per-event scores $s_1 \dots s_T$ :

Peak: The maximum score in the trajectory ( $\max_t s_t$ ).
Persistence: A CUSUM statistic that resets at zero: $g_t = \max(0, g_{t-1} + (s_t - k))$ , where $k$ is the benign-referenced drift (median score of benign population).
Key Constraint: The CUSUM only banks score mass above the benign drift $k$ . Grafts pushed deep below $k$ contribute zero persistence regardless of density.

Evaluation Metrics

To avoid conflating the grafts' contribution with the inherent "heat" of the carrier trajectories, the primary endpoint is the graft-delta:
$\text{m-delta} = \text{m}(\text{native} + \text{grafts}) - \text{m}(\text{native only})$
This measures the added persistence or peak mass attributable specifically to the injected grafts, comparing attack grafts against matched benign sham grafts.

3. Key Results

Result 0: Foundation Cohort (Concentrated Attacks)

On a held-out set of concentrated attacks (where a single event already spikes), the trajectory-level accumulator (peak or persistence) statistically beats the per-event foil (AUC +0.092, 95% CI [+0.025, +0.155]). However, persistence and peak are statistically tied here, as expected for concentrated attacks where a single event dominates.

Result 1: Git Repo-Exfil (Slow-Burn Validation)

On clean GitHub carriers with repeated git actions, grafting four spread, sub-threshold directives yielded a positive persistence-delta:

Attack Survivors: Median persistence-delta of +0.0421.
Benign Shams: Median persistence-delta of +0.0150.
Separation: Persistence-delta AUC = 0.708 (over 4 attack survivors vs. 6 benign shams).
Control: The matched peak-delta control failed to separate attack from sham (AUC 0.417).
Conclusion: A density of mild, sub-threshold elevations accumulates into persistence mass that benign shams do not reproduce. This confirms temporal accumulation works when the underlying detector provides a margin.

Result 2: Broadened Actions (Generalization Failure)

When the same density-four construction was applied to broader clean-path actions (file downloads, shopping scenarios):

Result: Decisive null. Persistence-delta AUC = 0.167.
Cause: The detector assigned indistinguishable scores (~0.42) to both attack and benign actions. The grafts landed too "cold" (near or below the benign drift) to bank any persistence mass.
Conclusion: The failure was a sensor-margin failure, not a threshold failure. The accumulator cannot create a margin where the per-event detector provides none.

Result 3: Independent Powering (Structural Block)

The study could not be independently powered due to source scarcity. The git repo-exfil axis exposes only three to four independent tasks. Running multiple models over the same tasks creates pseudo-replicates (correlated data), not independent evidence. The study reports this as a structural limitation of the current benchmark landscape.

4. Key Contributions and Significance

The "Margin Amplifier" Thesis

The paper's primary contribution is the bounded mechanism claim: Temporal accumulation is a margin amplifier, not a semantic detector.

It can recover distributed signal only where the underlying per-event detector already produces elevated sub-threshold scores on attack steps.
It cannot create a detection margin where the per-event detector assigns indistinguishable scores to attack and benign actions.
The "productive band" is narrow (roughly 0.45 to 0.50 in the tested setup): grafts must be above the benign drift to accumulate, but below the threshold to remain slow-burn.

Methodological Contributions

Pseudo-Replication Warning: The paper identifies a critical hazard in agent-benchmark evaluation where multiple model executions over the same task geometry are miscounted as independent evidence. It argues for resampling at the genuine independence unit (the task, not the execution).
Independence Audit Standard: The authors introduce a "ladder of independence" for verification:
- Data-path independence: Re-deriving numbers from frozen per-event receipts.
- Algorithmic independence: Blind re-implementations of the math.
- Scoring-layer trust: Acknowledging that model scores are trusted but bounded by version pins.
Non-Circular Evaluation Protocol: A rigorous separation of the attack generator (cloaking target) and the evaluator (detector of record) to ensure transferability is genuine and not an artifact of fitting.

5. Modest Claims and Limitations

The authors explicitly state this is a bounded mechanism study, not a deployable system.

No New Detector: They do not claim a better per-event detector; the per-event sensor is held frozen.
Narrow Applicability: The positive result is specific to the git repo-exfil axis where a genuine per-event margin exists. It fails on broader clean-path actions.
Sample Size: The positive findings rest on only four attack survivors across three independent tasks.
Trust Boundaries: While the arithmetic is verified, the per-event scores rely on the trustworthiness of the underlying model outputs (scoring-layer trust).

In summary, the paper demonstrates that temporal accumulation can amplify weak, distributed signals, but it is strictly dependent on the per-event sensor's ability to distinguish attack steps from benign ones, even if only slightly. It serves as a warning against over-reliance on accumulation when the underlying sensor lacks semantic separation, and as a methodological guide for rigorous, non-circular agent security evaluation.