🤖 AI

Can Transformer Memory Be Corrupted? Investigating Cache-Side Vulnerabilities in Large Language Models

This paper introduces Malicious Token Injection (MTI), a framework demonstrating that perturbing the key-value cache during inference can systematically corrupt transformer language models, revealing cache integrity as a critical and previously overlooked vulnerability in LLM security.

Original authors: Elias Hossain, Swayamjit Saha, Somshubhra Roy, Ravi Prasad

Published 2026-02-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Elias Hossain, Swayamjit Saha, Somshubhra Roy, Ravi Prasad

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) like a brilliant, fast-thinking student taking a long exam. To answer a question, this student doesn't just look at the current question; they need to remember everything they've written down so far to keep their answer consistent. In the world of AI, this "memory" is called the KV Cache (Key-Value Cache). It's a digital scratchpad where the model stores the most important parts of the conversation so it doesn't have to re-read the whole book every time it wants to write the next word.

This paper, titled "Can Transformer Memory Be Corrupted?", asks a scary but simple question: What happens if someone secretly scribbles on that scratchpad while the student is taking the test?

Here is the breakdown of their findings using everyday analogies:

1. The Hidden Weakness: The "Invisible" Scratchpad

Usually, we worry about hackers changing the student's textbooks (the model's training data) or tricking them with a weird question (prompt injection). But this paper found a different, sneakier way to break the system.

The researchers realized that the "scratchpad" (the KV Cache) is often left unprotected. Even if the textbooks and the exam questions are locked down, an attacker who can get inside the computer running the model can subtly tweak the numbers on that scratchpad.

The Analogy: Imagine a chef cooking a complex stew. The recipe is safe, and the ingredients are fresh. But if a saboteur sneaks in and quietly swaps a pinch of salt for sugar while the chef is tasting the pot, the whole dish could go wrong, even though the chef didn't change the recipe or the ingredients.

2. The Attack: "Malicious Token Injection" (MTI)

The authors created a tool called MTI V.1 to test this. They didn't break the model; they just "nudged" the memory. They tried three main ways to mess with the scratchpad:

The "Static" Nudge (Gaussian Noise): Adding a little bit of random static or fuzz to the memory, like turning the volume up slightly on a radio until the words get garbled.
The "Eraser" (Zeroing): Wiping out specific parts of the memory entirely, like taking a red marker to a page of notes.
The "Twist" (Rotation): Turning the memory sideways. The information is still there, but it's pointing in the wrong direction, confusing the model's understanding.

3. The Results: Small Nudges, Big Messes

The researchers tested this on popular AI models (like GPT-2 and LLaMA-2). They found that even tiny, almost invisible changes to the memory caused big problems:

Confusion: The model started guessing words it shouldn't. It became less confident and more likely to make up facts (hallucinate).
Task Failure:
- Simple Tasks: When asked to guess if a sentence was happy or sad, the model got confused and started getting it wrong.
- Complex Tasks: When asked to find a specific fact in a long document (like a trivia question), the model completely failed. It was like the student forgot how to read the question entirely.
- The "Agent" Test: When the AI was acting like a robot trying to plan a series of steps (like booking a flight), the memory corruption made it lose its way and pick the wrong actions.

4. The "Why": How It Spreads

The paper explains the math behind this simply: The AI decides what word to say next by looking at its memory. If the memory is slightly wrong, the AI's attention shifts.

The Analogy: Imagine you are trying to find a friend in a crowded room. You look at a map (the memory). If someone draws a tiny, wrong line on the map, you might look in the wrong corner. Once you look in the wrong corner, your whole plan for the rest of the night falls apart. The paper proves that these "tiny lines" on the map can mathematically guarantee that the AI's attention gets skewed.

5. Can We Fix It? (The Defenses)

The researchers tried a few quick fixes to see if they could stop the attack:

Clearing the Scratchpad: Periodically wiping the memory clean.
Randomizing the Notes: Randomly hiding parts of the memory to see if the model can still guess.
Smoothing the Noise: Averaging out the numbers to remove the "static."

The Verdict: These fixes helped a little bit, but they weren't a magic cure. They stopped the worst of the damage, but if the attack was strong or kept happening, the model still failed. It's like putting a bandage on a cut; it helps, but it doesn't stop the knife from cutting deeper if the attacker keeps trying.

The Bottom Line

This paper doesn't say that AI is broken or that you should stop using it. Instead, it sounds an alarm bell for the people who build and protect AI systems.

The Main Takeaway: We have been very good at protecting the AI's "brain" (the training data) and its "mouth" (the input prompts), but we have largely ignored its "short-term memory" (the KV Cache). If an attacker can get inside the computer running the AI, they can corrupt that memory and make the AI unreliable, confused, or dangerous, even without changing a single line of code.

The authors are calling for a new kind of security that treats this "scratchpad" as a critical safety zone that needs to be guarded just as strictly as the rest of the system.

Technical Summary: Investigating Cache-Side Vulnerabilities in Large Language Models

Problem Statement
While Large Language Models (LLMs) are increasingly secured through prompt-level safeguards and training-time alignment, their internal inference-time state remains an under-examined attack surface. Specifically, the Key-Value (KV) cache, which stores intermediate attention representations to accelerate autoregressive decoding, introduces a persistent internal memory that directly influences future predictions. Unlike prompt injection (which operates at the input interface) or weight poisoning (which requires training-time access), cache-side perturbations operate entirely after tokenization and embedding. These attacks bypass input-level filters and alignment checks by modifying cached key vectors, potentially biasing attention distributions and propagating errors through downstream predictions without altering model parameters or input prompts.

Methodology: The MTI V.1 Framework
To investigate this vulnerability, the authors introduce MTI V.1 (Malicious Token Injection), a modular framework that formalizes cache-side perturbations. The framework treats the KV cache as a distinct robustness boundary and defines a threat model where an adversary with inference-time access can perturb cached keys at selected layers and timesteps without modifying inputs or weights.

The framework characterizes perturbations along three axes:

Corruption Magnitude: The strength of the injected perturbation.
Injection Frequency: How often perturbations are applied across decoding steps.
Layer Placement: Which transformer layers are targeted.

The study employs three canonical corruption mechanisms:

MTI-Gaussian: Injects additive stochastic noise (sampled from a Gaussian distribution) into cached keys.
MTI-Zeroing: Removes cached keys entirely, simulating catastrophic erasure.
MTI-Rotation: Applies orthogonal transformations to cached keys, altering angular alignment with queries while preserving vector magnitude.

Additionally, an optimized variant (Algorithm 1) is used empirically to compute perturbations that maximize an adversarial objective via gradient ascent, though this is distinct from the non-adaptive, norm-bounded perturbations used in theoretical analysis.

Theoretical Analysis
The paper provides a theoretical analysis bounding how norm-limited cache perturbations propagate through attention mechanisms.

Perturbation-to-Logit Bound: The deviation in attention scores scales linearly with both the perturbation magnitude ( $\epsilon$ ) and the query norm ( $\|q_t\|_2$ ). The resulting deviation in unnormalized logits is bounded by $\|\Delta z_t\|_2 \leq \|q_t\|_2 \cdot \epsilon$ .
Softmax Stability: Under bounded-logit assumptions, the softmax mapping is Lipschitz continuous. This allows the translation of logit deviations into controlled changes in the attention distribution and output probabilities.
The analysis establishes that bounded cache corruption induces bounded deviations in attention distributions and next-token logits, characterizing worst-case sensitivity trends.

Experimental Results
The framework was evaluated across GPT-2 Medium, LLaMA-2 7B, and Gemma-7B models on diverse benchmarks including sentiment classification (SST-2), extractive question answering (SQuAD, HotpotQA), summarization (CNN/DailyMail), and agentic reasoning pipelines.

Token-Level Effects: Cache perturbations induce distributional shifts (measured by KL divergence) an order of magnitude larger than stochastic variation. For instance, rotation-based corruption on GPT-2 resulted in a KL divergence of 0.57 and a 16.7% drop in Top-1 accuracy.
Downstream Task Degradation: Sensitivity varies by task. While SST-2 showed moderate robustness, SQuAD (extractive QA) exhibited catastrophic failure, with F1 scores collapsing from 77.4 to 7.2 under identical perturbations. This suggests tasks requiring precise token alignment and long-context reasoning are highly sensitive.
Retrieval-Augmented Generation (RAG): Perturbations applied to retrieved context significantly degraded grounding fidelity and increased hallucination rates, indicating RAG pipelines are critically dependent on cache integrity at the evidence stage.
Agentic Pipelines: In ReAct and AutoGPT-style frameworks, closed-loop reasoning showed partial resilience to mild perturbations, though this robustness was limited to mild regimes.
Ablation Studies: Vulnerability is governed by the interaction of layer placement, perturbation strength, and temporal persistence. Mid-level layers were found to be consistently more vulnerable than early or deep layers. Continuous corruption caused greater degradation than intermittent schedules, indicating that cache perturbations are stateful and compound over time.
Defenses: Lightweight defenses (Cache Reset, Dropout Mask Randomization, Attention Smoothing) were evaluated. While they preserved baseline accuracy under mild perturbations with minimal runtime overhead, they were insufficient as standalone safeguards against persistent or strong corruption.

Significance and Claims
The paper posits that inference-time cache corruption represents a systematic threat to the reliability, calibration, and grounding of modern LLM pipelines. Key contributions include:

Formalizing a Threat Model: Identifying KV cache integrity as a critical, underexplored robustness boundary and formalizing cache-side perturbation as a principled threat model for trustworthy AI.
Theoretical Bounds: Providing theoretical bounds that characterize how bounded cache corruption propagates to attention distributions and output logits.
Empirical Evidence: Demonstrating that cache perturbations induce distributional shifts and task failures across language modeling, RAG, and agentic reasoning, even when prompts and weights remain unchanged.
Diagnostic Utility: Suggesting that controlled cache perturbations can serve not only as a threat model but also as a diagnostic tool for analyzing attention sensitivity and internal model dynamics.

The authors conclude that while simple defenses offer partial mitigation, explicit cache integrity mechanisms or monitoring strategies are necessary for robust deployment. The work positions inference-time cache integrity as a "first-class robustness boundary" for future research in trustworthy AI, emphasizing that these findings characterize a robustness boundary rather than asserting universal exploitability across all deployment scenarios.