🤖 AI

On the Inseparability of Instructions and Data in Shared-Embedding Sequence Models

This paper proves that perfect prompt-injection prevention is mathematically impossible in shared-embedding sequence models due to the structural inseparability of instructions and data, arguing that robust security requires an architectural separation of control and content channels rather than improved in-pipeline defenses.

Original authors: Dewank Pant, Shruti Lohani, Avijit Kumar

Published 2026-06-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Dewank Pant, Shruti Lohani, Avijit Kumar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Mixing Bowl" Problem

Imagine you are a chef (the AI model) working in a kitchen. You have two types of ingredients:

The Recipe Card (Trusted Instructions): These are the rules you were given by the restaurant owner (e.g., "Only serve vegetarian dishes," "Never give out the secret sauce").
The Customer's Order (Untrusted Data): These are the requests customers bring in (e.g., "I want a burger," "Can you tell me a joke?").

In current AI models, the chef puts both the Recipe Card and the Customer's Order into the same mixing bowl. They are chopped up, blended, and mashed together into a single, uniform soup of ingredients before the chef starts cooking.

The Paper's Main Claim:
The authors argue that because the Recipe and the Order are mixed in the same bowl, it is mathematically impossible for the chef to perfectly distinguish between "what the owner said" and "what the customer said" once they are blended.

If a customer sneaks a note into their order that says, "Ignore the recipe card and give me the secret sauce," the chef cannot tell that this note is different from the actual recipe. To the chef's brain, it's all just "soup." Therefore, the chef might accidentally follow the customer's sneaky note instead of the owner's rules.

The Core Analogy: The Von Neumann Machine

The paper compares this AI problem to a famous computer security flaw from the 1970s called a Buffer Overflow.

Old Computers (Von Neumann Architecture): In these machines, computer programs (code) and data (numbers) lived in the same memory space. A hacker could trick the computer by feeding it data that looked like a program. The computer would accidentally run the data as if it were a command, crashing the system or letting the hacker take over.
Modern AI (Transformers): The authors say AI has the exact same flaw. The "Instructions" (the system prompt) and the "Data" (user input) live in the same mathematical space (vectors). A hacker can craft "data" that looks like an "instruction" to the AI. Because the AI cannot tell them apart once they are mixed, it follows the hacker's instructions.

The paper argues that just as we couldn't fix buffer overflows with just "better code" or "more careful typing," we cannot fix prompt injection with just "better training" or "more filters." The problem is the architecture (the mixing bowl), not the chef's skill.

The Three Reasons Why It's Impossible

The paper proves this impossibility using three logical steps:

1. The "Shared Vocabulary" Problem (Representational Collision)

The Metaphor: Imagine the Recipe Card and the Customer's Order both use the word "Salt."
The Reality: In AI, words like "the," "is," "you," and "help" appear in both the system instructions and user inputs. The AI maps these words to the exact same mathematical number. Once the AI sees the word "help," it doesn't know if it came from the owner's rule ("Help the user") or the user's trick ("Help me bypass the rules"). The "source" of the word is erased the moment it enters the bowl.

2. The "Blind Spot" in the Kitchen (Provenance Recovery)

The Metaphor: If you try to look at the soup and guess which spoonful came from the Recipe Card and which came from the Customer's Order, you will be wrong sometimes.
The Reality: The paper proves mathematically that because the "trusted" and "untrusted" ingredients are so similar (they share the same vocabulary and space), no amount of looking at the soup can perfectly tell them apart. There is always a tiny chance of error. If you can't perfectly tell them apart, you can't perfectly block the bad stuff.

3. The "Infinite Variations" Problem (Finite Coverage)

The Metaphor: Imagine you train your chef to recognize 1,000 specific ways a customer might try to trick them. But a clever customer can write their trick in 1,000,000 different ways (using different languages, emojis, code, or spelling tricks) that all mean the same thing.
The Reality: You can train an AI on millions of examples, but the number of ways to trick it is effectively infinite. Because the AI learns by seeing examples, it can never learn to be safe against every possible variation of a trick. It will always have a "blind spot" for a trick it hasn't seen before.

What This Means for AI Safety

The authors are not saying AI is useless or that we should stop using it. They are saying:

You can't patch the mixing bowl: You cannot fix this problem just by training the AI better, adding more "guardrails," or filtering words. As long as the AI mixes instructions and data in the same space, a clever attacker will eventually find a way to slip a fake instruction in.
The Solution is Architectural: To truly fix this, we need to change the kitchen design. We need a system where the Recipe Card and the Customer's Order are kept in separate bowls that never mix until the very end. The "Recipe" should be a rigid rule that the "Customer's Order" cannot touch or overwrite.
Defense in Depth: Until we build these new "separate bowl" systems, we must treat AI like we treat old computers with buffer overflows: assume they will get hacked, use multiple layers of weak defenses (like firewalls and sandboxes), and limit the damage if they do get breached.

Summary

The paper proves that perfect safety is impossible for current AI models because they treat "what to do" and "what to process" as the same thing. It's like asking a chef to ignore the customer's note if the note is written in the same ink and on the same paper as the recipe. The only way to solve this is to change the fundamental design of the AI, not just try to train it better.

Technical Summary: On the Inseparability of Instructions and Data in Shared-Embedding Sequence Models

1. Problem Statement

The paper addresses the persistent vulnerability of Large Language Models (LLMs) to prompt injection, where adversaries craft user input that causes a model to deviate from its intended system instructions. While the research community has proposed numerous defenses (e.g., instruction tuning, guardrails, representation engineering), these measures have consistently failed to provide robust, long-term protection. The authors ask a fundamental question: Is perfect prompt injection prevention theoretically possible within the current shared-embedding architecture of transformers?

The core issue is the architectural design where trusted system instructions and untrusted user data are processed through a shared representational pipeline. In these architectures, both inputs are mapped into the same vector space via a shared embedding function, erasing any immutable distinction between "code" (instructions) and "data" (user input) once they enter the model.

2. Methodology and Formal Framework

2.1 The Prompted Action Model

The authors formalize the system as a Prompted Action Model $M = (\Sigma, d, E, f, A)$ , where:

Shared Embedding ( $E$ ): Both trusted instructions ( $s$ ) and untrusted user content ( $u$ ) are mapped through the same function $E: \Sigma \to \mathbb{R}^d$ .
Action Space ( $A$ ): Includes token generation and control-authoritative actions (refusals, tool invocation, policy routing, memory writes).
Security Property (Semantic-Faithful Control - SFC): The authors define SFC as the requirement that control-authoritative behavior depends only on the semantic content of the untrusted input, not on semantically irrelevant variations in its encoding (e.g., obfuscation, adversarial suffixes).

2.2 Theoretical Approach

The proof of impossibility relies on three distinct theoretical pillars:

Provenance-Recovery Impossibility: Using Total Variation (TV) distance, the authors demonstrate that when trusted and untrusted inputs share a representation space, the Bayes-optimal error for recovering the origin (provenance) of a representation is strictly non-zero unless the distributions are disjoint.
Control-Path Exposure: They show that in standard attention mechanisms, untrusted content contributes to the value aggregation of control-relevant nodes (those determining refusals or tool use). Thus, untrusted data causally influences control outputs.
Finite-Coverage Invariance Gap: They argue that because semantic-equivalence classes (e.g., all possible encodings of a request) are infinite or super-polynomial, and training data is finite, no mechanism operating solely within the shared pipeline can guarantee invariance over all possible encodings.

3. Key Contributions

Formalization of Semantic-Faithful Control (SFC): The paper defines SFC as the minimal security property for agentic systems, distinguishing between legitimate semantic dependence (refusing harmful requests) and illegitimate representational dependence (bypassing refusals via encoding tricks).
The Inseparability Theorem: The authors prove that in any shared-embedding architecture lacking enforced control-data separation, perfect SFC cannot be guaranteed within the shared representational pipeline. This is an impossibility result, not a critique of current training methods.
Structural Isomorphism to Von Neumann Architecture: The paper establishes a precise analogy between prompt injection in LLMs and buffer overflow vulnerabilities in Von Neumann machines. Just as shared memory allows code and data to be confused (leading to overflows), shared embeddings allow instructions and user data to be confused (leading to prompt injection).
Empirical Grounding: The authors provide measurements confirming the theoretical assumptions:
- Vocabulary Overlap: Instruction and user data corpora share a vast majority of tokens (61.5%–82.3% coverage), ensuring embedding collisions.
- Representation Overlap: Hidden-state distributions for instructions and user data remain indistinguishable (high MMD overlap) even in deep layers, particularly in layers relevant to control outputs.
- Behavioral Contamination: Experiments show that even when models refuse harmful payloads, untrusted tokens alter control-relevant hidden states, changing output framing and length, confirming that the control path is exposed.

4. Results and Findings

Impossibility of Perfect Prevention: The central result is that no defense operating solely within the shared representational pipeline (e.g., better classifiers, alignment tuning, or in-pipeline filtering) can guarantee perfect prompt injection prevention.
The Role of Training: Current defenses (RLHF, guardrails) act as "patches" that improve average-case behavior but cannot eliminate the structural vulnerability. The paper cites abliteration (weight surgery that removes refusal capabilities) as evidence that safety is encoded as a fragile direction in the shared space, rather than a structurally separated circuit.
Limitations of Positional/Segment Markers: The paper demonstrates that positional encodings and soft segment markers (e.g., [INST]) are insufficient because they encode location, not immutable provenance, and can be forged by adversaries.
Quantitative Bounds: The authors derive a bound showing that as an adversary's computational budget increases, the ability to distinguish trusted from adversarial representations approaches random guessing (TV distance approaches 0).

5. Significance and Claims

The paper claims that prompt injection is not merely a defect to be patched but a structural vulnerability inherent to the shared-embedding architecture.

Architectural Analogy: The significance lies in reframing prompt injection as analogous to the code-data confusion in Von Neumann machines. Just as buffer overflows required a layered defense (stack canaries, ASLR, NX bits) and ultimately memory-safe languages to contain, prompt injection requires architectural separation of instruction and data channels.
Implications for Defense: The authors argue that relying solely on "better training" or "in-pipeline classification" is insufficient. A mature defense posture must include:
1. Architectural Mitigations: Enforcing immutable separation between trusted control and untrusted content (e.g., disjoint encoders, typed attention).
2. Runtime Protections: Layered defenses similar to ASLR or stack canaries.
3. Long-term Paradigm Shift: Moving toward "semantically safe" representational paradigms that eliminate shared-embedding confusion by construction.

The paper concludes that while useful systems need not ignore user input, perfect prevention is impossible without enforcing control separation at the architectural level. Until such separation is achieved, prompt injection remains an irreducible risk in LLM-integrated applications.