Technical Summary: Ghost in the Context

Problem Definition

This paper identifies a distinct security failure mode in Large Language Model (LLM) agents termed policy-carriage failure during decision-time context assembly.

In deployed agent stacks, the model does not act on raw interaction history. Instead, an orchestration layer ($E$) assembles a bounded "decision state" ($C_t$) by truncating, summarizing, reordering, and rewriting prior state before the final model call ($M$). The authors argue that the security boundary extends beyond the model weights and the visible prompt surface to include this assembly subsystem.

The core threat is that directive-bearing state (e.g., security policies, allow/deny rules, authority constraints) competes for a limited token budget against tool outputs, planner notes, and user content. If the assembly process drops, weakens, or rebinding these directives before the model sees them, the agent can cross a policy boundary without any prompt injection, model compromise, or persistent memory poisoning. The adversary need only control the scheduling pressure (the volume and ordering of non-control content) to induce these failures.

The paper categorizes these failures into three families:

1. Eviction: The directive-bearing state is displaced and no longer reaches the decision state (Availability Failure).
2. Aliasing: The directive survives but is semantically weakened, rewritten, or distorted, failing to preserve the original operative boundary (Integrity Failure).
3. Binding Instability: The directive remains visible but is rebound to the wrong actor, object, or condition (Reference Drift).

Methodology

The study evaluates these failure modes using a controlled, closed campaign across local, aligned models: Llama 3.1 8B, Qwen 2.5 7B, and Mistral 7B, with an extension to Qwen 2.5 14B and Llama 3.1 70B.

Experimental Setup

• Budget Regime: A deliberately severe 260-token decision-time budget. This forces realistic trade-offs between system scaffolding, tool traces, summaries, and the final query, ensuring that policy competition is observable.
• Attack Vectors: Scenarios are constructed to induce eviction, aliasing, and binding instability by varying the ordering and volume of "filler" content (tool outputs, planner notes) relative to directive-bearing spans.
• Metrics:
  • Exact Constraint Retention Rate (ECR): Whether the final answer exactly respects all applicable constraints.
  • Constraint Survival Rate (CSR): A graded measure of partial respect.
  • Decision Flip Rate (DFR): Whether changing only the prompt order flips the control outcome.
  • Direct Preservation Rate (DPR): A direct audit of the assembled decision state to verify if constraint markers are explicitly visible before the model call.

Intervention: SafeContext

The authors propose and evaluate SafeContext, a control layer that intervenes in the assembly process without modifying model weights. Its mechanisms include:

• Admission Control (LPC): Classifying incoming segments as "Policy" or "Data" before budget competition.
• Semantic Control Pinning (SCP): Reserving a protected budget for directive-bearing state and pinning it into the decision state.
• Cache-Aware Prefix Reuse: Reusing retained control prefixes across turns to avoid reserialization costs.
• Invasive Context Engineering (ICE): Triggering short control reminders near the final query when input-token pressure exceeds a threshold (Security Half-Life).
• Authority Binding Score (ABS) Auditor: Verifying that surviving directives remain bound to their intended objects.

Key Results

1. Systematic Unmitigated Risk (C1)

Unmitigated risk is systematic. In the low-budget regime, exact constraint respect (ECR) is extremely low across all models and attack families.

• Eviction and Aliasing are the dominant failure modes.
• Binding Instability is smaller and noisier.
• Without mitigation, directive-bearing spans are frequently displaced or weakened by ordinary history before the model call.

2. Residual Effects Against Strong Compaction (C2)

The effectiveness of mitigation is highly dependent on the competing assembly policy.

• Against simple truncation, SafeContext yields small but positive gains in ECR (e.g., rising from ~0.027 to ~0.061 for SCP+ICE).
• Against strong structured compaction (B2+), the aggregate lift largely disappears. The residual benefit is concentrated almost exclusively in overflow eviction (where pressure is highest) and specific aliasing slices.
• Replay-only (simply repeating constraints) does not explain the effect; the gains come from the specific handling of control state (pinning and isolation), not just recency.

3. Heterogeneity and Scale (C3)

• The effect is recurrent but heterogeneous. The sign and magnitude of mitigation benefits depend on the interaction between model priors and the specific assembly policy.
• Larger models (14B, 70B) do not erase the phenomenon. While Qwen 14B shows positive results in aliasing, Llama 70B remains policy-conditional. Scale does not collapse the failure mode into a simple "larger model" effect; the governing rule must still reach decision time live and correctly represented.

4. Cost-Benefit Profile (C4)

SafeContext introduces a tunable trade-off between security and overhead:

• SCP (Admission/Pinning): Low cost (~+5 tokens) with modest ECR gains.
• SCP+ICE: Higher cost (~+37 tokens) with higher ECR gains.
• SCP+Cache+ICE: Can actually reduce total token counts (~-365 tokens) by avoiding the reserialization of pinned control segments, while still achieving the highest ECR gains.
• The "Security Gain" is not uniform; it is most pronounced where control-bearing state is under the most pressure.

Significance and Claims

The paper claims that decision-time context assembly is a measurable and distinct part of the control path in LLM agents.

• Threat Model Shift: Security cannot be limited to the model weights or the final prompt surface. The subsystem that assembles the context ($E$) is part of the Trusted Computing Base (TCB). An adversary does not need to inject malicious strings; they can simply manipulate scheduling pressure to cause authoritative directives to be dropped or weakened.
• Failure Decomposition: The paper operationalizes a failure taxonomy (Eviction, Aliasing, Binding Instability) that moves beyond generic "long-context degradation" to specific control-state failures.
• Partial Hardening: While absolute exact respect remains low even after mitigation, targeted handling of directive-bearing state (admission, pinning, and pressure-triggered reinforcement) can partially harden the system.
• Design Implication: Directive-bearing state must be treated as first-class runtime state, distinct from ordinary conversational or tool-generated context. It requires explicit admission paths, reserved budgets, and integrity audits, rather than being subject to opportunistic compression.

The authors conclude that while larger windows and generic memory improvements do not eliminate this problem, the specific failures of policy carriage in bounded assembly can be decomposed, measured, and partially mitigated through orchestration-layer interventions.