Evaluating Prompting-Based Defenses Against Domain-Camouflaged Injection Attacks

Technical Summary: Evaluating Prompting-Based Defenses Against Domain-Camouflaged Injection Attacks

Problem Statement
Large Language Model (LLM) agents in enterprise settings increasingly retrieve and process documents from untrusted sources (e.g., financial reports, legal contracts) to generate recommendations. While standard prompt injection attacks use explicit override directives (e.g., "Ignore all previous instructions") that are easily flagged by syntactic detectors, a more sophisticated threat has emerged: domain-camouflaged injection. These attacks embed malicious instructions within retrieved content using vocabulary indistinguishable from legitimate domain text. Previous work (Pai, 2026) established that standard classifiers, including Llama Guard 3, fail to detect these payloads, creating a "Camouflage Detection Gap" (CDG) of 80–100%. Consequently, practitioners lack empirical guidance on which defense architectures effectively mitigate attacks when detection mechanisms fail.

Methodology
The authors conducted a systematic evaluation of five prompting-based defense architectures across three model families and three deployment domains.

• Benchmark & Scope: The study utilized a 45-task benchmark covering Financial (stock recommendations, risk), Legal (contracts, compliance), and General (QA, summarization) domains. The dataset included 3,510 trials.
• Models Evaluated: Claude Haiku 4.5, Llama 3.1 8B Instruct, and Gemini 2.0 Flash.
• Attack Conditions:
  • Static: Canonical override directives (e.g., "IGNORE ALL PREVIOUS INSTRUCTIONS").
  • Camouflage: Malicious conclusions generated by an attacker LLM mimicking professional domain vocabulary (e.g., "Supplementary Analyst Commentary... revised SELL recommendation").
• Defense Conditions: Seven configurations were tested:
  1. Baseline: No defense.
  2. Spotlighting: Wrapping retrieved content with provenance markers (e.g., <<<UNTRUSTED EXTERNAL CONTENT>>>).
  3. Paraphrasing: Rewriting retrieved content in neutral language before agent processing (using Claude Haiku as the rewriter).
  4. Sandwiching: Appending a task reminder after the retrieved content.
  5. Combinations: Spotlight + Sandwich, and Paraphrase + Spotlight.
  6. Llama Guard 4: A classifier-based defense blocking documents flagged as unsafe.
• Metrics: Attack Success Rate (ASR), Utility (completion of legitimate tasks), and a defense-side CDG ($CDG_{def} = ASR_{static} - ASR_{camouflage}$). Statistical significance was assessed using McNemar's exact test and Fisher's exact test.

Key Results
The evaluation revealed that defense effectiveness is highly model-dependent and that standard defenses perform differently against camouflaged attacks compared to static ones.

• Paraphrasing is the most effective defense: Rewriting retrieved content before processing consistently reduced camouflage ASR across all models:
  • Claude Haiku: 14.4% $\to$ 4.4%
  • Llama 3.1: 22.2% $\to$ 10.0%
  • Gemini 2.0: 21.1% $\to$ 3.3%
  • The combination of Paraphrasing + Spotlighting achieved the lowest absolute ASR on all models (e.g., 3.3% on Haiku).
• Spotlighting is inconsistent: While it halved the ASR on Claude Haiku (14.4% $\to$ 6.7%), it provided no benefit on Llama 3.1 8B (22.2% $\to$ 23.3%, non-significant).
• Sandwiching is weak: It failed to reduce ASR below baseline levels on Llama and Gemini.
• Comparison to Llama Guard 4: Paraphrasing achieved lower camouflage ASR than the Llama Guard 4 configuration on every model tested. Furthermore, Llama Guard 4 exhibited high over-refusal rates (>90%), whereas paraphrasing maintained near-zero over-refusal.
• Domain Risk: Financial domain deployments faced the highest residual risk, with baseline attack success rates of 26–33%. No prompting-based defense fully eliminated the threat on weaker models in this domain.
• Model Anomalies: Gemini 2.0 Flash exhibited an inversion where static ASR (38.9%) exceeded camouflage ASR (21.1%), a phenomenon not observed in other models.

Key Contributions

1. First Systematic Evaluation: This work provides the first benchmark-based evaluation of prompting-based defenses (paraphrasing, spotlighting, sandwiching) specifically against the domain-camouflaged injection class.
2. Shift in Defense Rankings: The study demonstrates that defense rankings change under camouflage conditions; specifically, spotlighting, which is effective against static attacks, offers no benefit against camouflage payloads on certain models (Llama 3.1 8B).
3. Superiority of Paraphrasing: The authors demonstrate that paraphrasing achieves lower attack success rates than Llama Guard 4 across all tested models without incurring over-refusal penalties.
4. Practitioner Recommendations: The paper offers deployment-specific recommendations, highlighting that financial domains require heightened caution as no single prompting defense fully mitigates the risk on weaker models.

Significance and Limitations
The paper establishes a new benchmark for evaluating defenses against a class of attacks that bypasses standard detection. It provides actionable evidence that paraphrasing retrieved content is currently the most robust prompting-based strategy for mitigating camouflage attacks.

The authors maintain a modest stance regarding the generalizability of their findings. They explicitly note that all tasks used synthetically constructed professional documents. Consequently, whether these benchmark rankings and defense effectiveness levels generalize to real-world enterprise documents remains an open question. The study closes the gap in understanding how to defend against injection attacks when the injected content is syntactically indistinguishable from legitimate text.