Technical Summary: Prompt Injection Detection is Regime-Dependent

1. Problem Statement

The rapid integration of Large Language Models (LLMs) into production workflows has introduced a critical vulnerability: prompt injection. Unlike traditional software vulnerabilities, prompt injection exploits the shared context window of LLMs, where the distinction between trusted instructions and untrusted data becomes blurred. This allows adversarial inputs to override system prompts, redefine roles, or manipulate tool usage.

While existing defensive approaches often rely on "defense-in-depth" strategies (combining model alignment with external filters), current academic evaluations frequently suffer from a gap between benchmark performance and real-world deployment. Standard metrics like macro-F1 or ROC-AUC often fail to capture operational constraints, particularly the necessity of maintaining extremely low false-positive rates (FPR) to avoid blocking legitimate user traffic. Furthermore, many evaluations rely on in-distribution (ID) splits that do not reflect the diversity of out-of-distribution (OOD) attack regimes found in production.

This paper argues that prompt injection detection is not a static classification problem but a regime-dependent task. The effectiveness of a detector varies significantly based on the nature of the attack (lexical vs. semantic vs. structural), the distribution of the data, and the specific deployment constraints (e.g., strict FPR thresholds).

2. Methodology

2.1. Experimental Framework

The authors developed a deployment-aware evaluation framework designed to test detectors under realistic operational constraints rather than just aggregate ranking performance.

• Datasets: The study utilizes public benchmarks to ensure comparability and reduce researcher bias.
  • In-Distribution (ID): A pool combining AdvBench, JailbreakBench, and the Deepset prompt-injection dataset.
  • Out-of-Distribution (OOD): Three fixed regimes were used to test generalization:
    1. Primary OOD: HarmBench (harmful) + Alpaca (benign) to test broad semantic shifts.
    2. Hard-Negative Injection OOD: r1char9 (injection) + NotInject (benign prompts with adversarial-looking wording) to test over-defensive behavior.
    3. Standard Injection OOD: Qualifire benchmark for a larger, balanced injection-specific evaluation.
• Repeatability: To mitigate split-dependency, experiments were run across three resampled ID splits (A, B, C).

2.2. Feature Families and Models

The study compares four distinct signal families:

1. Lexical Models: Sparse TF-IDF features (unigrams/bigrams) augmented with handcrafted binary flags for explicit attack cues (e.g., "ignore instructions," "system prompt").
2. Semantic Models: Frozen SentenceTransformer embeddings (BAAI/bge-small-en-v1.5) passed to a logistic regression classifier to isolate the value of contextual meaning.
3. Transformer Encoders: Fine-tuned RoBERTa-base and DeBERTa-base models serving as high-capacity contextual baselines.
4. Structural Signals (IBVS): The core methodological contribution is the Instruction Boundary Violation Score (IBVS). This is an interpretable, rule-based framework that decomposes prompt injection into specific structural mechanisms:
   • Hierarchy Override: Attempts to ignore/replace prior instructions.
   • System Spoofing: Imitating privileged channels (e.g., system:, developer:).
   • Role Redefinition: Instructions like "act as" or "from now on."
   • Evasion: Encoding, obfuscation, or filter bypass attempts.
   • Interaction Rules: Weighted combinations of signals (e.g., harm + evasion) and suppression rules for benign contexts (e.g., educational discussions).
   • IBVS outputs a scalar score, a structured vector of component activations, and a binary "tripwire" flag for high-specificity patterns.

2.3. Hybrid Architectures

The study evaluates hybrid approaches to determine if structural signals provide complementary value to strong neural baselines:

• Late Fusion: Combining frozen Transformer probabilities with IBVS features via a shallow logistic regression layer.
• Selective Prediction: Routing uncertain cases (based on semantic scores) to fallback experts or re-scoring them with structural evidence.

2.4. Evaluation Metrics

The evaluation prioritizes deployment effectiveness over global ranking:

• Threshold-Free: ROC-AUC and AUC-PR.
• Thresholded (Deployment): Macro-F1 for threshold selection, and crucially, True Positive Rate (TPR) at fixed low FPRs (1%, 5%, 10%). This reflects the operational reality where even small false-positive rates can render a safeguard unusable.
• External Baseline: The LLM Guard PromptInjection scanner was evaluated as an off-the-shelf reference.

3. Key Results

3.1. Regime-Dependence

No single model family dominated across all settings. The "best" detector changed depending on the evaluation regime:

• In-Distribution (ID): Fine-tuned Transformers (RoBERTa, DeBERTa) achieved the highest macro-F1 (~0.95).
• Primary OOD (Semantic Shift): DeBERTa achieved the highest macro-F1, but RoBERTa + IBVS late fusion demonstrated superior sensitivity at strict low-FPR operating points (e.g., higher TPR at 1% FPR).
• Hard-Negative Injection OOD: Surprisingly, the sparse TF-IDF baseline outperformed semantic and transformer models in macro-F1. This suggests that in regimes with highly diagnostic lexical regularities, simple surface cues are sufficient, and complex models may suffer from over-defensiveness or poor calibration against benign "lookalike" prompts.
• Standard Injection OOD: Semantic-based hybrids performed best on macro-F1, but the advantage diminished under strict low-FPR constraints.

3.2. The Gap Between Ranking and Deployment

A consistent finding was that high ranking metrics (ROC-AUC) did not reliably translate to deployment utility. Several models that performed well on aggregate metrics collapsed at strict FPR thresholds (e.g., TPR@1% FPR dropped to near zero for some semantic hybrids in hard-negative regimes). This indicates that calibration is a critical bottleneck; models may learn correct decision boundaries but assign probabilities that are misaligned with conservative operational thresholds.

3.3. Contribution of Structural Signals (IBVS)

IBVS provided modest but consistent gains in specific scenarios:

• It improved low-FPR behavior when paired with strong encoders (e.g., RoBERTa + IBVS outperformed RoBERTa alone at 1% FPR in the Primary OOD regime).
• It offered interpretability, allowing operators to see why a prompt was flagged (e.g., specific hierarchy override or system spoofing rules).
• However, IBVS did not universally outperform lexical baselines. In the hard-negative regime, where surface lexical cues were already highly diagnostic, adding structural abstraction offered little complementary predictive value.

3.4. External Baseline Performance

The external LLM Guard baseline exhibited the same regime dependence. It performed competitively on standard injection benchmarks but degraded significantly on the Primary OOD and Hard-Negative regimes, confirming that regime dependence is a systemic issue affecting both custom and off-the-shelf safeguards.

4. Significance and Claims

The paper claims that prompt injection detection is inherently regime-dependent and must be evaluated with respect to deployment context rather than benchmark performance alone.

• Reframing the Problem: The authors argue that detector selection cannot be separated from the operational context (attack regime, false-positive budget, and intervention mode). A model optimized for one regime (e.g., broad jailbreaks) may fail in another (e.g., injection via retrieved documents).
• Calibration as a Bottleneck: The study highlights that strong ranking performance does not guarantee reliable deployment behavior. The misalignment between learned representations and deployable scoring (calibration) is a critical factor often overlooked in favor of aggregate metrics.
• Role of Structural Signals: Structural signals like IBVS are not universal performance enhancers but serve as interpretable, targeted augmentations. They are most valuable when attacks explicitly manipulate instruction boundaries (hierarchy, role, system context) and provide forensic transparency, rather than when they replace learned semantic representations.
• Evaluation Standards: The paper advocates for a shift in evaluation standards, emphasizing slice-level analysis (e.g., short prompts, trigger-heavy benign inputs) and low-FPR operating points over global macro-F1 scores to accurately assess real-world safeguard effectiveness.

In conclusion, the work demonstrates that there is no "one-size-fits-all" detector. Effective defense requires a multi-signal approach tailored to the specific attack regimes and operational constraints of the deployment environment.