Technical Summary: Zero-Shot Embedding Drift Detection (ZEDD)

1. Problem Statement

Prompt injection attacks represent a critical vulnerability in Large Language Model (LLM) applications, where adversarial inputs exploit indirect channels (e.g., emails, user-generated content) to bypass alignment safeguards and induce harmful outputs. Despite advancements in model alignment via Reinforcement Learning from Human Feedback (RLHF), state-of-the-art models remain susceptible to both manual and automated adversarial prompts. Existing defense mechanisms often suffer from significant limitations: they rely on heavy computational overhead, require task-specific retraining, depend on internal model access, or utilize inefficient rule-based filters. There is a pressing need for a robust, generalizable, and lightweight detection mechanism that operates without prior knowledge of specific attack patterns.

2. Methodology: Zero-Shot Embedding Drift Detection (ZEDD)

The authors propose ZEDD, a framework that detects prompt injections by quantifying semantic shifts in the embedding space between benign (clean) and suspect (injected) inputs. The core insight is that adversarial prompts induce subtle semantic drifts in vector representations, even when the surface text appears benign.

The ZEDD Pipeline

The system operates through three primary stages:

1. Embedding Extraction:

   • The framework utilizes fine-tuned encoder models (specifically Sentence-BERT All-MPNET-Base-V2, Llama 3 8B Instruct, Mistral 7B Instruct, and Qwen 2 7B Instruct) to generate vector representations for prompt pairs.
   • During a preliminary fine-tuning phase, the encoder learns to distinguish between "clean-clean" pairs and "injected-clean" pairs, optimizing the embedding space to better capture the differences between benign and adversarial inputs. This fine-tuning is performed once, enabling zero-shot deployment thereafter.

2. Drift Measurement:

   • Semantic drift is computed using Cosine Similarity. The drift score $D$ between a clean prompt $x$ and a candidate prompt $x'$ is defined as:
     $$Drift(x, x') = 1 - \frac{f(x) \cdot f(x')}{||f(x)|| \cdot ||f(x')||}$$
   • A higher drift score indicates a greater semantic deviation, suggesting a potential injection.

3. Flagging and Thresholding:

   • To classify prompts without labeled ground truth during inference, ZEDD employs an ensemble approach using Gaussian Mixture Modeling (GMM) and Kernel Density Estimation (KDE).
   • GMM: Fits a two-component distribution to the drift scores. The component with the lower mean corresponds to clean-clean pairs (low drift), while the higher mean corresponds to injected-clean pairs (high drift).
   • KDE Fallback: If GMM fails to converge, the system switches to KDE to identify distribution peaks and valleys.
   • Threshold Optimization: The decision threshold is calibrated via iterative binary search to maintain a strict False Positive Rate (FPR) cap of 3% while targeting an overall flagging rate of approximately 50%.

3. Key Contributions

The paper outlines three primary contributions:

1. A Zero-Shot Detection Framework: A prompt injection detection method based on embedding drift that requires no retraining of the target LLM, no internal model access, and no prior knowledge of specific attack types.
2. Statistical Flagging Mechanism: The integration of GMM and KDE to analyze embedding distributions, effectively flagging injected prompts while minimizing false positives.
3. Comprehensive Empirical Evaluation: An extensive evaluation demonstrating that embedding drift serves as a robust, transferable signal across diverse LLM architectures, outperforming traditional methods in both speed and accuracy.

4. Experimental Results

The authors evaluated ZEDD on the LLMail-Inject dataset, a re-annotated collection of 51,603 aligned prompt pairs spanning five injection categories: Jailbreak, System Leak, Task Override, Encoding Manipulation, and Prompt Confusion.

• Performance Metrics: Across tested models (Llama 3, Mistral, Qwen 2, and Sentence-BERT), ZEDD achieved:
  • Accuracy: Ranging from approximately 90.75% (Sentence-BERT) to 95.55% (Mistral 7B).
  • Precision: Exceeding 95% for most LLM-based encoders (up to 96.58% for Mistral).
  • False Positive Rate (Clean FPR): Maintained below 3% for most models (e.g., 2.2% for Qwen 2, 2.3% for Mistral), with a conservative cap ensuring low false alarms on benign inputs.
• Efficiency: The fine-tuning process for each model took approximately 15–18 minutes on an NVIDIA B200 GPU. The detection phase operates with minimal latency, making it suitable for latency-sensitive applications.
• Comparative Advantage: The results indicate that ZEDD outperforms existing models in precision and F1 scores while offering a significantly lighter computational footprint.

5. Significance and Limitations

Significance:
The paper positions ZEDD as a critical, lightweight defense layer that integrates seamlessly into existing LLM pipelines. By avoiding the need for model retraining or internal access, it addresses the scalability gap in securing LLM-powered systems against adaptive adversarial threats. The authors emphasize its utility as a "first line of defense" that balances high detection accuracy with operational efficiency.

Limitations and Future Work:
The authors acknowledge several constraints:

• Embedding Dependency: The efficacy of ZEDD is tied to the quality of the chosen embedding model; if an embedding model fails to capture the semantic nuances of a specific prompt, detection may fail.
• Scalability: While efficient, the method may require adaptive approaches to handle varying model sizes and larger datasets efficiently.
• Adversarial Evasion: There is a risk that highly sophisticated attacks could be manipulated specifically to bypass embedding-level drift detection.
• Future Directions: The authors plan to explore Few-Shot learning to improve accuracy (though this may compromise the lightweight nature), utilize adaptive approaches for resource conservation, and test the framework on diverse data formats beyond email.

In conclusion, ZEDD offers a practical, model-agnostic solution to the growing threat of prompt injections, providing a scalable mechanism to secure LLM applications with minimal engineering overhead.