A Multi-Agent Framework for Explainable Prompt Injection Detection in Large Language Models

Technical Summary: A Multi-Agent Framework for Explainable Prompt Injection Detection in Large Language Models

Problem Statement
Large Language Models (LLMs) have become integral to conversational agents, decision-support systems, and software development. However, their widespread deployment has introduced significant security vulnerabilities, most notably prompt injection attacks. In these attacks, malicious actors craft specific inputs to override system instructions, bypass safety measures, leak confidential information, or force the generation of harmful content.

Existing defense mechanisms primarily rely on rule-based filtering, keyword matching, and traditional machine learning. The paper identifies three critical limitations in current approaches:

1. Lack of Adaptability: Static rule-based systems fail to detect advanced attacks involving paraphrasing, semantic manipulation, or novel jailbreak techniques.
2. Black-Box Nature: Modern deep learning models often lack interpretability, providing no insight into why a specific prompt was flagged, which hinders security auditing and user trust.
3. Insufficient Generalization: Current solutions struggle to detect unseen or zero-day prompt injection attacks that do not fit pre-defined categories.

Methodology: The PromptShield Framework
To address these gaps, the authors propose PromptShield, an interpretable framework that combines transformer-based semantic representation with Explainable Artificial Intelligence (XAI). The framework operates through five distinct stages:

1. Data Preprocessing: Raw prompts undergo normalization, including lowercase conversion, removal of redundant symbols, tokenization, padding/truncation (max 256 tokens), and labeling into five classes: Benign, Prompt Injection, Jailbreak, Role Manipulation, and Information Extraction.
2. Transformer-Based Feature Extraction: The core of the framework utilizes a transformer encoder (similar to BERT/RoBERTa architectures) to convert input tokens into vector representations. Through self-attention mechanisms, the model learns contextual relationships and semantic dependencies within the prompt, rather than relying solely on isolated keywords.
3. Pattern Recognition and Classification: A dedicated Pattern Recognition Module analyzes the extracted features to identify specific adversarial patterns, such as instruction overrides, prompt leakage attempts, jailbreak indicators, and role escalation. A fully connected neural network then classifies the prompt based on these learned representations.
4. Scoring Function: The framework computes a composite security score $PS(x)$ for each input, defined as a weighted sum of the attack pattern score, context manipulation score, and risk score.
5. Explainability Module: To ensure transparency, the framework integrates SHAP (SHapley Additive exPlanations) for global feature importance and LIME (Local Interpretable Model-agnostic Explanations) for local instance-level explanations. This allows security analysts to visualize which specific tokens (e.g., "Ignore," "Reveal," "System Prompt") contributed most to a malicious classification.

Key Contributions
The paper outlines five primary contributions:

• Transformer-Based Detection: Development of a pattern recognition approach specifically designed to capture semantic and contextual nuances of prompt injection attacks.
• XAI Integration: The incorporation of SHAP and LIME to transform the detection process from a black-box operation into a transparent, auditable decision-making process.
• Broad Attack Coverage: The ability to detect both known and unseen attack vectors, including jailbreaks, role manipulation, and data extraction attempts.
• Comprehensive Benchmarking: A rigorous evaluation against traditional machine learning (Logistic Regression, Random Forest, XGBoost) and deep learning baselines (LSTM, BERT, RoBERTa).
• Unified Security Framework: A practical solution that bridges the gap between high-accuracy detection and the need for explainability in security-critical applications.

Experimental Results
The framework was evaluated on a dataset of 30,000 samples derived from public benchmarks (JailbreakBench, AdvBench, PromptInject) and manually generated samples. The results demonstrate that PromptShield outperforms all baseline models across standard classification metrics:

• Accuracy: PromptShield achieved 98.1%, surpassing RoBERTa (96.8%) and BERT (96.2%).
• Precision, Recall, and F1-Score: The model achieved scores of 97.9%, 97.6%, and 97.7% respectively, indicating a low rate of false positives and false negatives.
• ROC-AUC: The framework attained a score of 0.99, confirming excellent discrimination capability between benign and malicious prompts.
• Ablation Study: The study confirmed that adding the pattern recognition layer and XAI components to the base transformer model significantly improved performance over using the transformer alone.

Significance and Claims
The paper claims that PromptShield represents a significant advancement in LLM security by addressing the dual challenges of detection accuracy and interpretability. Unlike previous works that focus solely on attack taxonomy or static defenses, this framework offers a unified approach that:

• Provides actionable explanations for security analysts, enabling them to understand the logic behind detection decisions.
• Demonstrates robustness against evolving adversarial strategies, including instruction overriding and role escalation.
• Offers a scalable solution suitable for deployment in security-sensitive domains such as healthcare and enterprise systems.

The authors conclude that while the framework shows promising results, future work will focus on extending detection to multilingual prompts, developing real-time pipelines, and investigating federated learning for privacy-preserving detection.