Analysis of LLMs Against Prompt Injection and Jailbreak Attacks

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed in resource-constrained environments (e.g., edge devices, academic institutions, small enterprises) using open-source models. While prompt engineering allows these organizations to adopt LLMs efficiently, these models remain highly vulnerable to prompt injection and jailbreak attacks.

The core problem addressed is the lack of robust security for small-to-medium open-source LLMs (1B–7B parameters). Unlike large proprietary models, these smaller models often cannot afford the computational cost of retraining or fine-tuning for security. Consequently, there is a critical need to evaluate their inherent vulnerabilities and assess the efficacy of lightweight, inference-time defense mechanisms that operate without modifying model weights.

2. Methodology

2.1 Models Evaluated

The study empirically tested 10 variants across 6 open-source model families, ranging from 1B to 7B parameters:

• Llama 3.2: 1B, 3B
• Mistral: 7B
• Phi-3: 3.8B
• Qwen 3: 1.7B, 4B
• Gemma 3: 1B, 4B
• DeepSeek-R1: 1.5B, 7B

2.2 Attack Dataset Construction

The authors curated a diverse, manually constructed dataset combining real-world adversarial prompts from:

• Online communities (Reddit, Discord).
• Open-source repositories (GitHub).
• Academic benchmarks (HarmBench, JailbreakBench).
• Manually crafted variants.

The dataset included 94 prompt injection scenarios and 73 jailbreak scenarios (expanded to 74 per model in the defense phase), categorized into:

1. Question Answering: Hidden adversarial intent in innocent queries.
2. Basic Compliance: Testing adherence to explicit directions.
3. Instruction Override: Explicitly asking the model to ignore rules.
4. Role-playing Jailbreaks: Fictional personas to relax safety.
5. Long-format / Multi-step Prompts: Extended reasoning chains to escalate malicious intent.

2.3 Evaluation Framework

• Baseline Assessment: Models were tested without defenses to establish vulnerability rates.
• Defense Assessment: Five lightweight, inference-time defenses were applied.
• Judgment Mechanism: Responses were evaluated using a hybrid approach:
  • LLM-based Judge: A separate LLM (DeepSeek-R1) acted as a safety judge to classify outputs as "Vulnerable" or "Non-Vulnerable" based on semantic compliance with adversarial intent.
  • Human Verification: A subset of judgments was manually cross-validated to ensure accuracy and refine the judge's criteria.
• Failure Modes: The study specifically tracked "Silent Non-Responsiveness" (models refusing to output anything due to internal gating) as a distinct failure mode separate from successful jailbreaks.

2.4 Defense Mechanisms Tested

Five inference-time strategies were evaluated:

1. Input Filtering: Keyword blacklisting and regex matching (Pre-processing).
2. Self-Examination: A secondary model (or the same model) reviews the output before release (Post-processing).
3. System Prompt Defense: Hardening system instructions to enforce hierarchy and refusal templates.
4. Vector Defense: Embedding-based similarity matching against known attack patterns.
5. Voting Defense: Generating multiple responses and selecting the safest based on consensus.

3. Key Contributions

1. Comprehensive Empirical Evaluation: A large-scale study of 10 open-source LLMs against 94 prompt injection and 73 jailbreak scenarios.
2. Defense Benchmarking: Systematic assessment of five inference-time defenses, providing a ranking of their efficacy without retraining.
3. Identification of Silent Failures: Highlighting "Silent Non-Responsiveness" as a critical, underexplored failure mode where models block output entirely, degrading usability and debuggability.
4. Dataset Release: Public release of a curated adversarial prompt dataset to support reproducible research in LLM security.
5. Insight on Model Size: Demonstrating that safety robustness is non-monotonic with model size; larger models are not inherently safer if alignment strategies differ.

4. Key Results

4.1 Baseline Vulnerability (Without Defenses)

• Highly Vulnerable: Small models like Qwen 1.7B (71.3% vulnerable) and Gemma 1B (62.8% vulnerable) failed systematically.
• Moderately Vulnerable: DeepSeek-R1 1.5B (34%) and Llama 3.2 1B (30.9%) showed mixed results.
• Robust Models: Phi-3 (3.8B), Mistral (7B), DeepSeek-R1 (7B), Llama 3.2 (3B), Qwen 4B, and Gemma 4B achieved 0% vulnerability in the prompt injection test.
• Silent Failures: Some "0% vulnerability" models (e.g., Phi-3, Mistral) frequently returned empty responses or no refusal messages, indicating aggressive internal safety gating rather than true alignment.

4.2 Jailbreak Performance

• Gemma 3 (1B) was the most vulnerable, failing ~67% of jailbreak prompts.
• DeepSeek-R1 (1.5B) was the most robust among smaller models (only ~1.4% vulnerability).
• Attack Effectiveness: "Horselock" style multi-role and override prompts were the most effective, while some popular Reddit jailbreaks were found to be outdated.

4.3 Defense Effectiveness

• Self-Examination (Self-Defence): Consistently the most effective defense, reducing vulnerability by >50% compared to input filtering. It achieved 0% vulnerability for Qwen 4B and DeepSeek-R1 1.5B.
• Input Filtering: The least effective defense. It often failed against long, reasoning-heavy prompts and, in some cases, increased vulnerability compared to the baseline due to false negatives.
• System Prompt & Vector Defenses: Provided moderate improvements but were inconsistent across model sizes.
• Voting Defense: Improved robustness over input filtering but failed to outperform Self-Examination and incurred high computational overhead.

4.4 Critical Observations

• Model Size vs. Safety: Safety does not scale linearly with parameters. A 4B model (Qwen) was perfectly safe, while its 1.7B variant was highly vulnerable. Alignment strategy and refusal design are more critical than size.
• Long-Form Attacks: All defenses struggled significantly against long, context-rich, multi-step reasoning prompts.
• Intrinsic vs. Extrinsic: Intrinsic refusal mechanisms (Self-Examination) proved superior to external surface-level filtering.

5. Significance and Conclusion

This paper provides critical evidence that resource-constrained organizations cannot rely solely on external filtering (like keyword blocking) to secure open-source LLMs. The study concludes that:

1. Inference-time defenses are necessary but insufficient against sophisticated, long-form jailbreaks.
2. Self-Examination is currently the most viable lightweight defense strategy.
3. Silent non-responsiveness is a major usability risk that must be addressed in safety design.
4. Future security research must focus on integrating safety reasoning directly into model architectures rather than relying on post-hoc filtering, as external defenses are easily bypassed by advanced adversarial prompts.

The work underscores the urgent need for better alignment strategies in smaller models and the development of robust, intrinsic safety mechanisms that do not require expensive retraining.