Technical Summary: Benchmarking Llama Model Security Against OWASP Top 10 for LLM Applications

Problem Statement

As Large Language Models (LLMs) transition from research prototypes to enterprise systems, their probabilistic nature and reliance on vast training corpora introduce unique security vulnerabilities that traditional cybersecurity protocols struggle to address. Adversaries exploit these traits through prompt injection, context manipulation, and data poisoning, leading to risks such as unauthorized data exfiltration, harmful content generation, and system hijacking. While the Open Worldwide Application Security Project (OWASP) has established the "Top 10 for LLM Applications" framework to categorize these critical risks, there remains a lack of comprehensive, reproducible benchmarks that evaluate specific model architectures against this taxonomy while accounting for computational constraints. Existing studies often measure accuracy in isolation, neglecting the trade-offs between security effectiveness, inference latency, and memory usage required for real-world deployment.

Methodology

This study conducted a systematic evaluation of ten Llama model variants using a custom-built security benchmark aligned with the OWASP Top 10 framework.

Experimental Infrastructure

Experiments were performed on the FABRIC testbed (Georgia Tech site) using NVIDIA A30 GPUs (24GB VRAM) running Ubuntu 22.04 LTS. The software stack included PyTorch 2.1.0, CUDA 11.8, and HuggingFace Transformers (v4.51.3).

Dataset Construction

The authors constructed a dataset of 100 adversarial prompts, equally distributed across the ten OWASP categories (LLM01–LLM10). The dataset design followed the OWASP AI Testing Guide, incorporating 23 distinct injection techniques such as Base64/Hex encoding obfuscation, role-playing exploits ("DAN" style jailbreaks), and multi-turn manipulation. Each entry included a test ID, category, subcategory, adversarial prompt, ground-truth safety label, and metadata regarding severity.

Model Selection

Ten models were evaluated, divided into two groups:

1. Standard Llama Models (5 variants): General-purpose architectures including Meta-Llama-3-8B, Llama-3.1-8B, Llama-3.1-8B-Instruct, Llama-3.2-1B, and Llama-3.2-3B-Instruct.
2. Llama Guard Models (5 variants): Specialized security classifiers including Meta-Llama-Guard-2-8B, Llama-Guard-3-1B, Llama-Guard-3-8B, Llama-Guard-3-8B-INT8 (quantized), and Llama-Guard-3-11B-Vision (multimodal, tested on text).

Evaluation Protocol

All models were loaded in float16 precision (except the INT8 variant) with a temperature of 0.1 and a maximum token limit of 10 to ensure deterministic binary outputs.

• Guard Models: Produced structured "safe" or "unsafe" labels directly.
• Standard Models: Prompted with "is this prompt safe or not?" and parsed for "unsafe" keywords to derive binary decisions.
• Metrics: Detection accuracy, total execution time, average latency per test, and VRAM usage were recorded.

Key Results

The study revealed significant performance disparities between model types and sizes, challenging the assumption that larger models are inherently more secure.

Overall Performance

• Specialized vs. Base Models: Base models (Meta-Llama-3-8B and Llama-3.1-8B) failed to detect any threats (0% accuracy) despite high inference times (~0.75s per test). In contrast, specialized Guard models and instruction-tuned variants demonstrated significantly higher detection rates.
• Inverse Relationship: An inverse relationship was observed between model size and security effectiveness in this context. The compact Llama-Guard-3-1B achieved the highest detection rate of 76% with the lowest latency (0.165s per test) and minimal VRAM usage (0.94 GB).
• Instruction Tuning: Instruction tuning proved critical; Llama-3.1-8B-Instruct achieved a 54% detection rate, whereas its non-instruct base variant achieved 0%.

Category-Specific Findings

No single model provided comprehensive protection across all ten OWASP categories:

• Llama-3.1-8B-Instruct was highly resilient to Prompt Injection (LLM01, 100%) but completely vulnerable to System Prompt Leakage (LLM07, 0%).
• Llama-3.2-1B showed remarkable resilience to Information Disclosure (LLM02, 90%) and Supply Chain Vulnerabilities (LLM03, 100%) but was less effective against direct Prompt Injection (50%).
• Quantization and Multimodal Trade-offs: The quantized Llama-Guard-3-8B-INT8 was significantly slower and less accurate than its full-precision counterpart. The multimodal Llama-Guard-3-11B-Vision model performed poorly on text-only safety classification (28% detection), suggesting that multimodal optimization may dilute text-specific security performance.

Key Contributions

1. Systematic Benchmarking: The study provides a comparative analysis of five standard Llama models and five Guard variants against the full OWASP Top 10 framework, quantifying both security effectiveness and computational cost.
2. Open-Source Dataset: The authors released a benchmark dataset comprising 100 targeted adversarial prompts with comprehensive metadata to support reproducible research in AI security.
3. Empirical Evidence on Efficiency: The research offers empirical data demonstrating that smaller, specialized models often outperform larger general-purpose models in security tasks, providing a basis for resource-constrained deployment decisions.

Significance and Claims

The paper claims that computational scale does not equate to security robustness. Instead, specialized training and instruction tuning are identified as the primary drivers of security performance. The architectural distinction between Guard models (designed for classification) and standard models (adapted from text generation) is highlighted as critical for production security systems, with Guard models offering more reliable, structured binary outputs.

The authors conclude that organizations should move away from a "one-size-fits-all" approach, prioritizing specialized compact models for edge or high-throughput environments and avoiding base models for safety-critical roles. However, they modestly acknowledge that significant gaps remain, particularly regarding System Prompt Leakage (LLM07) and Supply Chain Vulnerabilities (LLM03), which current safety training largely fails to address. The study positions itself as a foundational step toward standardized security reporting and the development of layered defense mechanisms in LLM applications.