Technical Summary: PromptSentinel – Distribution Mismatch in Prompt Safety Classification

Problem Statement

The deployment of Large Language Models (LLMs) in consumer applications necessitates automated prompt safety classifiers to detect harmful or policy-violating requests. While existing classifiers often report high performance on synthetic benchmarks, this paper investigates whether these metrics generalize to human-authored inputs. The authors posit a distribution mismatch hypothesis: current benchmarks, often constructed using adversarial transformations and curated attack templates, fail to capture the diversity of natural human interaction (e.g., roleplay, creative writing, conversational requests). Consequently, performance measured on synthetic data may substantially overestimate real-world reliability, particularly regarding the "safe" class of prompts.

Methodology

The study, titled PromptSentinel, employs a controlled experimental design to isolate the effects of data distribution on classifier performance.

Dataset Construction

The authors constructed a training corpus of 110,060 prompts by merging four public datasets to ensure source diversity and class balance:

1. TrustAIRLab In-the-Wild Jailbreak Prompt: Used exclusively as a held-out test set (6,387 prompts, including 666 human-written jailbreaks).
2. ToxicChat: 4,981 real user conversations (Vicuna chatbot) annotated for toxicity.
3. Qualifire Prompt Injection Benchmark: A near-balanced dataset addressing class imbalance.
4. WildJailbreak: The largest public jailbreak dataset, providing both synthetic (automated transformations) and human-written variants.

The final training corpus was balanced (54.3% safe, 45.7% unsafe) through source diversity rather than oversampling.

Experimental Design

The authors evaluated a TF-IDF vectorization combined with a Logistic Regression classifier (with balanced class weights) across four research questions (RQs):

• RQ1 (Attack Type Generalisation): Tested whether training on direct harmful prompts generalizes to adversarial (jailbreak-style) prompts and vice versa, using subsets of the WildJailbreak dataset.
• RQ2 (Human-Written Jailbreak Detection): Evaluated the classifier trained on the diverse corpus against the held-out TrustAIRLab test set. This included stratification by prompt length and a comparison between TF-IDF and MiniLM sentence embeddings (both full-prompt and chunked).
• RQ3 (LLM-as-Judge): Investigated whether a zero-shot LLM judge (Llama-3.1-8B) could correct classifier errors (false positives) without sacrificing the detection of true positives.
• RQ4 (Four-Bucket Distribution Analysis): Isolated the effect of prompt origin by evaluating performance across four distinct buckets:
  • Synthetic Safe (WildJailbreak benign)
  • Synthetic Unsafe (WildJailbreak adversarial)
  • Human Safe (WildChat)
  • Human Unsafe (TrustAIRLab jailbreaks)

Key Results

1. Significant Performance Degradation on Human Data

The most critical finding is the substantial gap between synthetic and human-authored data:

• False Positive Rate (FPR): On synthetic benign prompts, the FPR was 1.0%. On human-authored safe prompts, the FPR surged to 32.0%.
• Recall (Harmful Detection): On synthetic adversarial prompts, recall was 98.0%. On human-written jailbreaks, recall dropped to 65.4%.

2. Attack Type vs. Source Origin

Within the synthetic WildJailbreak dataset, the classifier performed consistently well regardless of whether it was trained on direct or adversarial prompts (Recall ~0.978–0.981). This suggests that the performance gap is not driven by the specific type of attack (direct vs. adversarial) but rather by the source of the data (synthetic vs. human).

3. Limitations of Model Complexity

• Embeddings: Replacing TF-IDF with MiniLM sentence embeddings did not resolve the gap. In fact, full-prompt embeddings performed worse (Recall 0.580, Precision 0.105). Chunked embeddings improved recall on long prompts but caused catastrophic false positive rates (up to 97.0%) on long benign prompts.
• Prompt Length: Contrary to the expectation that short prompts are harder to detect, recall actually increased for longer prompts (reaching ~0.72 for >200 words), suggesting length alone does not explain the degradation.

4. LLM-as-Judge Trade-offs

Applying an LLM judge as a second-pass filter successfully recovered 94.6% of false positives (improving precision from 0.286 to 0.640). However, this came at a severe cost: the system retained only 24.0% of true positives (recall dropped from 1.000 to 0.240). Qualitative analysis suggested that many missed jailbreaks relied on "persona-override" framing, which the judge struggled to distinguish from legitimate roleplay.

Significance and Claims

The paper argues that distribution mismatch, specifically the lack of diversity in the "safe" class of training data, is a primary contributor to classifier failure in real-world scenarios.

• Benchmark Validity: The authors claim that current synthetic benchmarks may provide an incomplete and overly optimistic measure of deployable performance. High recall on synthetic data does not guarantee low false positive rates on human inputs.
• Architectural Limits: The study demonstrates that increasing model complexity (e.g., moving from TF-IDF to embeddings) or adding LLM judges is insufficient to eliminate the performance gap. The issue appears rooted in the data distribution rather than the model architecture.
• Future Direction: The paper concludes that prompt safety benchmarks must place greater emphasis on diverse human-authored benign prompts. The authors suggest that future work should directly manipulate safe-class diversity during training to establish a causal link between data composition and real-world robustness.

The authors remain modest in their claims, noting that the distribution mismatch hypothesis is currently correlational. They acknowledge that factors beyond safe-class diversity (such as linguistic style, conversational structure, and topic distribution) may also contribute to the observed gap, necessitating further controlled interventions.