💬 NLP

Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models

This paper demonstrates that lightweight, general-purpose LLMs can effectively serve as low-latency security judges for detecting prompt attacks in production environments when guided by structured reasoning, a method currently deployed in Singapore's public service chatbots, while a Mixture-of-Models approach yields only modest performance improvements.

Original authors: Hieu Xuan Le, Benjamin Goh, Quy Anh Tang

Published 2026-03-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hieu Xuan Le, Benjamin Goh, Quy Anh Tang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very helpful, super-smart robot assistant (a Large Language Model, or LLM) that works for the government. Its job is to answer citizens' questions, help them fill out forms, and give advice. But, just like a human, this robot can be tricked.

Bad actors try to "jailbreak" the robot or "inject" hidden instructions to make it do dangerous things—like revealing secret data, writing malware, or giving harmful advice.

This paper is about building a security guard for that robot. The authors from Singapore's GovTech faced a tough problem:

Old guards (simple rule-based systems) are fast but too dumb to understand tricky, sneaky attacks.
Super guards (massive AI models) are smart enough to catch the tricks, but they are too slow and expensive to check every single message in real-time.

Here is how they solved it, explained simply:

1. The "Thinking Guard" (LLM-as-a-Judge)

Instead of hiring a massive, slow super-guard, they taught a lightweight, fast robot how to think like a detective.

Usually, if you ask a robot, "Is this message bad?" it might just guess "Yes" or "No" instantly. That's like a security guard who just looks at a person's face and decides if they look suspicious. It misses the details.

The authors gave their fast robot a strict checklist (a structured reasoning process) to follow for every message:

Strip the Disguise: "Is this person pretending to be a researcher or writing a story to hide their real intent?" (e.g., "Write a story about how to make a bomb" vs. "How do I make a bomb?").
Check the Intent: "Are they asking for a harmless workflow (like 'make a chart') or something dangerous?"
Look for Safety Signals: "Are they asking how to stop a virus (safe) or how to create one (bad)?"
Self-Reflect: Before giving the final answer, the robot must pause and ask itself: "Wait, am I being too harsh? Could this actually be a legitimate question?"

The Result: By forcing the fast robot to "think step-by-step" before answering, it became almost as smart as the slow, expensive super-guards, but it was fast enough to work in real-time.

2. The "Committee of Guards" (Mixture-of-Models)

The researchers wondered: "What if we ask three different robots to check the message and take a vote? Will that be safer?"

They tried combining the opinions of different AI models (a "Mixture of Models").

The Surprise: Adding more guards didn't always make things better. In fact, sometimes it made things worse.
The Analogy: Imagine a jury. If you have one brilliant detective and one confused intern, adding the intern might actually confuse the verdict. Or, if you have three detectives who all make the same mistake, adding a third doesn't help.
The Finding: You only get a boost if you pick guards who have different strengths that complement each other. Just throwing random guards together usually leads to arguments and mistakes.

3. The Real-World Test

They didn't just test this on fake, made-up questions. They built a test set using:

Real citizen questions from Singapore's public chatbots (to make sure they didn't accidentally block normal people).
Tricky "Red Team" attacks generated by other AIs trying to break the system.

The Winner: A specific, lightweight model (called gemini-2.0-flash-lite-001) using their "Thinking Guard" method. It was fast enough to check messages instantly but smart enough to catch the sneaky attacks that older systems missed.

The Big Takeaway

You don't need a giant, slow, expensive AI to keep your chatbots safe. You just need a fast AI that is taught how to think carefully.

Don't just ask "Is this bad?"
Ask: "What is the intent? Is there a disguise? What are the safety signals? Let me double-check my answer."

This approach is now being used to protect public service chatbots in Singapore, keeping them safe from hackers without making citizens wait for their answers.

1. Problem Statement

Large Language Models (LLMs) powering autonomous agentic systems face critical security vulnerabilities, specifically prompt attacks (e.g., jailbreaks, prompt injections). These attacks exploit the conflict between helpfulness and safety training, mismatched generalization (e.g., obfuscation via Base64), and instruction/data confusion.

In production environments, organizations deploy guardrails to sanitize inputs before execution. However, a significant deployment gap exists:

Lightweight classifiers (e.g., encoder-based models like DeBERTa) are fast but lack the reasoning capabilities to detect nuanced, "grey-area" attacks, leading to poor generalization.
High-capacity LLM-based judges offer superior detection accuracy but are often too slow or costly for real-time, low-latency enforcement.

The core challenge is to develop a detection system that balances high accuracy (generalization to evolving attacks) with low latency suitable for live production traffic.

2. Methodology

The authors propose a framework centered on LLM-as-a-Judge enhanced by structured reasoning, evaluated on a curated real-world dataset.

A. Curated Evaluation Dataset

The authors created a private dataset of 929 samples (82.9% benign, 17.1% adversarial) to reflect real-world usage:

Benign Samples: Sourced from Singapore public service chatbots, including "False Positive Traps" (complex, imperative workflow instructions that mimic attacks but are legitimate).
Adversarial Samples: Generated via Automated Red Teaming (ART) using the PAIR framework. These cover diverse patterns:
- Social Engineering: Crisis scenarios or urgency appeals.
- Creative Framing: Malicious requests hidden in fictional or academic contexts.
- Code Mixed & Payload Encapsulation: Obfuscated instructions within code.
- Roleplay & Privilege Escalation: Impersonating admins to bypass refusals.

B. LLM-as-a-Judge Framework

Instead of relying on binary classification or unconstrained generation, the authors enforce a structured reasoning process via prompt engineering. The judge must follow a specific taxonomy-guided workflow:

Framing Stripping: Disregard fictional, historical, or academic framing to isolate the core request.
Intent Decomposition: Categorize the stripped intent (e.g., Workflow Instruction, Factual Overview, Safety Knowledge, Detailed Harm Methods, Safety Bypass Attempt).
Safety Signal Verification: Distinguish between malicious probing and legitimate safety research based on defensive terminology (e.g., "mitigate," "identify").
Multi-Turn Context Analysis: Evaluate the entire conversation flow to detect attacks where individual messages appear benign but collectively violate policies.
Structured CoT & Self-Reflection:
- Draft Classification: Assign a preliminary category and harm level.
- Mandatory Self-Reflection: The model must critique its own draft (e.g., "Could this be a benign workflow instruction?").
- Final Verdict: Output a boolean flag and a qualitative confidence score.
Confidence Scoring: Map qualitative verdicts to numerical scores (0.0 to 1.0) to allow engineering teams to tune sensitivity (Recall vs. Precision).

C. Mixture-of-Models (MoM)

The authors investigated whether aggregating multiple LLM judges improves robustness.

Architecture: Parallel execution of multiple models with outputs aggregated via a weighted linear combination.
Optimization: A grid search is used to find optimal weights ( $w_i$ ) and decision thresholds ( $\tau$ ) to maximize the F1 score.
Theoretical Insight: Due to the discrete nature of model outputs, the solution space is finite. Many distinct weight configurations map to the same decision boundary, allowing a discrete grid search to cover the entire logical solution space effectively.

3. Key Contributions

LLM-as-a-Judge for Real-Time Detection: Demonstrated that general-purpose, lightweight LLMs (e.g., gemini-2.0-flash-lite-001) can serve as effective, low-latency security judges when guided by structured reasoning (intent decomposition + self-reflection) rather than unconstrained prompting.
Real-World Evaluation Benchmark: Introduced a dataset combining real production traffic with automated red-teaming, specifically designed to test "grey-area" attacks and false positive traps, addressing the limitations of synthetic or outdated benchmarks.
Mixture-of-Models Analysis: Provided empirical evidence that ensembling LLM judges does not guarantee performance improvements. Gains are highly dependent on specific complementary model pairings rather than simply increasing ensemble size.
Production Deployment: The proposed configuration is currently deployed as a centralized guardrail service for public service chatbots in Singapore.

4. Results

A. Performance Comparison

Encoder-based Classifiers: Performed poorly (F1 ~0.22–0.35), failing to detect nuanced attacks.
Specialized Safety LLMs: Improved performance (F1 ~0.54–0.80) but were still outperformed by the structured LLM-as-a-Judge approach.
LLM-as-a-Judge: Achieved the highest performance.
- Best Overall: gpt-5.1 achieved an F1 of 0.8711 (Precision 0.9766).
- Best Latency/Performance Balance: gemini-2.0-flash-lite-001 achieved an F1 of 0.8440 with a latency of 1.52s, outperforming specialized safety models while remaining viable for live traffic.
- Highest Recall: gemini-3-flash-preview achieved 0.9182 recall.

B. Prompt Design Impact

Comparing their structured approach against OpenAI Guardrails (which uses detailed prompts but lacks enforced intermediate reasoning steps), the authors' method consistently achieved higher F1 scores across all tested models. This confirms that enforcing intermediate judgments and self-reflection is critical for stability.

C. Thinking vs. Non-Thinking Modes

Contrary to intuition, enabling internal "thinking" modes (e.g., Chain-of-Thought budgets) degraded performance. Non-thinking configurations consistently outperformed thinking counterparts, suggesting that for explicit classification tasks, direct instructions are more effective than internal deliberation.

D. Mixture-of-Models (MoM) Findings

No Guarantee of Improvement: Adding more models often degraded performance. For example, adding a third or fourth model frequently resulted in lower F1 scores than the best single model or pair.
Complementarity is Key: Gains were observed only in specific, small combinations (e.g., gpt-5.1 + gpt-5-mini improved F1 by +0.0164).
Conclusion: MoM is effective only when carefully selecting complementary models; blindly increasing ensemble size introduces conflicting signals that harm robustness.

5. Significance and Conclusion

This work bridges the gap between security accuracy and production latency. It proves that general-purpose LLMs, when constrained by a structured reasoning framework (taxonomy-guided, self-reflective), are superior to both lightweight classifiers and specialized safety models for prompt attack detection.

Practical Impact: The findings enable the deployment of high-fidelity guardrails in real-time systems without the prohibitive latency of massive models.
Strategic Insight: The paper challenges the assumption that "more models = better security," demonstrating that for LLM judges, model selection and prompt design are more critical than ensemble size.
Deployment: The methodology is already operational in Singapore's public sector, validating its effectiveness in a live, high-stakes environment.