💬 NLP

Do Encoders Suffice? A Systematic Comparison of Encoder and Decoder Safety Judges for LLM Adversarial Evaluation

This paper systematically evaluates whether fine-tuned ModernBERT encoder classifiers can serve as cost- and latency-efficient alternatives to expensive LLM-based judges for identifying harmful LLM outputs, demonstrating their reliability across various adversarial attack techniques and benchmark datasets.

Original authors: Han Jeon, Shiv Medler, Joseph Voyles, Matt Wood

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Han Jeon, Shiv Medler, Joseph Voyles, Matt Wood

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you run a massive, bustling library where people ask a robot librarian (the AI) for help. Sometimes, people try to trick the robot into telling them how to build a bomb or write a mean letter. You need a "Safety Guard" to stand at the door, read what the robot says, and stop any dangerous answers before they reach the user.

For a long time, the best Safety Guards were Super-Intelligent Librarians (Decoder-based LLMs like LlamaGuard or Claude). They are incredibly smart and can understand complex tricks, but they are slow, expensive to hire, and get tired easily.

This paper asks a simple question: Can we replace these expensive Super-Librarians with a faster, cheaper "Security Camera" (Encoder-based classifiers) that just spots danger without needing to be a genius?

Here is the breakdown of their experiment and findings:

1. The Experiment: The "Security Camera" vs. The "Super-Librarian"

The researchers built a new type of Safety Guard called Ettin. Think of Ettin as a high-tech security camera trained to instantly recognize a "bad" face.

The Training: They didn't just teach Ettin one rule. They showed it the opinions of seven different Super-Librarians. If most of them agreed a sentence was dangerous, Ettin learned to flag it.
The Test: They put Ettin to the test against a "Golden Standard" of known bad questions (from JailbreakBench and AILuminate) that it had never seen before. They also tested it against the original Super-Librarians to see who caught more bad stuff.

2. The Results: Speed vs. Smarts

The Good News (Speed and Cost):
The Security Camera (Ettin) is a speed demon.

Analogy: If the Super-Librarian takes 700 milliseconds (about the time it takes to blink) to read a sentence and decide if it's safe, the Security Camera does it in under 4 milliseconds.
Throughput: In a real-world test, the Security Camera could process 187 times more conversations per second than the Super-Librarian. It's like the difference between a snail and a race car.
Cost: Because it's so fast and runs on cheaper hardware, it costs a tiny fraction of the money to use.

The Mixed News (Accuracy):

The "Balanced" Camera (Ettin-150): This model is a good all-rounder. It catches about half the bad stuff and misses the rest, but it's very fast.
The "Paranoid" Camera (Ettin-400): This bigger model is much better at catching bad stuff (it misses very little), but it sometimes gets scared and flags safe things as dangerous too (lower precision).
The Comparison: The Super-Librarians were still the smartest overall, but the Security Cameras came surprisingly close, especially when the "bad guys" used simple tricks.

3. The Weakness: The "Puzzle" Problem

The paper found a specific place where the Security Camera struggles: Multi-turn attacks.

The Analogy: Imagine a criminal trying to sneak a bomb into the library.
- Simple Attack: They walk in holding a bomb. The Security Camera sees the bomb immediately.
- Complex Attack (Decomposition): The criminal breaks the bomb into 10 harmless-looking pieces (a screw here, a wire there) and asks the librarian for help with each piece separately. Only when you put all 10 pieces together does it look like a bomb.
The Issue: The Security Camera (Ettin) only looks at the final answer the robot gives. It doesn't see the 10 previous harmless questions the criminal asked to build the context. Because it can't see the whole "puzzle" being assembled over time, it often misses these complex, multi-step tricks. The Super-Librarians, who can read the whole conversation history, are much better at spotting these.

4. The Verdict: Do Encoders Suffice?

The paper concludes that yes, but with a caveat.

Think of the Security Camera (Encoder) as a first line of defense.

Use it for: The 90% of conversations that are simple, direct, or clearly safe/dangerous. It's cheap, fast, and handles the heavy lifting.
Don't use it for: The tricky, complex, multi-step "jailbreak" attempts where the danger is hidden in the conversation history. For those, you still need the expensive, slow Super-Librarian (Decoder) to do a deep dive.

In short: You don't need to hire a Super-Librarian for every single question. You can use a fast, cheap Security Camera to filter out the obvious stuff, and only call in the expensive expert when the situation gets complicated. This saves a massive amount of money and time while keeping people safe.

Technical Summary: Do Encoders Suffice? A Systematic Comparison of Encoder and Decoder Safety Judges for LLM Adversarial Evaluation

Problem Statement

As Large Language Models (LLMs) are increasingly deployed in critical, customer-facing applications (e.g., banking, airline services), the need for effective, low-latency, and cost-efficient safety guardrails has become paramount. Current safety evaluation paradigms rely heavily on decoder-based safety judges, including large autoregressive models used as "LLM-as-a-judge" prompts or fine-tuned decoder classifiers like LlamaGuard. While effective, these approaches suffer from high inference costs, significant latency, and potential performance degradation under distribution shifts or novel attack patterns. Furthermore, decoders process input sequentially (left-to-right), which may limit their ability to capture harmful intent across a full context compared to bidirectional architectures.

This paper investigates whether fine-tuned modern encoder classifiers (specifically from the ModernBERT family, such as Ettin) can serve as a reliable, cost-effective alternative to decoder-based judges for identifying harmful LLM outputs in adversarial scenarios.

Methodology

Dataset Construction and Partitioning

The study utilized a training corpus aggregating 14 source datasets, comprising 9,968 source prompts and 563,216 (prompt, response) pairs. Sources included standardized benchmarks (HarmBench, SafetyBench), real-world interactions (ToxicChat, WildJailbreak), and synthetic domain-specific data (medical, financial).

Leakage Prevention: To prevent data leakage where a model's response to a specific question in training could influence its performance on the same question in testing, the authors employed a question-group partitioning strategy. All rows sharing the same question text were assigned to the same partition (training or test) regardless of the target model or attack technique used.
Holdout Set: A strict out-of-distribution (OOD) holdout set of 42,899 examples was constructed from JailbreakBench and AILuminate, evaluated across four frontier models (Claude Sonnet 4.5, Gemini 3.1 Pro, GPT-5.4, GPT-OSS 120B) and four attack techniques.

Safety Judge Panel and Label Aggregation

To generate training labels, the authors assembled a panel of seven safety judges across three categories:

Open-weight classifiers: LlamaGuard 3 and 4.
Proprietary LLMs: Claude Judge, SorryBench, JailbreakBench Judge, AILuminate Judge.
Rubric-based scorer: StrongReject.

A majority-voting strategy was used to aggregate binary labels (harmful vs. safe). To address low inter-judge agreement (Fleiss' $\kappa \approx 0.23$ on the full dataset), a High-Confidence Filtering strategy was applied, retaining only unanimously safe rows and strong-majority harmful rows ( $\ge$ 5 of 7 votes). This reduced the dataset to 307K rows but increased agreement to $\kappa \approx 0.805$ .

Attack Techniques

The evaluation covered four adversarial prompting techniques:

Single-turn: Direct adversarial prompts.
Decomposition: Fragmenting a harmful request into benign sub-queries.
Escalation: Incrementally increasing request sensitivity over a conversation.
Context Manipulation: Embedding harmful requests within benign framing (e.g., academic or fictional narratives).

Models Evaluated

Encoder Classifiers: ModernBERT-family models (Ettin-150M and Ettin-400M), fine-tuned on the aggregated judge labels.
Baselines: LlamaGuard 3/4, Claude-as-a-judge, StrongReject, SorryBench, JailbreakBench, and AILuminate.
Target Models: 16 LLMs from six providers (OpenAI, Anthropic, Meta, Mistral, Google, Amazon) spanning legacy and modern cohorts.

Key Contributions

Systematic Comparison: The first large-scale benchmark comparing modern encoder classifiers against both LLM-as-a-judge systems and fine-tuned decoder-based safety models on adversarial safety benchmarks using ground-truth holdout labels.
Cost-Latency-Performance Analysis: A production-level quantification of the trade-offs, demonstrating that encoders offer significant efficiency gains over decoder-based alternatives.
Fine-Grained Attack Breakdown: An analysis of classifier performance across specific attack techniques (single-turn, decomposition, escalation, context manipulation) to identify alignment and divergence points between encoder and decoder judges.

Results

Performance on OOD Holdout

Encoder Advantages: Encoder classifiers demonstrated more stable calibration than decoder judges. LlamaGuard-4 under-predicted harm (FNR = 0.658), while judges like Claude and StrongReject over-predicted (harmful rates >25%).
Ettin Models:
- Ettin-150M: Achieved a balanced profile (F1 = 0.547, FNR = 0.422).
- Ettin-400M: Operated as a high-recall detector (Recall = 0.854, FNR = 0.146) with a higher false positive rate.
Impact of Label Noise: On a high-confidence subset of the holdout (where judges agreed strongly), Ettin-400M achieved an F1 of 0.894, suggesting that annotation noise, rather than model architecture, is a primary limiting factor for encoder performance.

Failure Modes

Multi-Turn Attacks: Encoders struggled most with decomposition attacks, where harmful intent is distributed across intermediate steps. Since encoders typically evaluate the final (question, response) pair without explicit access to the reasoning path of intermediate turns, they missed distributed harm patterns.
Model Scaling: Increasing model size from 150M to 400M parameters significantly mitigated this issue, reducing the false negative rate for decomposition attacks from 50.6% to 11.6%.

Cost and Latency

Throughput: On an Azure ML instance (8x V100 GPUs), Ettin-150 achieved 244.8 completions per second (CPS) at batch size 8, compared to 1.31 CPS for LlamaGuard-4-12B (an 187x improvement).
Latency: Median latency dropped from ~718 ms (LlamaGuard) to 3.6 ms (Ettin-150).
Cost Efficiency: Ettin-150 achieved 11.1 CPS per dollar per hour, compared to 0.06 for LlamaGuard-4-12B. Encoders could also run on low-cost CPU instances, further reducing deployment costs.

Significance and Claims

The paper concludes that fine-tuned encoder classifiers can serve as a cost and latency-efficient alternative to LLM-based evaluation, particularly as a first-line filter for safety classification. While they may not fully replace decoder-based judges for complex, multi-turn jailbreaks requiring deep reasoning over conversation history, they offer a practical solution for high-volume, real-time safety filtering.

The authors modestly claim that their findings offer guidance on when encoders suffice, noting that the low inter-judge agreement in training data ( $\kappa \approx 0.23$ ) likely constrained the true performance ceiling of the classifiers. Future work is suggested to test on human-curated datasets to further validate performance. The study does not claim encoders are universally superior but highlights their viability for specific production constraints where latency and cost are critical.