Technical Summary: Opir – Efficient Multi-Task Safety Classification

Problem Statement

The deployment of Large Language Models (LLMs) in chat, agentic, and middleware environments necessitates real-time safety filtering to detect unsafe prompts, toxic language, jailbreak attempts, and harmful responses. Current state-of-the-art guardrail systems (e.g., Llama Guard, WildGuard, ShieldGemma, PolyGuard) predominantly rely on large autoregressive decoder models (7B–22B parameters). While effective, these models incur significant computational costs and latency, as every guarded interaction requires one or more additional LLM forward passes. Furthermore, many existing systems prioritize binary safe/unsafe decisions, lacking the granularity required for real-world categorization across toxicity, jailbreaks, and specific harmful content domains. There is a critical need for classifiers that can distinguish benign sensitive text from covert harmful content while operating with a substantially smaller deployment footprint and lower latency.

Methodology

Opir addresses these challenges by introducing a family of encoder-based guardrail models built on the GLiClass architecture. Unlike decoder-based approaches that generate structured verdicts autoregressively, Opir jointly encodes input text and candidate labels using a bidirectional encoder. This enables zero-shot multi-label classification at a fraction of the cost of generative models.

Core Components

1. Architecture: Opir utilizes GLiClass, which extends the GLiNER architecture for sequence classification. It employs a uni-encoder with configurable pooling (average, first-token, etc.) and scoring mechanisms (dot-product, bilinear, or learned heads).

   • Multi-Task Variants: Built on DeBERTaV3-large and mDeBERTaV3-base for comprehensive safety, toxicity, jailbreak, and zero-shot categorization.
   • Edge Variants: Built on compact encoders (Ettin-encoder-32m and mmBERT-small) with fewer than 100M parameters, optimized for binary safe/unsafe classification on edge devices.

2. Safety Taxonomy: The models are trained on a three-level taxonomy comprising 996 labels:

   • Level 1: 16 top-level categories (e.g., toxicity, violence, self-harm, AI system security).
   • Level 2: 126 mid-level categories.
   • Level 3: 854 leaf labels.
   • Crucially, the taxonomy includes benign and safety-preserving categories (e.g., counterspeech, defensive cybersecurity, appropriate refusal) to mitigate the "over-refusal" failure mode common in strict policy-prompted guardrails.

3. Data Construction: The training dataset is a synthesis of:

   • Taxonomy-grounded generation: 30 unsafe prompts generated per taxonomy node via an LLM-as-author pipeline.
   • Adversarial Mining: Hard negatives evolved to bypass existing safety models (inspired by red-teaming pipelines).
   • Benign Contrast Examples: Safety-related but safe text to reduce false positives on sensitive contexts.
   • Response Generation: Responses generated by a fine-tuned Qwen3-4B model and validated by an LLM-as-a-judge panel.
   • Multilingual Translation: Data translated into 23 languages using DeepSeek-V3.1.
   • Open Subsets: Portions of Aegis2 and WildGuardMix training data.

4. Training Strategy: Training involves a two-stage process: initial training on the main dataset followed by post-training on augmented data (10% sample). Augmentation techniques include label dropout, prompt perturbation, and the insertion of safe-looking distractor instructions to improve robustness against prompt injection and instruction-hierarchy violations. The system supports Elastic Weight Consolidation (EWC) for continual learning.

Key Contributions

1. A Comprehensive Safety Taxonomy: A 3-level, 996-label taxonomy covering ordinary toxicity, LLM-specific attacks (including indirect prompt injection and instruction-hierarchy violations), and benign sensitive contexts.
2. A GLiClass-Based Model Family: Development of Opir variants (multi-task and edge-oriented) that support binary classification, multi-label toxicity/jailbreak detection, and zero-shot categorization across 23 languages.
3. Synthetic and Open-Data Recipe: A reproducible pipeline combining taxonomy-derived generation, adversarial hard-negative mining, and multilingual translation.
4. Multi-View Evaluation Harness: An open-sourced evaluation suite supporting GLiClass, GLiNER2, and decoder-based backends (vLLM), covering binary safety, toxicity, jailbreaks, and subcategory views across 12 safety datasets and 17 categorization splits.
5. Extensive Benchmarking: A comparative analysis against eight contemporary guardrail systems (including GLiNER2-based and generative models) demonstrating competitive accuracy with significantly reduced latency.

Results

The paper evaluates Opir against eight contemporary guardrail systems (including GLiGuard, Gliner-Guard-Omni, WildGuard, Nemotron Safety Guard, PolyGuard, and Qwen3Guard) across 12 binary safety datasets and 17 categorization tasks.

• Accuracy:

  • Binary Safety: Opir-multitask-large achieved the second-highest row-average macro F1 (0.8045) among 11 systems, trailing only the 8B-parameter Nemotron Safety Guard v3 (0.8061) and outperforming other decoder-based models on average. It won two individual datasets (WildGuard prompt safety and JBB-Behaviors safety).
  • Edge Performance: Opir-edge-multilang (<100M parameters) won on Aegis prompt safety (0.9321) and WildGuard response safety (0.9194).
  • Categorization: Opir-multitask-large achieved the highest average accuracy (0.5432) across 17 categorization datasets, winning 11 of 17 rows. It showed substantial margins over baselines on Aegis categories, HarmBench prompts, and WildGuard subcategories.
  • Limitations: The models struggled with the OR-Bench family (over-refusal probes), where GLiGuard-300M outperformed Opir. The authors attribute this to a calibration issue where Opir's taxonomy maps these prompts to broader benign categories, leading to higher abstain rates.

• Latency and Throughput:

  • At 1024 tokens, Opir-multitask-large achieved 50.51 samples/s with a p50 latency of 25.65 ms.
  • The edge variants demonstrated superior efficiency: Opir-edge reached 499.49 samples/s with a p50 latency of 9.25 ms.
  • Compared to the strongest decoder-based baseline (Nemotron Safety Guard v3 at 97.63 ms p50), Opir variants operate 10–33 times faster, offering comparable safety accuracy at a fraction of the inference cost.

Significance and Claims

The paper positions Opir as a solution to the latency and cost bottlenecks of current LLM guardrails. By leveraging an encoder-based architecture, Opir achieves order-of-magnitude latency reductions compared to 7B–22B decoder models while maintaining competitive accuracy. The authors claim that Opir variants are "competitive on or ahead of the strongest open-weight baselines on the majority of benchmark datasets while operating with a substantially smaller deployment footprint."

The inclusion of benign sensitive categories is highlighted as a specific mitigation against over-refusal, a known failure mode in strict safety filters. The authors emphasize that Opir is designed for real-time moderation, safety routing, and offline analysis, but explicitly state it should not be used as the sole basis for high-impact decisions (legal, employment, etc.) or as a substitute for human review and policy design. The work demonstrates that efficient, encoder-based models can effectively handle complex, multi-task safety classification without the computational overhead of generative guardrails.