A Reproducible, License-Aware Distillation Recipe for CPUDeployable Safety Classification

Technical Summary: A Reproducible, License-Aware Distillation Recipe for CPU-Deployable Safety Classification

Problem Statement
Deploying safety layers for Large Language Models (LLMs) on commodity hardware is currently constrained by the limitations of existing open guard models. State-of-the-art guards, such as the Llama Guard 3 line, typically range from 1 to 9 billion parameters and are optimized for Graphics Processing Units (GPUs). When forced onto Central Processing Units (CPUs), these models exhibit latencies in the range of seconds per request, rendering them impractical for many local, cost-sensitive, or latency-critical applications. While lightweight encoders exist, they often cover only narrow slices of safety policies (e.g., toxicity only) rather than a comprehensive multi-hazard taxonomy. Furthermore, existing safety benchmarks often lack independent gold sets labeled separately from the system under test, and frequently omit a "benign" slice, making it impossible to measure "over-defense" (the blocking of harmless prompts).

Methodology
The paper presents a reproducible, license-aware knowledge distillation pipeline designed to transfer safety signals from a strong teacher model to a fleet of smaller, CPU-deployable students.

• Teacher Selection and Labeling: The authors selected Llama Guard 3 8B as the teacher based on a comparative shootout against a gold benchmark, where it achieved the highest macro-F1 and lowest benign false-positive rate among candidates. The teacher labeled a corpus of approximately 97,000 prompts drawn from 24 public datasets.
• Taxonomy: The safety categories are mapped to a seven-class taxonomy aligned with the MLCommons AILuminate v1.0 standard. The classes cover physical harm, hate/defamation, sexual content, self-harm, sensitive information, harmful advice, and illicit acts. The output is multi-label, though the training data is effectively single-label.
• Data Pipeline and Licensing: A critical methodological feature is the partitioning of the training corpus at the license boundary.
  • The Commercial Bucket (75,112 rows) contains only data with permissive licenses allowing commercial use.
  • The Noncommercial Bucket (22,332 rows) contains data under noncommercial or copyleft terms.
  • This allows for the creation of two distinct recipes: a "deployable" model trained only on the commercial bucket, and a "research" model trained on the union of both. This design makes the cost of the license restriction directly measurable.
  • Leakage Control: Strict deduplication (hashing and MinHash) ensures no evaluation data leaks into training.
• Label Overrides: The pipeline includes auditable, in-memory overrides for specific cases where the teacher exhibited blind spots (e.g., under-firing on synthetic PII or misclassifying adversarial suites as safe). These overrides are applied during training but do not alter the stored teacher labels, preserving auditability.
• Student Fleet: A diverse fleet of students was trained to reproduce the teacher's signal:
  • Lexical: TF-IDF + Logistic Regression baseline.
  • Shallow: TextCNN and BiLSTM (~4M parameters).
  • Encoders: MiniLM-L6 (22M) and DistilBERT (66M).
  • Generative: Qwen2.5-0.5B with Low-Rank Adaptation (LoRA).
• Training Strategy: The only decisive training ingredient identified was per-class rebalancing (using positive-class weights based on the square root of the negative-to-positive ratio) to address severe class imbalance, particularly for sparse categories like self-harm.
• Evaluation Protocol: Models were evaluated on an independent gold benchmark of 6,361 rows, divided into four slices: dev (development), headline (balanced, standard-aligned), stress (adversarial), and benign (harmless prompts). Metrics included macro-F1, per-class F1, and benign false-positive rates, all reported with 95% bootstrap confidence intervals.

Key Results

• Distillation Gap: The distilled students (ranging from 16x to ~2,000x smaller in parameter count) match the teacher's performance on adversarial text within overlapping confidence intervals but do not statistically surpass the teacher on clean reference data.
• Over-Defense Reduction: The generative student (Qwen2.5-0.5B) significantly outperforms the teachers in reducing false alarms on harmless prompts, achieving a benign false-positive rate of 3.8% compared to 4.8% for the 8B teacher and 11.5% for the 1B teacher.
• Latency: On a commodity CPU (AMD Ryzen 5 5600X), the entire student fleet (excluding the generative model) classifies requests in under 28 ms (median), with shallow and lexical models operating in under 1.5 ms. The generative model operates at ~128 ms, still an order of magnitude faster than the multi-second latency of the 8B teacher on CPU.
• Optimal Trade-off: The DistilBERT encoder is identified as the optimal choice for the quality-latency trade-off, offering higher adversarial macro-F1 than the generative student while being roughly five times faster.
• License Cost: Restricting training to the commercial bucket (excluding noncommercial data) results in a negligible drop in headline macro-F1 (+0.005 difference) but actually improves the benign false-positive rate, suggesting the license constraint is "nearly free" in terms of performance cost.
• Taxonomy Limitations: Per-category analysis reveals that the "harmful advice" category performs poorly across all models (including the 8B teacher), with F1 scores around 0.20. This suggests the limitation lies in the category definition rather than model capacity.

Significance and Claims
The paper explicitly frames its contribution as a resource and benchmark contribution rather than a claim of state-of-the-art performance. It does not claim to surpass Llama Guard 3. Instead, its significance lies in:

1. Reproducibility and Auditability: Providing a fully reproducible pipeline where the impact of data licensing is quantifiable and where label overrides are auditable.
2. CPU Deployability: Demonstrating that safety classifiers of comparable quality to large guards can run on commodity CPU hardware with millisecond latency.
3. Measuring Over-Defense: Establishing a rigorous evaluation protocol that includes a benign slice to measure false positives, a metric often missing in safety literature.
4. Identifying Bottlenecks: Highlighting that the current ceiling for specific safety categories (like harmful advice) is defined by the taxonomy and data definitions, not by the scale of the model.

The authors conclude that while distillation successfully transfers safety signals to smaller models, the "clean" performance remains ahead of the teacher, and the primary value of the work is the methodology for creating auditable, license-compliant, and latency-efficient safety layers.