Guardian-as-an-Advisor: Advancing Next-Generation Guardian Models for Trustworthy LLMs

1. Problem Statement

Current Large Language Model (LLM) safety mechanisms rely heavily on hard-gated guardians (classifiers). These models detect risk and immediately block the user query, replacing the output with a static refusal template. The paper identifies three critical flaws in this paradigm:

• Over-Refusal & Misalignment: Hard gating often conflicts with the base model's specific "model spec" (e.g., a creative assistant expected to offer safe suggestions). This leads to the rejection of policy-compliant queries and a loss of utility.
• Narrow Taxonomies: Prevailing safety taxonomies focus almost exclusively on "safety" (toxicity, jailbreaks, privacy) while neglecting other pillars of trustworthiness: Robustness (handling natural noise, typos, code-mixing) and Honesty (avoiding hallucinations, acknowledging limitations).
• Rigidity: Even "Explainable Classifiers" that provide a rationale still enforce a hard block, preventing the base model from self-correcting or providing nuanced, safe responses.

2. Methodology: Guardian-as-an-Advisor (GaaA)

The authors propose Guardian-as-an-Advisor (GaaA), a soft-gating pipeline that steers the base model rather than blocking it.

Core Mechanism

Instead of blocking a query, the guardian model ($g_\phi$) analyzes the input $x$ and outputs a binary risk label ($c \in \{\text{Harmless, Harmful}\}$) and a concise natural language explanation ($e$).

• Soft Gating: If the label is "Harmless," the query is passed unchanged. If "Harmful" (or "Harmless with concerns"), the explanation is prepended to the original query to form an augmented prompt $\tilde{x}$.
• Re-inference: The base model $f_\theta$ processes $\tilde{x}$. The explicit advice nudges the model to comply with safety policies and handle robustness/honesty issues without overriding its original generation capabilities.

The GuardSet Dataset

To train and evaluate this paradigm, the authors constructed GuardSet, a large-scale dataset ($208k+$ samples) unifying:

• Harmful Data: Toxicity, Jailbreaks, Privacy leakage.
• Harmless Data: Standard QA, reasoning, and creative tasks.
• Trustworthiness Slices: Crucially, the dataset includes specific slices for Robustness (perturbed inputs) and Honesty (queries requiring self-awareness), which are often ignored in safety datasets.
• Construction: A three-stage pipeline involving collection, label mapping/explanation synthesis (using DeepSeek-R1), and validation (LLM-as-a-Judge + human spot-checks).

The GuardAdvisor Model

The guardian model, GuardAdvisor, is trained via a two-stage process:

1. Supervised Fine-Tuning (SFT): Trained on GuardSet to learn the structured output format (Label + Explanation).
2. Reinforcement Learning (RL): Uses Group-Relative Policy Optimization (GRPO).
   • Reward Design: To prevent "reward hacking" (where the model outputs a correct label but a contradictory explanation), the reward is determined by an LLM-as-a-Judge. The judge enforces:
     • Label uniqueness and correctness.
     • Semantic consistency between the label and the explanation.
   • Generalization: The RL stage uses Out-of-Distribution (OOD) data disjoint from the SFT set to prevent overfitting and improve generalization on harmless inputs.

3. Key Contributions

1. Paradigm Shift (GaaA): Introduces a soft-gating framework that preserves the base model's autonomy while injecting explicit, contextual risk guidance, effectively reducing over-refusal.
2. GuardSet Dataset: Releases a comprehensive, multi-domain dataset ($208k+$) that unifies safety, robustness, and honesty, addressing the gap in current trustworthiness evaluation.
3. GuardAdvisor Model: A specialized guardian trained with SFT and RL to produce semantically consistent label-explanation pairs, outperforming general-purpose models in advisory tasks.
4. Theoretical & Empirical Safety: Provides proofs showing that GaaA does not degrade safety compared to hard-gating (under refusal-constrained decoding) and demonstrates minimal latency overhead.

4. Experimental Results

• Detection Accuracy: GuardAdvisor achieves competitive accuracy (90.52% average) compared to strong baselines like Llama-Guard-4 and Granite-Guardian, despite handling a more fine-grained labeling scheme (distinguishing robustness/honesty).
• Response Quality (Win Rate): When used to augment inputs, GuardAdvisor significantly improves the base model's performance on trustworthiness dimensions:
  • Robustness: 63.11% win rate (vs. 39.48% for GPT-4o).
  • Honesty: 68.79% win rate (vs. 54.47% for GPT-4o).
• Latency Analysis:
  • GuardAdvisor inference consumes <5% of the base model's compute.
  • End-to-end overhead is only 2–10% in realistic scenarios where harmful inputs are rare (<1%).
  • A parallel inference strategy (running the guardian and base model simultaneously) can further minimize latency.
• Ablation Studies: Show that balancing the training data (specifically adding OOD data for harmless generalization) is critical for reducing over-refusal while maintaining safety.

5. Significance

This work represents a significant step toward practical, trustworthy LLM deployment. By shifting from "blocking" to "advising," GaaA resolves the trade-off between safety and utility. It allows models to remain helpful and creative while adhering to safety specs, and crucially, it expands the definition of trustworthiness to include robustness against noise and honesty about limitations. The framework is lightweight, low-latency, and compatible with existing deployment pipelines, making it a viable solution for next-generation AI safety.