Beyond a Global Norm: Personalizing Toxicity Sensitivity in Language Models Without Retraining

Technical Summary: Beyond a Global Norm: Personalizing Toxicity Sensitivity in Language Models Without Retraining

Problem Statement

Current toxicity mitigation in Large Language Models (LLMs) is predominantly framed as a global alignment problem, where models are optimized against a single, aggregated notion of acceptable language. However, perceptions of harmful language are inherently subjective, context-dependent, and vary systematically across individuals and communities based on identity, experience, and social norms. Global alignment approaches, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), aggregate heterogeneous judgments into a single training signal. While effective for general safety, this often subsumes minority perspectives and fails to accommodate users who differ in what they consider harmful, acceptable, or necessary in specific contexts.

The paper addresses the challenge of personalized toxicity alignment: steering model generation to match individual users' specific sensitivity profiles without the computational cost and rigidity of retraining separate models for each user.

Methodology

The authors conduct the first comparative evaluation of training-free alignment methods across three distinct inference-time intervention stages. The evaluation utilizes the PRISM dataset (Kirk et al., 2024), which links participants, prompts, and acceptability ratings.

1. Personalization Target Construction:
For each user, a toxicity-sensitivity profile is constructed using the Perspective API. The profile is a vector of sensitivity scores ($s_{u,c}$) across six categories: Toxicity, Severe Toxicity, Identity Attack, Insult, Profanity, and Threat. These scores are computed as the dislike-weighted mean toxicity of responses the user rated, where "dislike" is derived from the inverse of the user's acceptability rating.

2. Evaluated Methods (Seven Total):
The study compares seven methods grouped by their intervention stage:

• Pre-Decoding (Input Modification):

  • URIAL (Prompt Conditioning): Adds user-specific safety instructions and in-context examples to the prompt, guiding the model toward the user's primary sensitivity category without altering the original task.
  • PBPO (Prompt Rewriting): Uses a separate model to rewrite the user's prompt into a "neutralized" version based on their six-dimensional toxicity profile before passing it to the target model.

• In-Decoding (Generation Control):

  • PCAA (Activation Steering): A confidence-gated hybrid of contrastive activation addition. It computes a population-level toxicity direction and a user-specific residual based on a Bradley-Terry preference model. These are combined to steer hidden states during the forward pass.
  • CGD (Classifier-Guided Token Selection): At each decoding step, a toxicity classifier (Detoxify) scores top-$k$ token candidates. The user's sensitivity vector weights these scores to adjust logits, penalizing tokens that violate the user's specific sensitivities.
  • EADS (Expert–Anti-Expert Steering): Runs three synchronized decoding branches (base, expert for desired style, anti-expert for toxic style). The logits are combined to shift the distribution away from the user's most sensitive toxicity dimension.

• Post-Decoding (Output Selection):

  • CRR (Candidate Re-Ranking): Generates $N$ candidates from the base model and re-ranks them using a classifier. Two distance metrics are tested: a user-weighted Manhattan distance (CRR-L1) and a Mahalanobis distance (CRR-M) that accounts for inter-category correlations.

3. Evaluation Metrics:

• Alignment: Mean Absolute Error (MAE) reduction between the generated response and the user's preferred response (from PRISM) across the six toxicity dimensions.
• Trade-offs: Knowledge retention (measured via MMLU accuracy) and generation fluency (measured via perplexity).
• Personalization Validity: A median-centered cosine similarity test to determine if reductions occur in the specific categories a user is sensitive to, rather than a generic global shift.

Key Results

• Effectiveness: All seven methods successfully reduced the alignment error (distance to user-specific targets) by 28% to 47% compared to the unsteered baseline. The largest gains were observed in Severe Toxicity and Threat categories.
• Performance by Stage:
  • URIAL (Pre-decoding) achieved the highest overall MAE reduction (47.4%), but this was partly driven by a broad shift toward safer text rather than precise personalization.
  • PCAA (In-decoding) was the most stable method across different random seeds.
  • CRR (Post-decoding) and PCAA/CGD (In-decoding) demonstrated more targeted personalization, aligning reductions specifically with the user's sensitive categories.
• The Personalization vs. Alignment Trade-off: A critical finding is that high aggregate alignment does not equate to high personalization. Methods like URIAL, PBPO, and EADS showed near-chance performance in targeting specific user sensitivities, suggesting their success stems from generic toxicity reduction. Conversely, methods like CGD and CRR showed stronger alignment with individual profiles but smaller aggregate MAE reductions.
• Quality Trade-offs:
  • Knowledge Retention: Pre-decoding methods (URIAL, PBPO) caused the most significant drop in MMLU accuracy, indicating a loss of knowledge retention. In-decoding methods generally preserved knowledge better.
  • Fluency: Pre-decoding methods maintained low perplexity (high fluency), while in-decoding methods showed moderate perplexity. EADS offered a conservative trade-off with low perplexity and moderate alignment.
• Demographic Variance: Improvements were not evenly distributed across racial and age groups. For instance, URIAL performed well for White and Mixed users, while CGD was superior for Asian and Hispanic/Latino users. Notably, PBPO decreased alignment for Mixed-race users, highlighting that aggregate metrics can mask subgroup disparities.

Significance and Claims

The paper claims to present the first comparative evaluation of training-free methods for personalized toxicity alignment. Its primary significance lies in demonstrating that:

1. Inference-time steering is viable: Off-the-shelf models can be adapted to individual sensitivity preferences without retraining.
2. Personalization is a multi-objective problem: There is an inherent trade-off between alignment effectiveness, the specificity of personalization, and general language quality (fluency and knowledge retention). No single intervention stage dominates across all metrics.
3. Aggregate metrics are insufficient: Evaluating personalization solely by average error reduction is misleading, as it can conflate generic safety improvements with genuine user-specific adaptation.
4. Subgroup effects matter: Methods that appear effective on average may inadvertently amplify disparities or degrade performance for specific demographic groups.

The authors conclude that toxicity sensitivity alignment requires methods and evaluations that jointly account for user preferences, language quality, and fairness across diverse subgroups, rather than optimizing for a single global norm.