💬 NLP

Toxic Subword Pruning for Dialogue Response Generation on Large Language Models

This paper introduces ToxPrune, a novel and efficient algorithm that prunes subwords associated with toxic vocabulary from trained Large Language Models, effectively preventing toxic content generation while simultaneously enhancing dialogue diversity and performance without the high costs or risks of traditional weight-based safety alignment.

Original authors: Hongyuan Lu, Wai Lam

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hongyuan Lu, Wai Lam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, creative robot friend who loves to chat. Sometimes, this robot gets a little too wild and starts spouting off rude, offensive, or "toxic" words. Usually, to fix this, developers have to take the robot apart, retrain it, or install a complex security guard that checks every sentence before it leaves. This is expensive, slow, and sometimes the robot still finds a way to break the rules.

This paper introduces a much simpler trick called ToxPrune (Toxic Subword Pruning). Think of it not as retraining the robot, but as editing its dictionary before it even starts speaking.

Here is how it works, using some everyday analogies:

1. The "Lego" Analogy

Large Language Models (LLMs) don't see words like "fuck" or "stupid" as single blocks. Instead, they see them as smaller pieces, like Lego bricks (called "subwords").

The word "fucking" might be built from two bricks: "f" and "ucking."
The word "reading" might be built from "read" and "ing."

Sometimes, a "toxic" brick is shared between a bad word and a good word. For example, the brick "f" might be used in "f*cking" (bad) but also in "fun" (good).

2. The "Scissors" Method

The authors propose a simple algorithm: Take a pair of scissors and cut out the specific Lego bricks that make up the bad words.

The Process: You give the computer a list of words you don't want (like a "naughty list"). The computer breaks those words down into their tiny Lego bricks.
The Pruning: It then goes into the robot's brain and says, "If you try to build a sentence using these specific bricks, you can't." It effectively deletes those bricks from the robot's available toolkit.
The Result: The robot tries to answer your question, but because the "bad bricks" are missing, it is forced to build a different sentence using only the "good bricks."

3. What Happens When You Do This?

The paper tested this on two types of robots:

A. The "Bad" Robot (NSFW-3B)
This robot was trained specifically to be rude and say dirty words.

Before: If you asked, "What are your hobbies?", it would reply, "My hobbies are f*cking boring."
After ToxPrune: Because the "f*cking" bricks were removed, the robot couldn't say that. Instead, it was forced to say, "My hobbies are reading mysteries and driving a truck."
The Surprise: Not only did it stop being rude, but it actually became better at chatting. It started using more diverse words and didn't get stuck repeating the same bad phrases. It was like removing a bad habit that was actually holding the robot back from being creative.

B. The "Good" Robot (Llama-3.1)
This robot was already polite and safe.

Before: It was polite but sometimes a bit repetitive.
After ToxPrune: Even though it was already safe, removing a few common "bricks" (that happened to be shared between toxic and non-toxic words) made it more diverse. It started using a wider variety of words to express the same ideas, making the conversation feel more natural and less robotic.

4. Why Is This Different?

Usually, if you want to stop a robot from saying bad things, you might try to:

Rewrite its brain: Expensive and slow (like going to school for a new degree).
Put a filter on the door: A security guard that checks every sentence. If the guard is too strict, they might block useful sentences too.

ToxPrune is different because it's like changing the ingredients in the kitchen. You don't need a security guard; you just don't have the toxic ingredients on the shelf. The robot physically cannot cook a toxic meal because the ingredients are gone.

The Bottom Line

The paper claims that by simply removing the tiny building blocks of toxic words from a language model's dictionary, you can:

Stop toxic robots from saying bad things, even if they were trained to be toxic.
Improve the quality of the conversation, making it more diverse and interesting.
Do it instantly without needing to retrain the model or pay for expensive computing power.

It's a "lightweight" fix that turns a robot's inability to say a bad word into an opportunity to say something better.

Technical Summary: Toxic Subword Pruning for Dialogue Response Generation on Large Language Models

Problem Statement

The rapid advancement of Large Language Models (LLMs) has introduced significant risks regarding the generation of harmful or toxic content. While existing safety research primarily focuses on defending "safe" models against jailbreak attacks or toxic prompts, a critical gap remains for models that are already toxic. This scenario arises when individual developers unintentionally create toxic models or when attackers gain access to model weights to fine-tune models with harmful content.

Current mitigation strategies face several limitations:

Safety Alignment: Fine-tuning pre-trained models for safety is computationally expensive and often inaccessible to individual developers.
External Classifiers: Using external toxicity classifiers to detect and block toxic output is imperfect. Attackers can bypass these via malicious injection prompts, and continuous blocking can render the model useless by preventing any output.
Complex Detoxification: Existing methods for detoxifying toxic LLMs are often too complex for rapid deployment.
Prompt Injection Vulnerability: Even well-aligned models can be triggered into toxic generation via prompt injection.

The paper posits that a lightweight, inference-time solution is needed to prune toxic capabilities directly from the model's decoding process without requiring expensive retraining or weight updates.

Methodology: Toxic Subword Pruning (ToxPrune)

The authors propose ToxPrune, a novel algorithm that prunes subwords (subtokens) associated with toxic words from the vocabulary of a trained LLM during the decoding process.

Core Mechanism

Customized Toxic List: The method accepts a user-defined list of toxic words.
Subword Tokenization: These words are tokenized into subwords using the model's tokenizer (e.g., BPE). The authors note that toxic and non-toxic words may share subwords; ToxPrune handles this by pruning the specific subwords derived from the toxic list.
Distribution Modification: During inference, the probability distribution of the next token is modified. Instead of sampling from the full vocabulary $V$ $V$ , the model samples from a restricted set $V \setminus V_{toxic}$ $V ∖ V_{t o x i c}$ , where $V_{toxic}$ $V_{t o x i c}$ represents the pruned toxic subwords.
- Formally, for top- $k$ sampling, the distribution changes from $P(\bar{x}_i | \dots) \in V^{(k)}$ to $P(\bar{x}_i | \dots) \in V^{(k)} \setminus V^{(k)}_{toxic}$ .
- For Beam Search, the probability of sequences containing pruned words is set to 0, effectively dropping them from the candidate list.
Expansion and Truncation:
- Paraphrased Blacklist: To increase coverage, the initial toxic word list can be expanded using a paraphrase generator to include variations of toxic terms.
- Truncated Whitelist: A pre-defined whitelist can be used to preserve specific keywords by removing toxic subwords that overlap with safe words, ensuring critical non-toxic terms remain usable.

Implementation

ToxPrune operates entirely at the inference stage. It does not require updating model weights, retraining, or external classifiers. It integrates with standard sampling strategies (Top- $k$ , Top- $p$ , Beam Search) by simply filtering the candidate vocabulary before selection.

Key Contributions

Novel Algorithm: The introduction of ToxPrune, a method that deletes toxic subwords to directly affect the decoding process of dialogue generation.
Dual Validation: The method is validated on both a known toxic LLM (NSFW-3B) and an officially aligned, non-toxic LLM (Llama-3.1-6B Base).
Empirical Evidence: Extensive case studies and automatic evaluations demonstrate that ToxPrune effectively mitigates toxic generation while maintaining or improving dialogue quality.

Experimental Results

Datasets and Models

Datasets: Experiments were conducted on PERSONACHAT-ORI and PERSONACHAT-REV (a manually revised, harder version), which are multi-turn dialogue datasets conditioned on personas.
Models:
- NSFW-3B: An open-source toxic LLM trained to generate dirty words.
- Llama-3.1-6B: The official base version of Llama-3.1 with safety alignment.
Baselines: Comparisons included a paraphrase-based approach and a Toxic Classifier (TC) using GPT-4 for regeneration.

Performance on Toxic LLMs (NSFW-3B)

Toxicity Reduction: ToxPrune drastically reduced toxicity scores (judged by GPT-4) from 0.89 (baseline) to 0.13 (with 100% pruning).
Quality Improvement: Contrary to the expectation that pruning would degrade performance, most metrics improved:
- BLEU: While BLEU-1 saw a minor drop, BLEU-2 through BLEU-4, and weighted BLEU increased.
- ROUGE: All ROUGE metrics (R-1, R-2, R-L) improved.
- Diversity: Distinct-1 and Distinct-2 scores increased significantly (e.g., Distinct-2 from 0.712 to 0.800).
Comparison: Baselines using paraphrasing or toxic classifiers failed to mitigate toxicity effectively because the underlying NSFW-3B model consistently regenerated toxic content.

Performance on Non-Toxic LLMs (Llama-3.1-6B)

Safety: Toxicity remained at 0.00.
Diversity: ToxPrune surprisingly improved diversity metrics (Distinct-1 and Distinct-2) on the clean model, suggesting that masking frequent subwords shared between toxic and non-toxic contexts flattens the word distribution, leading to more varied outputs.
Quality: BLEU and ROUGE metrics showed slight variations, with some improvements in ROUGE, indicating that the pruning did not harm the model's core generative capabilities.

Human Evaluation

Toxic LLM: Human annotators significantly preferred ToxPrune over the baseline NSFW-3B across Appropriateness, Informativeness, Engagingness, and Human-likeness. The toxicity of the baseline was largely eliminated.
Non-Toxic LLM: ToxPrune did not degrade performance on Llama-3.1-6B and showed a statistically significant improvement in Informativeness (diversity).

Significance and Claims

The paper claims that ToxPrune offers a simple, effective, and novel approach to AI safety with unique advantages:

Remediation of Toxic Models: It can rescue models that have been compromised or intentionally trained to be toxic, restoring their ability to generate meaningful, non-toxic dialogue without retraining.
Enhancement of Safe Models: It can improve the diversity of dialogue in already safe models, challenging the notion that pruning vocabulary is inherently harmful to generation tasks.
Robustness Against Attacks: By removing toxic tokens from the sampling list entirely, ToxPrune prevents toxic generation even under prompt injection attacks, as the model physically cannot output the pruned tokens.
Efficiency: Unlike safety alignment or model editing, ToxPrune requires no model updates, is computationally cheap, and allows for dynamic customization of the toxic word list.

The authors conclude that while previous work suggested pruning BPE tokens harms tasks like machine translation, this study demonstrates its surprising utility in the domain of AI safety and dialogue generation.