Technical Summary: Detection and Safeguarding of Chinese Toxic Content from Users and LLMs: A Survey

Problem Statement

The proliferation of toxic content on Chinese social media and the emerging risks associated with Large Language Models (LLMs) present significant challenges to online safety and social stability. While research on toxic content detection in Western languages (particularly English) is well-established, Chinese toxic content detection remains fragmented. This gap is exacerbated by the unique linguistic characteristics of Chinese, such as the heavy reliance on culture-specific context, abundant internet slang, homophones, and character substitutions used to evade moderation. Furthermore, the rapid advancement of LLMs has introduced a dual challenge: LLMs are increasingly used to evaluate user-generated toxicity, yet they are also susceptible to "jailbreak" attacks that induce them to generate toxic content. Existing literature lacks a systematic synthesis of Chinese-specific toxic content detection and LLM safeguarding, leaving open challenges in handling implicit toxicity, multimodal threats, and dynamic linguistic evolution.

Methodology

This paper employs a systematic survey methodology to consolidate research on Chinese toxic content detection and safeguarding. The approach is structured around a proposed research taxonomy covering two primary dimensions:

1. User-Generated Toxic Content Detection: Focusing on Chinese social media platforms (e.g., Weibo, Zhihu, Tieba).
2. LLM-Generated Toxic Content Safeguarding: Focusing on preventing Chinese LLMs from producing harmful outputs.

The review process involved:

• Taxonomy Construction: Defining categories for toxic content (offensive language, hate speech, sarcasm, multimodal toxicity) and safeguarding scenarios (jailbreak attacks, safety alignment).
• Literature Retrieval: A progressive search strategy using Google Scholar up to January 2026, focusing on studies that treat Chinese toxic content detection or safeguarding as core tasks.
• Synthesis: Analyzing representative datasets, methodological frameworks (input design, model architecture, training strategies), and evaluation metrics.

Key Contributions

1. Comprehensive Taxonomy and Dataset Review

The paper establishes a unified framework for Chinese toxic content, categorizing it into:

• Offensive Language: Including abusive language, cyberbullying, and general offensive expressions.
• Hate Speech: Covering targeted attacks based on social identity, bias, and discrimination.
• Sarcasm: Distinguishing between irony and mocking language that relies on context.
• Multimodal Toxicity: Addressing harmful memes, videos, and audio.

The authors summarize key datasets (e.g., COLD, COLA, ToxiCN, CHMEMES) and highlight their limitations, such as inconsistent annotation criteria and a lack of coverage for dynamic internet slang and specific toxicity types like fraud or exaggerated advertising.

2. Methodological Analysis of Detection

The survey categorizes detection methods into three levels:

• Input-Level: Techniques include Data Augmentation (pseudo-labeling, back-translation), Variant Decoupling (normalizing homophones and character splits), and Knowledge Enhancement (integrating external cultural knowledge and LLM-generated rationales).
• Model-Level: The review contrasts traditional Encoder-Classifier architectures (fine-tuned BERT/RoBERTa with attention mechanisms) against Generative Prompt Learning approaches (using Chain-of-Thought and reasoning-oriented prompts to handle implicit toxicity).
• Training-Level: The paper analyzes Multi-Task Learning (joint and progressive learning) and Transfer Learning (cross-lingual and cross-model distillation), noting the challenges of negative transfer due to cultural and linguistic mismatches between English and Chinese.

3. LLM Safeguarding Landscape

The paper reviews the dual aspects of LLM safety:

• Jailbreak Attacks: It details attack construction methods (structured generation vs. automated search), scenario setups (single-turn vs. multi-turn), and Chinese-specific perturbation augmentations (pinyin mixing, character decomposition) designed to bypass safety filters.
• Safety Alignment: The survey covers Training-time Optimization (Instruction Tuning, RLHF, Constitutional AI) and Inference-time Guardrails (input screening, output moderation, token-level intervention). It highlights the trade-off between safety and utility, noting issues like "over-refusal" where models block benign content.

4. Identification of Challenges and Future Directions

The authors identify critical gaps in current research:

• Data Challenges: Lack of unified annotation criteria, limited diversity in datasets, and the dynamic evolution of toxic expressions (e.g., slang like "nephritis" evolving into a derogatory term).
• Model Challenges: Difficulty in detecting implicit attacks, over-sensitivity to specific topics (leading to bias), and limited capability in understanding multimodal toxic content.

Consequently, the paper outlines six future research directions:

1. Multi-Level Data Optimization: Creating unified taxonomies and expanding benchmarks across platforms and models.
2. Dynamic Knowledge Editing: Updating model knowledge to adapt to new cultural metaphors and slang without full retraining.
3. Fairness Research: Developing context-aware mechanisms to reduce over-sensitivity and bias toward specific groups.
4. Cross-Scenario Adaptation: Improving generalization across different platforms and community norms.
5. Interpretability and Attribution: Analyzing failure modes and internal model mechanisms to improve reliability.
6. Multimodal Implicit Jailbreak Detection: Addressing threats that span text, image, and video to evade single-modality filters.

Results and Claims

The paper does not present new experimental results but synthesizes findings from existing literature to map the technical landscape. Key observations include:

• Fragmentation: Research on Chinese toxic content is currently scattered across tasks, datasets, and methodologies, lacking a cohesive overview.
• Linguistic Specificity: Standard Western methods often fail on Chinese content due to the unique nature of Chinese obfuscation (homophones, character splitting) and cultural context.
• LLM Vulnerability: While safety alignment has improved, LLMs remain vulnerable to sophisticated jailbreaks, particularly those leveraging Chinese-specific perturbations and multimodal inputs.
• Industrial vs. Academic Gap: Industrial moderation systems often employ multi-stage, policy-aware workflows that differ significantly from the single-modality models dominating academic benchmarks.

Significance

The authors claim this is the first systematic review dedicated specifically to Chinese toxic content detection and safeguarding. Its significance lies in:

• Consolidating Progress: It bridges the gap between Western-centric research and the specific needs of the Chinese online ecosystem.
• Guiding Future Research: By identifying open challenges (e.g., dynamic evolution of slang, multimodal threats), it provides a roadmap for researchers to develop more robust and culturally grounded solutions.
• Bridging User and LLM Safety: It uniquely connects the detection of user-generated toxicity with the safeguarding of LLMs, recognizing that these two domains are increasingly interdependent in the era of generative AI.

The paper concludes that while substantial progress has been made, achieving reliable, fair, and robust safety in Chinese online ecosystems requires continued innovation in data construction, model interpretability, and dynamic adaptation to evolving linguistic threats.