Exploring the Capability Boundaries of LLMs in Mastering of Chinese Chouxiang Language

1. Problem Statement

While Large Language Models (LLMs) have achieved significant success in general Natural Language Processing (NLP) tasks, their ability to understand and process Chouxiang Language (a unique, evolving Chinese internet subcultural language) remains largely unexplored.

• Context: Chouxiang Language originated around 2015 as a mechanism to evade censorship and express negative sentiments but has evolved into a neutral, highly symbolic subcultural code used by Chinese youth.
• Characteristics: It employs complex transformations including homophonic substitution, visual symbol analogy (using emojis and character radicals), and semantic translation, creating a "non-standard semantic space" that is intelligible only within specific cultural contexts.
• Gap: Existing NLP research on Chinese internet language often focuses narrowly on toxic or perturbed language detection, overlooking the neutral, humorous, and social functions of Chouxiang. Furthermore, current benchmarks exhibit a Western-centric bias, lacking evaluation frameworks for non-Western subcultural variants.

2. Methodology

A. The Mouse Benchmark

The authors introduce Mouse, the first specialized benchmark designed to evaluate LLM proficiency in Chouxiang Language. It consists of 1,099 Chouxiang Language Evaluation Instances (CXEIs).

• Data Construction: A hybrid approach combining web crawling from platforms like Baidu Tieba and Bilibili, supplemented by manual construction by native speakers proficient in the culture.
• Taxonomy: The benchmark formalizes Chouxiang Language into two dimensions:
  1. Representational Components: Homophonic, Visual, and Semantic.
  2. Intents: Eight categories including Comments, Emotional Expression, Group Identity, Humor/Memes, and Toxicity.
• Six Evaluation Tasks:
  1. Translation (TR): Decoding Chouxiang text into Standard Chinese.
  2. Representational Component Classification (RC): Identifying if text contains homophonic, visual, or semantic features.
  3. Intent Recognition (IR): Classifying the pragmatic intent of the text.
  4. Toxicity Detection (TD): Distinguishing between genuine malice and benign subcultural irony/self-deprecation.
  5. Meaning Selection (MS): Selecting the correct meaning from multiple candidates.
  6. Cloze Completion (CC): Filling in blanks in a dialogue context to test contextual adaptability.

B. Experimental Setup

• Models Evaluated: 14 LLMs, including open-source models (Qwen3 series from 0.6B to 32B, Mistral family) and commercial APIs (GPT-5.2, DeepSeek-V3.2, Doubao-Seed-1.8, Qwen3-Max).
• Evaluation Metrics:
  • Translation: Uses an LLM-as-a-Judge (DeepSeek-V3.2) with a rule-based anti-cheating filter, scoring based on semantic alignment with human references (0–2 scale).
  • Classification/Selection: Uses Accuracy, Balanced Accuracy, Macro-F1, and Matthews Correlation Coefficient (MCC) to detect hallucinations and handle class imbalance.
  • Reliability: Validated using Quadratic Weighted Kappa (QWK) to ensure alignment between the LLM judge and human annotations.

3. Key Contributions

1. Subculture Formalization: Introduced Chouxiang Language to the NLP community, defining its taxonomy of representational components and intents.
2. Mouse Benchmark: Created the first comprehensive benchmark for this domain, covering six distinct NLP tasks to evaluate processing capabilities.
3. Comprehensive Analysis: Provided extensive experimental results on SOTA models, analyzing performance gaps, the impact of model scale, and the specific challenges posed by different camouflage types (homophonic vs. visual).

4. Key Results

• Overall Performance: Current SOTA LLMs exhibit clear limitations in mastering Chouxiang Language. While they perform well on tasks involving contextual semantic understanding (e.g., Meaning Selection), they struggle significantly with zero-shot decoding and structural analysis.
• Task-Specific Findings:
  • Translation & Component Classification: Performance is generally low. Models struggle to reconstruct the original meaning from complex visual and homophonic distortions.
  • Toxicity vs. Structure: Models detect toxicity more easily (approx. 15% higher accuracy) than identifying representational components. This suggests a historical research bias toward toxic language detection has made models sensitive to "bad" words but blind to the linguistic logic of the subculture.
  • Model Scale Paradox: Scaling generally improves performance, but the Qwen3-32B model showed a counterintuitive decline compared to Qwen3-14B on several tasks. The authors attribute this to an "overthinking" effect where larger models engage in unnecessary reasoning, leading to hallucinations, whereas smaller models rely on more direct pattern matching.
• Hallucination Analysis: Small-scale and open-source models often exhibit negative Matthews Correlation Coefficient (MCC) values in complex tasks, indicating systematic misinterpretation (hallucination) rather than random guessing.
• Camouflage Impact:
  • Homophonic: Successful recognition of homophones directly correlates with better translation scores.
  • Visual: Visual camouflage remains the most challenging category, likely requiring visual-spatial reasoning capabilities that current text-based LLMs lack.
  • Toxicity: Camouflaged toxic content significantly degrades translation fidelity, as models struggle to recover the underlying intent when evasion techniques are used.

5. Significance and Future Directions

• Cultural Awareness: The study highlights the Western-centric bias in current LLMs and benchmarks. It demonstrates that understanding non-Western subcultures is essential for true cultural alignment.
• Beyond Toxicity: It shifts the focus from merely detecting toxic language to understanding the functional and social roles of internet slang, including humor, identity formation, and emotional expression.
• NLP Evolution: The findings suggest that future NLP systems need to be more robust against "camouflaged" semantics and capable of handling complex, multi-modal (text + emoji) symbolic reasoning.
• Resource Availability: The authors have made the Mouse dataset and code publicly available, fostering further research into multicultural integration and the dynamics of evolving internet languages.

In conclusion, the paper establishes that while LLMs are powerful generalists, they currently lack the specific cultural and linguistic "decoding" skills required to navigate the complex, evolving landscape of Chinese Chouxiang Language, necessitating specialized benchmarks and culturally aware training strategies.