Technical Summary: Probing Cultural Awareness in LLMs: A Case Study of Cross-Culture Aesthetic Stylistics

Problem Statement

While Large Language Models (LLMs) have demonstrated proficiency in explicit cultural knowledge (e.g., facts, regional cuisine) and linguistic proficiency (e.g., dialects, slang), their ability to master aesthetic stylistics remains underexplored. Aesthetic stylistics are defined as the implicit, strategic choices made among grammatically correct and semantically equivalent expressions to evoke cultural resonance without altering core semantics. This paper investigates whether LLMs possess genuine cultural awareness regarding these subtle stylistic nuances, specifically within the context of Chinese language variations between the Chinese Mainland (CN) and Hong Kong (HK).

Methodology

1. Dataset Construction: C4STYLI

The authors introduce C4STYLI (Cross-Culture Chinese-Chinese Stylistics), a benchmark designed to minimize surface-level linguistic interference (via script normalization between Simplified and Traditional Chinese) while maximizing stylistic salience. The dataset comprises two sub-components:

• C4STYLI-T: 1,530 pairs of translated movie titles from English sources, paired with their official CN and HK releases and plot summaries.
• C4STYLI-S: 2,837 advertising slogans from CN and HK, including brand, product, and year metadata.
• Linguistic Analysis: The authors quantified stylistic differences using source-target divergence (via back-translation embeddings), sentiment intensity changes (VADER), and syntactic features (modifier-to-noun ratios). Results indicate HK titles exhibit greater semantic divergence and affective amplification compared to CN titles, while HK slogans rely more heavily on English tokens and predicate-heavy formulations.

2. Evaluation Framework

The study evaluates 18 LLMs across six model families (OpenAI, Meta-Llama, Google Gemma, DeepSeek, Qwen, SenseNova) using two primary tasks:

• Behavioral Recognition (Classification): Models predict the cultural origin (HK vs. CN) of a text given contextual information (e.g., plot summary for titles, brand info for slogans). Performance is measured via accuracy, precision, recall, and F1 scores.
• Productive Competence (Generation): Models are instructed to generate culturally tailored texts (titles or slogans) for a specific region.
  • Automatic Evaluation: Lightweight classifiers trained on frozen hidden states of Qwen3-8B determine the stylistic alignment of generated text.
  • Human Evaluation: Annotators from HK and CN judge the cultural fidelity of outputs from select models (GPT-4.1, Qwen3-32B, DeepSeek-V3.2).

3. Structural Ablation (Probing)

To determine if models rely on integrated structural understanding or superficial lexical anchors, the authors employ Logistic Regression (LR) probes on the hidden states of Qwen3-8B. They compare classification probabilities on original sequences versus sequences with shuffled tokens (preserving frequency but destroying syntax).

Key Results

1. Recognition Gap and Domain Sensitivity

• Human-Model Disparity: LLMs significantly underperform human baselines (Human Accuracy: ~89-93% vs. Best LLM: ~79-80%).
• Domain Variation: Models perform better on advertising slogans than movie titles. Slogans provide richer context, whereas short movie titles offer fewer tokens for inference.
• Asymmetric Biases: LLMs exhibit text-type-dependent biases. In movie titles, models tend to misclassify CN texts as HK; in slogans, the reverse occurs.
• Training Language: Models trained primarily on Chinese corpora generally outperform English-trained models, but this advantage does not consistently extend to Cantonese-specific models.

2. Decoupling of Recognition and Production

• Decoupling Phenomenon: High recognition accuracy does not guarantee high-quality generation. Models that excel at identifying styles often fail to generate them faithfully.
• Generation Asymmetry:
  • For movie titles, models generate HK-style text more accurately than CN-style text.
  • For slogans, CN-style generation is generally more accurate than HK-style generation.
• Failure Modes:
  • Movie Titles: Models rely on heuristics (e.g., associating literal translations with CN and adaptive ones with HK), leading to errors when these heuristics are violated.
  • Slogans: Models struggle with HK-specific humor and rhetoric, often defaulting to CN-style abstract narratives or relying on explicit geographic markers (e.g., inserting "Hong Kong") rather than embedding cultural nuance.

3. Structural Ablation Findings

• HK Stylistics: When syntactic structures are shuffled, the probability of correctly identifying HK style remains high or increases. This suggests HK style recognition relies on sparse, high-intensity lexical anchors (surface-level words) rather than holistic structural understanding.
• CN Stylistics: Shuffling tokens causes a catastrophic drop in classification probability for CN texts. This indicates CN style recognition relies on integrated syntactic coherence.

Key Contributions

1. Benchmark (C4STYLI): A novel dataset mitigating script-level interference to focus on implicit aesthetic stylistics in Chinese-Chinese translation.
2. Behavioral Gap Analysis: Empirical evidence of the gap between LLMs and humans in stylistic recognition, highlighting domain-specific variations and the decoupling of recognition and production capabilities.
3. Encoding Asymmetry: Structural ablation reveals that LLMs encode HK stylistic preferences via superficial lexical shortcuts, whereas CN stylistic preferences are captured through deeper structural representations.

Significance and Claims

The paper claims to provide a diagnostic framework for evaluating the "implicit how" of cultural awareness in LLMs, moving beyond explicit facts to aesthetic stylistics. The authors assert that current LLMs possess limited sensitivity to specific cultural structures (particularly HK), often relying on surface-level cues rather than deep cultural understanding. This suggests that while models can "match" styles using high-salience lexical anchors, they lack the integrated structural understanding required for genuine cultural alignment. The work underscores the need for more sophisticated evaluation of cultural nuance in AI development, particularly for regions with complex sociolinguistic contexts like Hong Kong.