📄 social_science

Evaluative Cultural Hallucination in Large Language Model Assessment of Tenor in Scenic Area Public Sign Translations

This study reveals that large language models frequently exhibit "evaluative cultural hallucination" by significantly overestimating the pragmatic and socio-cultural appropriateness (Tenor) of scenic area public sign translations, often mistaking surface formality or recognizable references for genuine cultural suitability.

Original authors: Qi Guo, Tengteng Yap

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qi Guo, Tengteng Yap

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

When you walk through a historic temple or a sacred burial ground in China, the signs guiding you are more than just directions. They are a bridge between cultures, translating not only words but also the deep respect, history, and solemnity required to speak about a place where ancestors rest or deities are honored. This is the realm of public-sign translation, a task that demands a delicate balance between clarity and cultural sensitivity. For decades, experts have judged these translations by looking at "tenor," a concept that describes the relationship between the writer and the reader. It asks whether the tone is right: is it too casual for a sacred site? Is it too stiff for a nature trail? Does it show the proper level of respect for a historical figure? As cities and tourist sites grow, the need to check thousands of these signs has outpaced the ability of human experts to review them all, leading many to wonder if artificial intelligence can step in to help.

Large language models, the powerful computer systems behind modern chatbots, have shown they can read and write with impressive fluency. They can spot grammar mistakes and suggest smoother phrasing. But a new study from researchers at the University of Malaya asks a harder question: can these machines understand the invisible weight of culture? Specifically, they wanted to know if an artificial intelligence would mistakenly praise a translation that sounds polite and formal but actually misses the deep cultural meaning required for a sacred or historical site. The researchers gathered 116 bilingual signs from eight major tourist attractions in Xuzhou, China, ranging from nature guides to descriptions of ancient tombs and Buddhist temples. They asked both human experts and a sophisticated AI model to grade the English translations based on how well they captured the right tone and respect.

The results revealed a significant gap between human intuition and machine judgment. While the human experts gave many of the translations low scores for failing to capture the necessary cultural respect, the AI model consistently gave them high marks. In nearly half of the cases examined, the AI rated a translation as high quality even when experts said it was culturally inadequate. The problem was most severe with signs related to religion and rituals. For these signs, the AI was wrong about the quality of the translation in 65 percent of the cases. The researchers found that the AI was not simply making random errors; it was following a specific, predictable pattern of misunderstanding. It mistook the surface appearance of a text for its deep cultural value.

When the researchers looked closely at why the AI gave these high scores, they found it was relying on four specific shortcuts. First, the model often praised a translation simply because it sounded formal or institutional, like something you might see in a museum. It assumed that if the words sounded official, the tone was correct, even if the specific cultural respect was missing. Second, the AI was fooled by recognizable names. If the translation kept the name of a historical figure or a religious object, the model assumed the job was done, ignoring whether the surrounding words treated that figure with the proper reverence. Third, the model sometimes confused general politeness with specific cultural solemnity. It would see words that sounded respectful in a broad sense and decide the translation was perfect, failing to notice that the specific ritual nuances were lost. Finally, the AI was too tolerant of missing information. It would give a high score to a translation that was short and easy to read, even if it had cut out important details about rituals or history that were essential to the original meaning.

The study suggests that this is not just a case of the machine being slightly off, but a form of what the authors call "evaluative cultural hallucination." This does not mean the machine is making up facts; rather, it is hallucinating that a translation is culturally appropriate when it is not. It sees the surface features—formal words, recognizable names, or a polite tone—and convinces itself that the deep cultural requirements have been met. This is particularly dangerous for signs in religious or ritual settings, where getting the tone wrong can be seen as disrespectful or even offensive. The researchers found that the AI was much better at judging signs about nature or general information, where the tone is less complex. But when the text involved sacred spaces, burial sites, or ancient rituals, the machine's confidence became a liability.

This research does not say that artificial intelligence cannot help with translation. Instead, it warns that we cannot simply trust the machine's score, especially when culture is at stake. The AI is excellent at checking if a sentence is grammatically correct or if a name is spelled right, but it struggles to understand the invisible rules of respect and hierarchy that govern how we speak about the sacred. The study concludes that for signs in tourist areas, particularly those dealing with history and religion, human experts must remain in the loop. The machine can offer a first look, but its reasoning must be checked by a person who understands that a formal tone is not the same as a respectful one, and that a recognizable name does not guarantee a culturally accurate translation. Until machines can truly understand the weight of culture, the most reliable way to ensure these signs honor the places they describe is to keep a human hand on the wheel.

Technical Summary: Evaluative Cultural Hallucination in LLM Assessment of Tenor in Scenic Area Public Sign Translations

Problem Statement
The study addresses a critical gap in the application of Large Language Models (LLMs) to Translation Quality Assessment (TQA), specifically regarding the "Tenor" dimension of House's TQA framework. While LLMs offer scalable alternatives to costly human expert evaluation, their reliability in assessing the socio-cultural and pragmatic appropriateness of translations remains uncertain. This is particularly acute in scenic-area public sign translations, where texts often involve religious references, ritualized expressions, and historical hierarchies. The core problem is that LLMs may produce "evaluative cultural hallucinations": unjustifiably positive assessments of translations that human experts deem culturally or pragmatically inadequate. Unlike factual hallucinations (generating false content), this phenomenon involves the model misinterpreting surface-level textual features (e.g., formality, recognizable terms) as evidence of deep cultural and pragmatic adequacy.

Methodology
The research employed an exploratory mixed-methods design involving 116 Chinese-English bilingual public signs collected from eight 4A/5A tourist attractions in Xuzhou, China. The corpus was categorized into three thematic groups: informational/natural signs ( $n=18$ ), historical-archaeological signs ( $n=35$ ), and religious-ritual signs ( $n=63$ ).

Dual Assessment: Each sign was evaluated by three CATTI-certified human experts and by the LLM "GPT-5.5 Thinking" (accessed via ChatGPT interface). Both used a five-point scale (1–5) based on House's framework, focusing on formality, social distance, politeness, cultural respect, and ritual sensitivity.
Operational Definition: "High-overestimation cases" were defined as instances where the expert median score was $\le 2$ (indicating clear Tenor inadequacy) while the LLM score was $\ge 4$ (indicating high appropriateness).
Qualitative Analysis: The 56 identified high-overestimation cases (48.3% of the corpus) underwent qualitative coding to identify recurrent rationale patterns in the LLM's justification for its high scores. Inter-rater reliability for coding was substantial (Cohen's $\kappa = 0.84$ ).

Key Results
The study found a significant divergence between human and machine evaluations, characterized by a systematic LLM overestimation of Tenor quality.

Quantitative Discrepancy: The mean expert score was 2.33, compared to a mean LLM score of 4.29, resulting in a mean difference of +1.97. In 79.3% of cases, the LLM score exceeded the expert score.
High-Overestimation Prevalence: 48.3% of the total samples met the high-overestimation criterion. This rate varied significantly by thematic category:
- Informational/Natural signs: 27.8%
- Historical-Archaeological signs: 28.6%
- Religious-Ritual signs: 65.1% (the highest concentration of overestimation).
Rationale Patterns: Qualitative coding of the LLM's justifications revealed four recurrent patterns of "surface adequacy substitution," where the model substituted easily detectable cues for complex cultural-pragmatic judgment:
1. Formal-register overreliance (83.9%): The model assigned high scores based on the translation sounding "formal," "neutral," or "institutional," mistaking tone for cultural appropriateness.
2. Semantic recognizability substitution (69.6%): High scores were justified by the preservation of recognizable names, objects, or historical references, regardless of whether the pragmatic force or respect was maintained.
3. Generic reverence substitution (44.6%): The model described translations as "respectful" or "solemn" in a general sense, failing to distinguish between generic politeness and specific ritual or doctrinal requirements.
4. Omission-tolerant simplification (35.7%): The model accepted concise or simplified translations as appropriate, overlooking the loss of culturally significant details, particularly in religious and ritual contexts.

Key Contributions

Conceptualization of Evaluative Cultural Hallucination: The paper introduces and operationalizes the concept of "evaluative cultural hallucination." This distinguishes LLM failures in TQA from factual hallucinations, defining them as unjustifiably positive judgments supported by culturally insufficient rationales.
Identification of Rationale Patterns: The study moves beyond simple score disagreement to systematically classify the specific reasoning errors LLMs make when evaluating culturally sensitive texts. It demonstrates that LLMs often conflate surface features (formality, recognizability) with deep pragmatic adequacy.
Thematic Vulnerability Mapping: The research highlights that LLM reliability is dimension-specific. While LLMs may perform adequately on informational texts, their performance degrades significantly in religious and ritual contexts where Tenor depends on specific cultural hierarchies and sacred distance.

Significance and Claims
The authors claim that their findings serve as a cautionary account for tourism administrators, translators, and evaluators relying on LLMs for public-sign assessment. The study argues that LLM scores and rationales should not be accepted as direct substitutes for expert judgment, particularly in high-risk contexts involving religion, burial culture, or historical hierarchy.

The paper posits that the identified rationale patterns (formal-register overreliance, etc.) can serve as a diagnostic tool. By auditing LLM rationales for these specific shortcuts, evaluators can detect potential "evaluative cultural hallucinations" before accepting a high quality score. The authors conclude that while LLMs are promising for scalable TQA, their use requires "expert-informed, culturally sensitive safeguards," such as refined prompts that explicitly check for ritual distance and culturally marked respect, rather than relying on general assessments of formality or fluency. The study does not claim that LLMs are universally unreliable but emphasizes that their reliability is not uniform across all dimensions of translation quality, with Tenor in culturally dense texts being a specific point of failure.

Technical Summary: Evaluative Cultural Hallucination in LLM Assessment of Tenor in Scenic Area Public Sign Translations

More like this