💬 NLP

JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors

The paper introduces JuICE, a multilingual benchmark dataset designed to evaluate the limitations of LLM-judges in detecting subtle, context-dependent cultural errors, revealing that current models struggle to identify "thick" cultural nuances that native speakers readily recognize.

Original authors: Jiho Jin, Junho Myung, Juhyun Oh, Junyeong Park, Rifki Afina Putri, Sunipa Dev, Vinodkumar Prabhakaran, Alice Oh

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiho Jin, Junho Myung, Juhyun Oh, Junyeong Park, Rifki Afina Putri, Sunipa Dev, Vinodkumar Prabhakaran, Alice Oh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a very smart, well-read librarian to check stories written by a new, super-fast robot writer. The robot is great at facts: it knows that Paris has the Eiffel Tower and that water is wet. But the robot sometimes misses the "vibe" of a place. It might write a story set in Bangladesh where a character stops for a fancy coffee, when in reality, everyone in that neighborhood stops at a roadside tea stall. Or it might describe the smell of jasmine in an Indonesian city as "pleasant," not realizing that in that culture, that specific scent is strongly linked to funerals and ghosts.

This paper, JuICE, is about testing how good our "librarian" (which is actually an AI called an LLM-Judge) is at catching these subtle, cultural mistakes.

Here is the breakdown of what they did and found, using simple analogies:

1. The Problem: The "Thick" vs. "Thin" Mistake

The authors realized that mistakes come in two flavors:

Thin Mistakes (Surface Level): These are like typos or wrong facts. "The capital of France is London." Easy to spot.
Thick Mistakes (Deep Cultural Level): These are like a chef putting pineapple on a pizza in a traditional Italian pizzeria. The ingredients are real, but the combination feels "off" to a local. It's not factually wrong; it's culturally awkward, insensitive, or missing a key piece of the local puzzle.

Existing tests only checked for "Thin" mistakes. They wanted to see if AI judges could catch the "Thick" ones.

2. The Tool: JuICE (The Cultural Detective Kit)

The team built a massive dataset called JuICE. Think of it as a giant library of 1,050 stories and advice columns generated by robots, covering four different countries (USA, South Korea, Indonesia, Bangladesh) in their local languages and English.

They didn't just let the robots grade themselves. They hired 44 real human locals (like native speakers from those countries) to read the stories and highlight exactly where the robot "got the vibe wrong."

They found 7,470 specific errors.
They categorized them into things like "Cultural Incoherence" (mixing things that don't go together), "Cultural Missingness" (forgetting a crucial local tradition), and "Cultural Connotation" (using a word that means something sad or spooky to locals).

3. The Experiment: Can the Robot Judge Catch the Robot Writer?

They took the best AI models available (the "Judges") and asked them to read the stories and find the errors the humans had found. It was like asking a robot librarian to find the cultural mistakes in a story written by another robot.

The Results were disappointing:

Even the smartest AI Judge only caught about 52% of the errors (an F1 score of 0.52).
They were pretty good at finding "Thin" mistakes (typos, wrong facts).
They were terrible at finding "Thick" mistakes. For example, they almost never caught errors about Cultural Missingness (forgetting a local tradition) or Cultural Connotation (misunderstanding the emotional weight of a symbol).

The Analogy: It's like asking a robot to find a needle in a haystack. The robot is great at finding the shiny, obvious needles (facts), but it keeps missing the dull, camouflaged needles (cultural nuances) that a human would spot instantly.

4. The Twist: Giving the Judge a Cheat Sheet

The researchers wondered: "What if we give the AI Judge a dictionary of what these 'Thick' mistakes look like?" They provided the AI with a specific list of error categories and some examples (a "cheat sheet").

The Result: It helped a little bit. The AI found slightly more errors, especially the deep cultural ones. But it still didn't catch them all. It's like giving someone a map; they get a little better at navigating, but they still don't have the local intuition of someone who grew up there.

5. The Conclusion

The paper concludes that current AI judges are not ready to be the final authority on cultural quality. They are too focused on surface-level facts and miss the deep, "thick" cultural context that makes a response feel right or wrong to a local person.

To build truly culturally aware AI, we can't just rely on robots checking robots. We need frameworks that understand the depth, history, and "feeling" of a culture, not just the facts.

In short: The paper built a test to see if AI can spot cultural awkwardness. It found that while AI is good at spotting facts, it is currently blind to the subtle, deep cultural vibes that humans understand instinctively.

Technical Summary: JuICE – A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors

1. Problem Statement

As Large Language Models (LLMs) are deployed globally, they are increasingly tasked with culturally embedded activities such as drafting personal communications, brainstorming creative ideas, and storytelling. These tasks require more than factual accuracy; they demand "thick" cultural understanding—contextual appropriateness, symbolic resonance, and tacit expectations that native speakers use instinctively.

Current cultural benchmarks often treat culture as a flat set of facts, relying on fact verification or norm entailment. Furthermore, while the "LLM-as-a-Judge" paradigm has become the dominant scalable approach for evaluation, it remains unclear whether these judges can detect "thick" cultural errors. Such errors are not always explicit norm violations or factual mistakes but manifest as subtle incoherence in setting, connotation, specificity, or missing context. For instance, a story might be factually plausible yet culturally jarring to a local reader (e.g., describing jasmine as a pleasant city smell in Indonesia, where it is associated with funerals). The core problem addressed by this paper is the lack of a benchmark to evaluate whether LLM-judges can reliably identify these nuanced, localized cultural errors in long-form generation.

2. Methodology

Dataset Construction: JuICE

The authors present JuICE (Benchmark for LLM-Judge in Identifying Cultural Errors), a multilingual dataset designed to capture fine-grained cultural and linguistic errors.

Scope: The dataset comprises 1,050 query-response pairs across four countries (United States, South Korea, Indonesia, Bangladesh) and four languages (English, Korean, Indonesian, Bengali).
Tasks: It covers three open-ended task types: story generation, creative ideation, and how-to advice.
Annotation Pipeline: A human-LLM collaborative pipeline was employed involving 44 native speakers and frontier LLMs (gemini-3-pro, gpt-5.1).
1. Stage 1 (Identification): Two native speakers and two LLM-judges independently annotated responses for cultural or linguistic errors, highlighting spans and providing explanations.
2. Stage 2 (Validation): All identified spans underwent cross-validation. Human-detected spans were validated by other annotators, while model-detected spans were validated by humans. An error span was included in the final dataset only if all human validators agreed it was erroneous.
Scale: The process yielded 7,470 span-level annotations across 33,621 sentences.

Error Taxonomy

Building on the "thick evaluation" framework (Qadri et al., 2025), the authors operationalize a taxonomy distinguishing between "thin" (surface-level) and "thick" (interpretive) errors:

Thin Errors: Explicit Linguistic Errors (mechanical mistakes) and Cultural Inaccuracy (verifiable factual errors).
Thick Errors:
- Implicit Linguistic Errors: Failures in tone, register, or sociolinguistic nuance.
- Cultural Incoherence: Contextual clashes of cultural elements.
- Cultural Specificity Error: Misrepresenting boundaries of cultural concepts (over/under-generalization).
- Cultural Connotation Error: Mishandling symbolic weight or emotional resonance.
- Cultural Missingness: Absence of critical cultural context required for completeness.

Evaluation Tasks

Using JuICE, the authors evaluated various state-of-the-art proprietary and open-source LLMs acting as judges on two tasks:

Erroneous Span Detection: The judge identifies specific text spans containing errors. Performance is measured using Intersection over Union (IoU) with a threshold of $t=0.15$ , calibrated against a semantic oracle.
Erroneous Sentence Classification: A binary classification task determining if a specific sentence contains errors, given the full context.

3. Key Results

Performance of LLM-Judges

Overall Difficulty: Even the strongest LLM-judge (gemini-3.1-pro-preview) achieved an F1 score of only 0.52 on the span detection task.
Recall Limitations: Across all evaluated models, the recall for human-detected errors was at most 0.50, indicating that models miss at least half of the errors identified by native speakers.
Thin vs. Thick Disparity: A significant performance gap exists between thin and thick errors.
- Models perform relatively better on Explicit Linguistic Errors and Cultural Inaccuracy (thin categories), with recalls ranging from 0.37 to 0.72.
- Performance drops sharply for thick cultural errors. Recall for Cultural Missingness fell to as low as 0.21–0.35, and Cultural Connotation errors were similarly difficult (0.28–0.60).
Comparison with Prior Methods: Traditional approaches like norm entailment (CultureBank) and claim verification (CaLMQA) failed to capture these errors. For example, norm entailment accepted responses that were factually consistent with general norms but culturally incoherent in specific contexts (e.g., specific ingredient pairings in Korean desserts).

Impact of Guidance

The authors tested whether providing explicit guidance (thick error taxonomy definitions and few-shot examples) improved performance using gemma-4-31B-it.

Results: The combination of taxonomy and few-shot examples improved the F1 score marginally (from 0.4457 to 0.4591).
Recall Gains: This guidance led to a statistically significant increase in recall for thick errors (7.6 percentage point increase) compared to thin errors (3.9 percentage point increase), suggesting that explicit frameworks help models recognize deeper cultural nuances, though they do not fully close the performance gap.

4. Contributions

Novel Evaluation Task: The paper proposes evaluating the cultural competence of LLMs through their judging capabilities, specifically their ability to detect and localize cultural errors in generated text.
JuICE Dataset: The release of a multilingual, multicultural dataset with 7,500 span-level annotations validated by native speakers across Korean, Indonesian, Bengali, and English.
Empirical Findings: The demonstration that current LLM-judges systematically miss thick cultural errors, achieving low F1 scores (0.52) and recall (≤0.50) on human-validated errors, particularly for connotation, specificity, and missingness.

5. Significance and Limitations

Significance:
The findings suggest that robust cultural evaluation must move beyond surface-level detection and factual verification. The paper argues for frameworks that account for the "depth and situatedness of cultural meaning." It highlights a critical gap where LLMs can generate factually plausible text that remains culturally "wrong" to native speakers, a failure that current automated judges struggle to detect.

Limitations:

Subjectivity: Cultural judgments are inherently plural and subjective; the dataset represents a conservative set of consensus-validated errors and may not capture every subtle or debatable error.
Demographics: Annotators were recruited primarily from university-affiliated populations, potentially overrepresenting the error detection capabilities of younger, highly educated speakers.
Cross-Cultural Comparability: Error distributions vary significantly by country and language, meaning scores should not be interpreted as direct measures of a model's understanding of one culture versus another.
Pipeline Reliance: Some post-processing steps (e.g., category assignment) relied on LLM assistance, meaning the labels are structured approximations of human judgment rather than a single authoritative ground truth.

The authors conclude that future work should focus on developing judge models and evaluation protocols that explicitly target thick errors, potentially by incorporating culturally grounded rubrics or retrieval from local knowledge sources.