💬 NLP

VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP

The paper introduces VIVID, the first culturally grounded benchmark for Vietnamese figurative language, which reveals that current state-of-the-art models, including Vietnamese-specialized ones, significantly struggle with nuanced idioms and proverbs due to literal over-interpretation and a lack of cultural competence.

Original authors: Tu Tran Do, Nhat Ngoc Nguyen, Khanh-Tung Tran, Hoang D. Nguyen, Tu Minh Phuong, Long Hoang Dang

Published 2026-08-05

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Tu Tran Do, Nhat Ngoc Nguyen, Khanh-Tung Tran, Hoang D. Nguyen, Tu Minh Phuong, Long Hoang Dang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand human jokes, sarcasm, and old-fashioned sayings. This isn't just about knowing what words mean; it's about understanding the culture behind them. In the world of computer science, this field is called Natural Language Processing (NLP). Think of NLP as the bridge between human speech and computer code. For a long time, scientists have been building bridges for big, popular languages like English, but many smaller or culturally rich languages have been left on the other side of the river. A key challenge in this field is "figurative language"—phrases where the words don't mean what they literally say. For example, if someone says, "It's raining cats and dogs," a computer that only knows literal definitions might start looking for falling animals! To solve this, researchers need special tests, or "benchmarks," to see if their AI models can actually get the joke or the lesson behind the words, rather than just guessing based on patterns.

Enter VIVID, a new and colorful tool created by researchers to test how well AI understands Vietnamese idioms and proverbs. Think of VIVID as a giant, culturally specific "trivia night" designed specifically for Vietnamese sayings. The researchers gathered 1,636 of these tricky phrases—some are short, rhythmic proverbs full of life lessons, while others are fixed idioms that can't be understood word-for-word. They didn't just throw them into a pile; they organized them like a museum exhibit, tagging each one with "difficulty levels" (like whether it uses ancient words, refers to farming, or relies on sarcasm) and "themes" (like love, criticism, or work).

The team then invited eight of the smartest AI models in the world to take this test. These included both models built specifically for Vietnamese and massive, general-purpose models that speak many languages. They asked the AIs to explain the sayings and to guess what category they belonged to. The results were a bit of a wake-up call. Even the most advanced AI models, including the famous GPT-4o, struggled mightily. On average, they got less than half of the answers right. The study found that the AIs often made silly mistakes, like taking a metaphor too literally (thinking a phrase about a buffalo's skin was actually about a person's personality) or missing the sarcastic tone entirely. Surprisingly, giving the AI a few examples to learn from (a technique called "few-shot prompting") didn't always help; in fact, for the top model, it sometimes made things worse by confusing the AI with the wrong style of answer.

The paper suggests that while these AI models are getting better at general language, they still lack "cultural competence." They are like tourists who can read a map but don't understand the local customs. The researchers conclude that current models are missing the deep, nuanced understanding needed to truly grasp the heart of Vietnamese figurative language. VIVID is now available as a public tool, acting as a mirror to show developers exactly where their AI is failing, so they can build systems that don't just speak the language, but truly understand the culture behind it.

Technical Summary: VIVID – A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP

1. Problem Statement

Figurative language understanding remains a critical challenge in Natural Language Processing (NLP), requiring models to integrate linguistic knowledge, cultural context, and pragmatic reasoning. While significant progress has been made in high-resource languages (e.g., English, Korean) and specific Vietnamese tasks like sentiment analysis or natural language inference, a systematic evaluation framework for culturally grounded figurative language in Vietnamese is absent.

Vietnamese idioms and proverbs present unique interpretation difficulties due to:

Archaic Sino-Vietnamese vocabulary rarely used in contemporary contexts.
Subtle pragmatic nuances, such as sarcasm, irony, and evaluative connotations.
Folk knowledge rooted in traditional agricultural practices and customary wisdom.

Existing benchmarks primarily focus on binary idiomaticity detection rather than deep cultural understanding. Furthermore, traditional automatic metrics (e.g., BLEU) fail to capture the semantic nuances of figurative interpretation, while human evaluation is often expensive and unscalable. This gap prevents researchers from assessing whether Vietnamese-specialized or multilingual Large Language Models (LLMs) can accurately interpret these expressions or identifying which linguistic features pose the greatest challenges.

2. Methodology

2.1. Dataset Construction: VIVID

The authors introduce VIVID (Vietnamese Idioms for Validation and Interpretation Depth), a benchmark comprising 1,636 validated Vietnamese idioms and proverbs. The dataset was constructed through a three-stage pipeline:

Data Collection: Sourced from two authoritative dictionaries (Nguyen Lan, 2010; Vu Dung et al., 2000) to ensure linguistic authenticity and capture both standardized forms and regional variants.
Data Categorization: A dual-layer annotation scheme was developed:
- Linguistic Complexity Taxonomy: Organizing expressions into three aspects with five distinct traits:
  - Meaning: Literal expressions (prone to over-interpretation) and Pragmatic nuances (sarcasm/irony).
  - Lexical: Sino-Vietnamese terms and Uncommon vocabulary (archaic words).
  - Cultural Knowledge: Folk knowledge-based expressions (agricultural/cultural practices).
- Semantic Themes: Categorization into seven themes reflecting Vietnamese cultural discourse: Criticism, Life Lessons, Work and Nature, Social Relationships, Love, Virtues, and Other.
Data Validation: A rigorous two-stage process involving automatic annotation (GPT-4.1) followed by independent review and consensus-based adjudication by two native Vietnamese linguistics experts.

2.2. Evaluation Framework

The study employs a comprehensive framework combining generative and discriminative tasks, utilizing an LLM-as-a-Judge approach validated against human judgment.

Generative Task: Models generate explanations for idioms. An LLM (GPT-4.1) acts as a judge to score these explanations on a 0–5 scale based on factual accuracy, completeness, and alignment with native speaker interpretations.
- Prompting Strategy Analysis: Four strategies were compared: Zero-shot, Demonstration (Few-shot), Chain-of-Thought (CoT), and Aspect-based Evaluation.
- Reliability: Aspect-based evaluation achieved the strongest correlation with human judgment (Cohen's $\kappa$ = 0.792), outperforming other strategies.
Discriminative Task: Models perform multiple-choice classification for:
- Topic identification (7 semantic themes).
- Linguistic characteristic identification (5 complexity traits).
- Evaluation uses exact-match scoring to ensure consistency across open and closed-source models.

2.3. Benchmark Models

Eight state-of-the-art models were evaluated, including:

Vietnamese-specialized: VinaLLaMA-7B, Vistral-7B-Chat, GreenMind-14B.
Multilingual Open-source: Llama-4-Scout, Llama-SEA-LION-v3, Qwen-3.
Closed-source: GPT-4o, Gemini Flash 2.5.

3. Key Results

3.1. Performance Gaps

Even state-of-the-art models achieve less than 50% correctness on average.

Multilingual vs. Specialized: Vietnamese-specialized models drastically underperform multilingual systems. For instance, VinaLLaMA-7B scored 0.13, whereas GPT-4o scored 2.46 (on a 0–5 scale).
Model Scale: Larger models generally perform better. Among comparable 14B models, Vietnamese-specialized GreenMind-14B (1.09) performed similarly to multilingual Qwen-3-14B (1.09), suggesting that current language-specific training does not provide a clear advantage for cultural figurative understanding at similar scales.

3.2. Robustness to Complexity

Models struggle significantly with specific complexity traits:

Sino-Vietnamese terms were the least challenging category.
Pragmatic nuances and Folk knowledge posed the greatest difficulties, with models frequently failing to grasp sarcasm or agricultural context.
Systematic Failure Modes: Analysis revealed literal over-interpretation (inferring metaphors where none exist), lexical gaps (mistranslating archaic words), cultural disconnection, and pragmatic flattening (replacing irony with neutral tones).

3.3. Prompting Effects

Few-shot Prompting: Generally improved performance for most models by providing in-context examples.
Anomaly in GPT-4o: Notably, GPT-4o exhibited performance degradation under few-shot prompting. Analysis suggests this is due to stylistic overfitting, where the model mimics the moralizing tone of the examples rather than focusing on semantic fidelity, leading to "didactic" hallucinations and semantic drift.

4. Contributions

The paper makes three primary contributions:

VIVID Benchmark: The first systematic benchmark for Vietnamese figurative language, featuring 1,636 validated idiom-explanation pairs with dual-layer annotations (complexity traits and semantic themes).
Evaluation Protocols: A novel framework combining an LLM-as-a-Judge approach for generative assessment (validated with $\kappa$ = 0.792) and discriminative classification tasks.
Comprehensive Analysis: Empirical evidence demonstrating that model scale matters more than language-specific training for cultural understanding, while exposing persistent gaps in handling pragmatic nuances and folk knowledge.

5. Significance and Limitations

Significance:
VIVID provides an essential tool for advancing figurative language understanding in culturally rich contexts. It exposes the limitations of current LLMs in achieving genuine cultural competence, moving beyond surface-level pattern matching. The findings suggest that developing culturally aware NLP systems requires more than just scaling parameters or fine-tuning on language-specific data; it necessitates addressing deep semantic and pragmatic gaps.

Limitations:

Evaluation Transparency: The reliance on a proprietary closed-source model (GPT-4.1) for the LLM-as-a-Judge framework raises concerns regarding long-term reproducibility and transparency.
Validation Scale: The reliability analysis between the LLM judge and human annotators was conducted on a subset of 200 samples due to the high cost of manual annotation, which may not fully capture the diversity of all linguistic forms in the dataset.

The authors conclude that VIVID establishes a robust foundation for future research aimed at developing NLP systems capable of nuanced, culturally grounded figurative interpretation.