Reheat Nachos for Dinner? Evaluating AI Support for Cross-Cultural Communication of Neologisms

1. Problem Statement

Non-native English speakers (NNS) face significant challenges in cross-cultural communication with native speakers (NS) due to neologisms (newly coined words or existing words with new meanings, e.g., "main character energy," "grindset"). These terms are deeply embedded in cultural contexts, slang, and internet discourse, making them difficult for traditional dictionaries to capture accurately.

While NNS increasingly rely on Artificial Intelligence (AI) tools to learn these terms, existing research has not evaluated the efficacy of AI in real-world, informal communication scenarios. Prior studies often rely on constrained formats (e.g., multiple-choice questions) that lack ecological validity. Consequently, it remains unclear:

• Which types of AI support best help NNS learn and use neologisms appropriately?
• Can NNS self-reported confidence accurately predict their actual communicative competence?
• How large is the gap between NNS and NS usage, and can AI bridge it?

2. Methodology

The authors conducted a between-subjects human-subjects study involving 234 NNS participants (native speakers of Spanish, German, or Chinese) and 144 NS evaluators (based in the US).

Experimental Design

The study simulated a realistic scenario where an NNS encounters a neologism in a social media post, learns it, and writes a message to a hypothetical NS friend ("Jo"). The process involved three stages for each of the 8 selected neologisms (e.g., brain rot, canon event, reheat nachos):

1. Learning: Participants learned the word under one of five randomly assigned support conditions.
2. Production: Participants wrote a scenario and a message to "Jo" using the neologism.
3. Comprehension: Participants judged the contextual appropriateness of the word in two provided writing samples.

Support Conditions

Five conditions were compared:

1. Control: No learning support (social media post only).
2. AI Definition: LLM-generated dictionary-style definition.
3. AI Rewrite: LLM-rewritten social media post in simpler English.
4. AI Explanation: LLM-generated explanation including meaning, usage, tone, audience, and connotations (3–5 sentences).
5. Non-AI Dictionary: Official Merriam-Webster dictionary page (serving as a high-quality human-expert baseline).

Evaluation Metrics

• NS-Rated Communicative Competence: NS evaluators rated NNS-produced messages on three dimensions:
  • Well-formedness: Grammar and coherence.
  • Contextual Appropriateness: Sociolinguistic fit.
  • Understandability: Ease of understanding and correctness of meaning.
• NNS Comprehension Competence: The distance between NNS ratings of writing samples and the ground-truth ratings provided by NS evaluators.
• Self-Reported Perceptions: NNS confidence and perceived helpfulness of the support.

3. Key Contributions

• Ecologically Valid Evaluation: The study moves beyond multiple-choice benchmarks to evaluate AI support in a realistic, multi-stage communication task involving production and comprehension.
• Comparative Analysis of AI Modalities: It systematically compares different interaction patterns (Definition vs. Rewrite vs. Explanation) against a non-AI dictionary baseline.
• Gap Analysis: It quantifies the performance gap between NNS and NS writing and analyzes the limitations of current AI tools in bridging this gap.
• Perception vs. Reality: It investigates the reliability of NNS self-assessment as a proxy for actual communicative competence.

4. Key Results

Effectiveness of Support (RQ1)

• AI Explanation is Superior: The AI Explanation condition consistently yielded the highest NS-rated communicative competence across all dimensions (Well-formedness, Contextual Appropriateness, and Understandability) compared to the Control.
• Dictionary Limitations: Surprisingly, the Non-AI Dictionary (Merriam-Webster), despite providing the most comprehensive information, only improved a subset of dimensions. This is attributed to its high information density, which increased cognitive load and task difficulty for participants.
• Comprehension Gap: While AI Explanation improved production quality, it did not significantly improve NNS participants' ability to judge the contextual appropriateness of words in others' writing (Comprehension Competence). The distance between NNS and NS ratings remained high across all conditions.

Perception vs. Competence (RQ2)

• Overestimation: NNS participants consistently overestimated their communicative competence. Their self-reported confidence and perceived helpfulness did not correlate strongly with the actual NS ratings.
• False Confidence: Conditions like AI Rewrite significantly boosted participant confidence and perceived helpfulness (due to simpler language reducing mental burden) but did not result in statistically significant improvements in actual NS-rated competence compared to the Control.

NNS vs. NS Gap (RQ3)

• Significant Discrepancy: A significant gap remains between NNS-produced and NS-produced writing. NS evaluators rated NS-written samples significantly higher in well-formedness and contextual appropriateness than even the best-performing NNS groups (AI Explanation).
• Root Causes: Qualitative analysis revealed that NNS often lack the mechanism to detect imperfect AI outputs. For example, with the term "reheat nachos" (a metaphor for recycling old ideas), AI definitions sometimes provided only literal meanings, leading NNS to use the term incorrectly in a literal sense.

5. Significance and Implications

• Design for Future AI Tools:
  • Richer Context: AI tools should move beyond simple definitions to provide contextualized explanations including tone, audience, and cultural connotations (as seen in the success of the "AI Explanation" condition).
  • Uncertainty Communication: AI must explicitly communicate uncertainty (e.g., "I'm not sure, but...") to prevent NNS from over-relying on potentially hallucinated or incomplete definitions.
  • Iterative Feedback: Single-turn interactions are insufficient. Future tools should support iterative loops where users draft messages and receive targeted feedback.
• Societal Impact: The paper cautions that while AI can help bridge communication gaps, over-reliance on AI for slang may erode the unique social bonding function of shared cultural knowledge. If AI makes slang universally accessible, it may lose its power to signal in-group membership.
• Limitations of Self-Report: The study demonstrates that NNS self-reports are unreliable proxies for actual competence, suggesting that AI learning platforms should incorporate external validation or NS feedback mechanisms rather than relying solely on user confidence.

In conclusion, while AI Explanation is the most effective current support for learning neologisms, current AI tools still fall short of native-level competence. The primary barrier is not just the lack of information, but the NNS's inability to critically evaluate the quality and nuance of the AI's output.