Far Out: Evaluating Language Models on Slang in Australian and Indian English

1. Problem Statement

While Large Language Models (LLMs) have shown proficiency in Standard English, they exhibit systematic performance gaps when processing non-standard language varieties. Specifically, the ability of LLMs to comprehend variety-specific slang (colloquialisms unique to specific cultural or geographical contexts) remains underexplored.

Existing research often focuses on Standard American English or fails to distinguish between different English varieties. This paper addresses the gap by investigating how well state-of-the-art LLMs understand slang in two distinct varieties:

• Indian English (en-IN): Often spoken as an additional language.
• Australian English (en-AU): Often spoken as a first language.

The core research question is: How well do language models understand lexical variations in language varieties in terms of colloquial expressions or slang?

2. Methodology

Dataset Construction

The authors constructed two complementary datasets to evaluate model performance:

1. WEB (Web-sourced): Contains 377 unique slang phrases (64 en-IN, 313 en-AU) sourced from Urban Dictionary and validated by native speakers. These include definitions and real-world usage examples.
2. GEN (Generated): Contains 1,492 synthetically generated scenarios using Google Gemini Pro 2.5. For each slang phrase in WEB, the model generated four unique, diverse scenarios (with named characters and specific settings) to test if models can recognize slang in novel contexts.

Evaluation Tasks

The study evaluates models on three specific tasks using a "masked" approach where the slang phrase is hidden:

1. Target Word Prediction (TWP): An open-vocabulary task where the model must generate the missing slang phrase from its entire vocabulary.
   • TWP (Standard): No specific instruction on variety.
   • TWP* (Guided): Explicit instruction to generate a phrase suitable for the specific language variety.
2. Target Word Selection (TWS): A discriminative, multiple-choice task where the model selects the correct slang phrase from a set of four options (1 correct, 3 distractors from the same variety).

Models Evaluated

Seven pre-trained models were tested, ranging from 110M to 8B parameters:

• Encoders: BERT-Base, RoBERTa-Large, XLM-RoBERTa-Large.
• Decoders: Granite-4.0-1B, Llama-3.2-3B-Instruct, Olmo-2-7B-Instruct, Qwen3-4b-Instruct.

Metrics

• Accuracy: Exact match with the ground truth.
• Similarity: Cosine similarity between Sentence-BERT embeddings of the predicted and reference phrases.

3. Key Contributions

1. First Comparative Study: The first evaluative study to systematically test LLMs on slang across Indian and Australian English using both encoder and decoder architectures.
2. Novel Datasets: Introduction of the WEB and GEN datasets, comprising 377 unique slang phrases and 1,492 synthetic scenarios, specifically curated for en-IN and en-AU.
3. Systematic Performance Analysis: Evaluation of 7 models revealing a fundamental asymmetry between generative and discriminative capabilities in variety-specific language.
4. Error Taxonomy: A qualitative analysis categorizing model failures into five types: Literalisation, Generic Substitution, Semantic Drift, Contextual Misinterpretation, and True Failure.

4. Key Results

A. Generative vs. Discriminative Asymmetry

• Generative Failure (TWP/TWP*): Models perform poorly in open-ended generation.
  • Average Accuracy: 0.02 – 0.04.
  • Average Similarity: 0.27 – 0.28.
  • Even with guided prompts (TWP*), performance remains negligible.
• Discriminative Success (TWS): Models perform significantly better when the search space is constrained.
  • Average Accuracy: 0.49.
  • Average Similarity: 0.61.
  • Insight: Models can recognize correct slang when presented with options but struggle to generate it spontaneously.

B. Domain Effects (WEB vs. GEN)

• Models perform better on WEB (real-world data) than GEN (synthetic data).
• For TWP, accuracy drops from 0.04 (WEB) to 0.01 (GEN).
• This suggests potential data contamination (models trained on Urban Dictionary data) and indicates that synthetic data provides a more robust, challenging evaluation of true understanding.

C. Language Variety Effects (en-IN vs. en-AU)

• Models consistently perform better on Indian English (en-IN) than Australian English (en-AU).
• TWS Accuracy: 0.54 (en-IN) vs. 0.44 (en-AU).
• Hypothesis: The pre-training data likely contains more representation or stylistic overlap with Indian English slang compared to Australian slang.

D. Error Analysis

Analysis of the worst-performing 120 instances (using the best model, OLMO) revealed:

• Most Common Errors: Generic Substitution (replacing slang with a standard term), Semantic Drift (getting the topic right but the specific term wrong), and Contextual Misinterpretation.
• True Failures: Rare for en-IN but more common for en-AU in the WEB dataset.
• Conclusion: Models generally understand the semantic meaning (e.g., "pie" for "maggot bag") but fail to retrieve the culturally embedded linguistic form.

5. Significance and Implications

• Limitations of Current LLMs: Despite being trained on massive corpora, state-of-the-art models lack deep cultural and pragmatic understanding of non-standard English varieties. They struggle with the dynamic, community-specific nature of slang.
• Bias and Utility: The inability to handle variety-specific slang poses risks for applications like content moderation, education, and customer service, potentially leading to bias against speakers of these varieties.
• Future Directions: The paper calls for:
  • More inclusive training data that goes beyond Standard American/British English.
  • Development of variety-aware training objectives.
  • Continued learning approaches to adapt to evolving linguistic phenomena.

In summary, the paper demonstrates that while LLMs have improved in recognizing slang when given clues (discriminative tasks), they fundamentally fail to generate or spontaneously understand the cultural nuances of variety-specific slang (generative tasks), highlighting a critical gap in linguistic diversity handling.