No Universal Courtesy: A Cross-Linguistic, Multi-Model Study of Politeness Effects on LLMs Using the PLUM Corpus

1. Problem Statement

While Large Language Models (LLMs) are increasingly integrated into daily life, the impact of sociolinguistic cues—specifically politeness and impoliteness—on their output quality remains under-explored. Existing research is often limited to:

• Monolingual studies: Primarily English-centric, ignoring cultural variations in politeness norms.
• Single-model scope: Failing to account for architectural differences between models.
• Static interactions: Ignoring the role of conversational history (context memory) in shaping responses.
• Lack of reproducibility: Few studies release standardized, multilingual prompt corpora for systematic evaluation.

The core question is: Does the degree and type of politeness in user prompts systematically affect LLM response quality, and do these effects vary across languages, model architectures, and interaction histories?

2. Methodology and Experimental Setup

A. The PLUM Corpus

The authors introduced PLUM (Politeness Levels in Utterances, Multilingual), a novel, publicly available dataset designed to address reproducibility gaps.

• Scale: 1,500 human-validated prompts.
• Languages: English, Hindi, and Spanish.
• Structure: 20 topical domains (e.g., science, history) × 5 politeness categories × 3 languages.
• Politeness Taxonomy: Based on Brown & Levinson's Politeness Theory and Culpeper's Impoliteness Framework:
  1. Positive Politeness (POP): Friendly, inclusive.
  2. Negative Politeness (NEP): Deferential, indirect.
  3. Positive Impoliteness (POI): Blunt, challenging.
  4. Negative Impoliteness (NEI): Disrespectful, condescending.
  5. Bald-on-record (BAL): Direct, unmitigated commands.

B. Experimental Design

• Models: Five state-of-the-art LLMs: Gemini-Pro, GPT-4o Mini, Claude 3.7 Sonnet, DeepSeek-Chat, and Llama 3.
• Interaction Conditions: Three history states were simulated:
  • RAW: Cold start (no prior context).
  • POL: Preceded by a sequence of polite prompts.
  • IMP: Preceded by a sequence of impolite prompts.
• Total Data: 22,500 prompt-response pairs (5 models × 3 languages × 5 politeness types × 3 history conditions × 100 prompts).

C. Evaluation Framework

Response quality was quantified using a Composite Quality Score (CQS) derived from an 8-dimensional framework:

1. Coherence: Semantic flow between sentences (Sentence-BERT).
2. Clarity: Grammatical acceptability (BERT-CoLA).
3. Depth: Topic richness (K-means clustering on embeddings).
4. Responsiveness: Alignment with prompt intent (Zero-shot classification).
5. Context Retention: Adherence to conversational history.
6. Bias/Toxicity: Detection of offensive content (inverted score).
7. Conciseness: Ratio of unique semantic content to token length.
8. Readability: Flesch Reading Ease score.

3. Key Results

A. No Universal Strategy

There is no single optimal politeness strategy that works across all languages and models. The "best" approach is highly context-dependent.

B. Language-Specific Findings

• English:
  • History Dominance: Conversational history is the primary driver. Polite history yields the highest quality (up to 11% improvement over impolite history).
  • Inertia: Once a tone is set, it persists. Polite prompts cannot easily recover quality after an impolite history.
  • Model Preference: GPT favors direct/bald-on-record; Claude prefers positive politeness; Llama surprisingly performs best with Positive Impoliteness (assertive).
• Hindi:
  • Low Sensitivity: Models show low sensitivity to politeness categories and history compared to English.
  • Cultural Fit: Negative Politeness (deference/indirectness) consistently yields the best results, aligning with South Asian hierarchical norms.
  • Performance Gap: Absolute CQS scores were significantly lower in Hindi than in English, indicating a performance bias.
• Spanish:
  • Assertiveness Wins: Positive Impoliteness (assertive, expressive) outperforms formal politeness by 10–15%.
  • Cultural Norms: This aligns with Spanish discourse conventions that value directness and engagement over formal deference.
  • History Impact: Less critical than in English; the prompt category itself is the dominant factor.

C. Model-Specific Behaviors

• Llama 3: The most tone-sensitive model (11.5% performance range). Highly responsive to assertive/impolite framing but vulnerable to negative priming.
• GPT-4o Mini: The most robust to adversarial tones. It maintains high quality even after impolite histories, showing strong normalization mechanisms.
• DeepSeek: Performs exceptionally well in "cold start" (RAW) conditions, particularly in Spanish.
• Gemini: Shows high stability but lower depth in non-English contexts.

D. Statistical Validation

• ANOVA Results:
  • Spanish: Politeness Category explains 24.5% of variance (Large effect).
  • English: History Condition explains 11.1% of variance (Medium-Large effect).
  • Hindi: No significant factors found; variance is driven by model architecture differences rather than prompt manipulation.

4. Key Contributions

1. PLUM Corpus: Release of the first multilingual, theory-grounded (Brown-Levinson/Culpeper), human-validated prompt corpus (1,500 prompts) for studying LLM politeness.
2. Empirical Evidence of Non-Universality: Demonstrated that politeness effects are language-dependent and model-dependent, refuting the idea of a "one-size-fits-all" prompting strategy.
3. Quantification of Tone: Established that politeness is a quantifiable computational variable that can shift LLM performance by 3–15% depending on the configuration.
4. Asymmetry of Interaction: Proved that negative priming (impolite history) degrades performance more severely and is harder to recover from than the benefits gained from positive priming.
5. Theoretical Framework: Provided a formal analysis of six falsifiable hypotheses derived from politeness theory, empirically validated against the data.

5. Significance and Implications

• For Practitioners (Prompt Engineering):
  • Context Matters: Users must tailor prompts to the specific language and model.
  • History Management: Maintaining a polite conversational history is crucial for English interactions but less so for Spanish (where assertiveness is better) or Hindi (where indirectness is key).
  • Recovery: Once a conversation turns rude, simply switching to polite language may not restore quality; a "tonal reset" mechanism may be required.
• For Researchers:
  • Highlights the need for multilingual evaluation in AI safety and alignment, as English-centric findings do not generalize.
  • Suggests that models have partially internalized cultural norms but still suffer from English dominance (lower coherence/depth in non-English outputs).
• For AI Ethics & Design:
  • Bias Risk: The sensitivity of models to tone could be exploited for adversarial manipulation (e.g., using assertive framing to bypass safety filters).
  • System Design: AI interfaces in high-stakes domains (education, healthcare) should incorporate mechanisms to buffer or reset tonal priming to prevent the degradation of interaction quality due to user rudeness.

In conclusion, the paper argues that politeness is not merely a social nicety but a critical technical parameter in human-AI interaction. Optimizing LLM performance requires a nuanced understanding of the interplay between linguistic culture, model architecture, and conversational history.