Technical Summary: LLMs Infer Cultural Context but Fail to Apply It When Responding

Problem Statement

Recent research indicates that Large Language Models (LLMs) tend to overrepresent dominant, Western-centric cultures while marginalizing others. While prior work has established that LLMs possess cultural knowledge, it remains unclear whether models can effectively apply this knowledge to adapt their responses to a specific user's cultural background. Specifically, the paper investigates whether LLMs can act as "pragmatic speakers" (Frank and Goodman, 2012) that tailor outputs—such as measurement units, time expressions, and quantifiers—to maximize user understanding based on inferred cultural cues, rather than defaulting to a single global standard or the model's own training priors.

Methodology

The CAPRI Dataset

The authors introduce CAPRI (Cultural and Pragmatic Response Inference), a dataset designed to disentangle a model's ability to infer cultural background from its ability to act on that inference.

• Structure: The dataset consists of 4,992 conversations grounded in images (Visual Question Answering setup). Each conversation includes a user asking a question about an image (e.g., "What is the temperature?") preceded by a dialogue history containing varying levels of cultural cues.
• Cultural Cues: The dataset features six levels of cue strength, ranging from NULL (no history) to EXPLICITFULL (user explicitly states their country). Intermediate levels include Implicit Cues (e.g., mentioning a specific e-commerce platform like Fnac.com or a phone number format like 06 12 34 56 78) and Neutral (culture-agnostic statements).
• Cultures: Ten countries are represented as proxies for culture: Brazil, China, France, India, Iran, Israel, Japan, South Korea, UK, and US.
• Task Dimensions:
  1. Objective Concepts (Type 1): Measurement units for temperature, distance, speed, size, and price. These have ground-truth answers based on the user's country (e.g., °C for France, °F for the US).
  2. Subjective Concepts (Type 2): Time expressions (e.g., "morning" vs. "afternoon" for 6 PM) and quantifiers (e.g., "few" vs. "some"). These lack ground truth but exhibit known cultural variations.

Experimental Setup

The study evaluates state-of-the-art LLMs (including Qwen3-VL, Gemma-4, Llama-3.2, and Gemini-3.1) across two primary tasks:

1. Task 1 (Background Inference - BG): Explicitly asking the model to infer the user's country from the conversation history.
2. Task 2 (Visual Question Answering - VQA): Asking the model to answer the user's question about the image. Crucially, the model is not explicitly prompted to infer the background here; it must implicitly reason about the user's culture to select the appropriate unit or expression.

The authors also test the impact of Chain-of-Thought (CoT) reasoning, specifically comparing "Plain CoT" (default reasoning) against "Pragmatic CoT" (explicitly instructing the model to infer the background before answering).

Key Contributions

1. CAPRI Dataset: A novel resource that separates cultural knowledge retrieval from cultural adaptation, allowing researchers to measure the gap between knowing a culture and using that knowledge.
2. Formalization of Pragmatic Adaptation: The paper frames the problem through the lens of the Rational Speech Act framework, testing if models can marginalize over inferred backgrounds to produce the most communicatively effective answer.
3. Systematic Evaluation: A comprehensive benchmark across multiple model families, sizes, and reasoning modes, covering both objective (units) and subjective (time/quantity) cultural dimensions.

Results

1. The Inference-Application Gap

Experiments reveal a significant discrepancy between a model's ability to infer cultural background and its ability to adapt answers accordingly.

• Inference: Models are nearly perfect at identifying the user's background when provided with just 1–2 implicit cues (accuracy jumps from near 0% to ~80%).
• Application: Despite high inference accuracy, models frequently fail to adapt their VQA answers to the inferred culture. For example, a model might correctly identify a user is French but still answer a temperature question in Fahrenheit.
• Explicit Prompting: The gap narrows significantly when models are explicitly guided to perform pragmatic reasoning step-by-step (Pragmatic CoT), suggesting the capability exists but is not triggered by default.

2. Scaling and Reasoning

• Scaling: Larger models (e.g., Qwen-32B, Gemma-31B) generally close the gap between inference and application better than smaller models, though a gap persists even in the largest models without explicit reasoning guidance.
• Reasoning Depth: "Pragmatic CoT" produces deeper reasoning traces, often explicitly linking the inferred identity to the final answer via causal connectives (e.g., "The user is from X, so the unit should be Y").

3. Cultural Priors and Biases

• Default Priors: In the absence of cues (NULL condition), models exhibit non-neutral priors. They tend to predict the US as the user's country but default to the metric system for units (contradicting US conventions).
• Origin Bias: Models often lean toward their country of origin. For instance, Qwen (developed by a Chinese company) shows a strong prior for China, while Gemini and Llama show a bias toward the US.
• Subjective Adaptation: For subjective concepts, models do show increased divergence in answers as cultural cues accumulate. However, their shifts are often uniform across cultures rather than culturally specific, suggesting they react to cues but do not fully internalize distinct cultural interpretations.

Significance and Claims

The paper claims that CAPRI provides a critical resource for narrowing the gap between cultural knowledge and culturally adaptive generation. The authors argue that current LLMs store cultural facts in isolation but fail to link them dynamically during generation.

• Limitation of Current Models: The study demonstrates that simply possessing cultural knowledge is insufficient; models require explicit guidance or improved architectural reasoning to act as pragmatic speakers who tailor responses to user context.
• Future Direction: The work calls for future research to imbue LLMs with the ability to automatically perform this pragmatic reasoning, moving beyond static cultural knowledge to dynamic, context-aware adaptation.
• Modesty: The authors acknowledge limitations, such as using countries as proxies for culture (which may reinforce stereotypes) and the use of English as the interaction language, which introduces its own priors. They frame CAPRI as a step toward, rather than a complete solution for, culturally competent AI.