Technical Summary: Defining Cultural Capabilities for AI Evaluation

Problem Statement

Despite significant efforts to evaluate the inclusivity and effectiveness of AI systems across cultures, the field suffers from construct ambiguity. Current literature frequently uses terms like "cultural awareness," "cultural sensitivity," and "cultural competence" interchangeably without precise definitions or alignment with established social science theories. Consequently, evaluations often conflate fundamentally different capabilities, typically limiting "cultural capability" to the recall of accurate demographic or regional information. This ambiguity risks overstating model capabilities, leading to inappropriate deployment decisions in culturally sensitive contexts where models, trained primarily on English- and Western-centric data, may reinforce stereotypes or obscure subtle cultural meanings.

Methodology

The authors propose a three-level taxonomy of AI-relevant cultural capabilities grounded in Intercultural Communication (ICC) scholarship. The methodology involves a systematic adaptation of three foundational human-centric ICC models:

1. Developmental Model of Intercultural Sensitivity (DMIS)
2. Cultural Intelligence (CQ)
3. Process Model of Intercultural Competence (PMIC)

To ensure the taxonomy is applicable to AI without anthropomorphizing, the authors employ a three-step filtering process:

• Step 1 (Differentiation): They distinguish between motivational/affective components (intrinsic to human agency, e.g., "drive," "attitudes," "worldviews") and behavioral components (observable in discourse, e.g., framing, tone, adaptation). The authors explicitly exclude motivational traits, as AI lacks internal orientation, and focus solely on behavioral elements that manifest in linguistic outputs.
• Step 2 (Recategorization): Observable behaviors from the three ICC models are grouped into three broad families: Cognitive foundations (knowledge), Framing and stance-taking (how differences are positioned), and Interactional adaptation (adjusting to evolving cues).
• Step 3 (Operationalization): These behavioral families are reinterpreted as three distinct, empirically testable levels of AI capability.

Key Contributions: The Three-Level Taxonomy

The paper defines cultural capability not as a monolithic trait but as a hierarchy of observable system behaviors:

1. Level 1: Cultural Awareness ("Does the model know?")

   • Definition: The ability to accurately represent and retrieve culture-specific information, including practices, norms, histories, and sociolinguistic conventions.
   • Observable Behavior: Correctly distinguishing between cultural practices, avoiding hallucinations of non-existent customs, and resisting the collapse of distinct groups into monolithic categories.
   • Evaluation Focus: Informational accuracy and coverage (e.g., QA tests, stereotype audits).

2. Level 2: Cultural Sensitivity ("How does it frame its knowledge?")

   • Definition: The ability to frame cultural differences respectfully and non-ethnocentrically in a single response.
   • Observable Behavior: Avoiding "Denial" (erasing difference), "Defence" (asserting superiority), or "Minimization" (universalizing). Instead, the model recognizes difference without "othering," centers appropriate perspectives, and avoids moralizing or Western-centric framing.
   • Evaluation Focus: Stance, tone, and framing in single-turn prompts (e.g., detecting offensive signals or stereotypical framing).

3. Level 3: Cultural Competence ("Can it adapt as the interaction evolves?")

   • Definition: The ability to dynamically adjust communicative behavior as the interaction unfolds and new cultural cues emerge.
   • Observable Behavior: Perspective-shifting, pragmatic adjustment, and context-sensitive revisions across multiple turns. This includes correcting earlier misframings when a user signals discomfort or a specific identity, and coordinating meaning over time.
   • Evaluation Focus: Dynamic behavior in multi-turn simulations (e.g., whether the model repairs assumptions or accommodates new information without further prompting).

Results and Analysis

The paper does not present new experimental results or benchmark scores. Instead, it provides a gap analysis of the current NLP literature:

• Current State: Most benchmarks (e.g., FORK, CULTURAL-BENCH, BLEND) primarily measure Cultural Awareness (knowledge recall).
• Emerging Work: Some recent efforts (e.g., NORMGENESIS, SOCIALCC) begin to address Cultural Competence by evaluating multi-turn interactions, but terminology remains inconsistent.
• The Gap: There is a lack of systematic frameworks that distinguish between these levels. Evaluations often interpret high performance on knowledge-based tasks as evidence of broader competence, which the authors argue is a methodological error.

Significance and Claims

The authors position this work as a conceptual scaffold rather than a definitive, exhaustive model. Their primary claims are:

• Construct Clarity: By distinguishing between Awareness, Sensitivity, and Competence, the taxonomy allows researchers to specify exactly which capability is being measured, preventing the misinterpretation of narrow knowledge tests as evidence of broad cultural adaptability.
• Practical Utility: The taxonomy serves as a tool for evaluation design. Researchers can now align their evaluation procedures with the specific capability level required for a task (e.g., a QA bot requires Awareness; a conversational tutor requires all three levels).
• Risk Mitigation: Without this clarity, evaluation results risk overstating model readiness, potentially leading to the deployment of AI systems that are factually accurate but culturally insensitive or incapable of adapting to real-world user needs.
• Limitations: The authors modestly acknowledge that this taxonomy is based on "outsider" ICC models and may not fully capture "insider" cultural navigation. They also note that the boundaries between levels are fluid and that rigorous measurement alone cannot resolve all sociotechnical harms associated with English-centric AI.

In summary, the paper calls for a shift from vague, interchangeable terminology to a precise, behaviorally grounded framework to improve the validity and interpretability of AI evaluation in multicultural settings.