💬 NLP

A Cross-lingual Comparison of Human and Classification Model Entrainment Behavior in Code-switched Speech Settings

This paper presents a cross-lingual analysis of human entrainment in code-switched speech across Mandarin-English, Hindi-English, and Spanish-English dialogues, revealing that while lexical entrainment generalizes, acoustic and stylistic entrainment varies by context, and demonstrating that current classification models fail to prioritize the specific features most salient to human behavior.

Original authors: Debasmita Bhattacharya, Siying Ding, Alayna Nguyen, Julia Hirschberg

Published 2026-07-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Debasmita Bhattacharya, Siying Ding, Alayna Nguyen, Julia Hirschberg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a party where everyone is chatting. You might notice that when two people really click, they start to mirror each other. One person speaks fast, and the other speeds up; one uses a lot of "umms" and "likes," and the other starts doing the same. They might even start using the same slang or matching their voices to sound more alike. In the world of science, this is called entrainment. It's like a subconscious dance where people unconsciously copy each other's style to build a connection, feel understood, and make the conversation flow smoothly.

Scientists have studied this "dance" for a long time, mostly in people speaking just one language, like English. But the world is full of people who mix languages, switching back and forth between, say, Spanish and English, or Hindi and English, in the middle of a sentence. This is called code-switching. The big question is: Does this mirroring dance still happen when people are mixing languages? And here's the twist: We have built fancy computer programs (AI) to detect this mirroring. But do these computers actually understand how humans do it, or are they just guessing based on the wrong clues?

This paper takes a deep dive into that question. The researchers looked at real conversations where people mixed Mandarin with English, Hindi with English, and Spanish with English. They wanted to see if the "mirroring dance" looks the same across these different language pairs. Then, they tested a bunch of computer models to see if the AI was paying attention to the same things humans were.

Here is what they found:

The Human Dance: Mostly the Same, But with Local Flair
When humans mix languages, they do mirror each other, but the way they do it depends on the languages involved.

The Words: Across all three language pairs, people consistently mirrored each other's word choices. If one person used a lot of common words or specific filler sounds, the other person tended to copy that. This part of the dance was very consistent, no matter which languages were being mixed.
The Voice and Rhythm: This is where it got interesting. In the Spanish-English and Mandarin-English groups, people showed very little consistent mirroring of voice pitch or speed. However, in the Hindi-English group, the mirroring was actually much stronger at the turn-level (between immediate speakers), with speakers consistently converging on their voice features. The researchers suggest this difference might be because Mandarin has a unique tonal quality (where the pitch changes the meaning of a word) that makes the "voice dance" different, while Hindi-English speakers align more closely with the patterns seen in Spanish-English. However, it is important to note that even in Hindi-English, this strong mirroring did not extend to the broader conversation level; speakers did not show consistent convergence over the entire dialogue.
The Switching Style: How people switch languages also varied. In Spanish-English and Mandarin-English, people mirrored how often and how they switched languages. But in Hindi-English, the mirroring was much weaker. The authors suggest this might be because Hindi-English speakers switch languages so naturally and frequently that there's less "room" for one person to influence the other's style—it's like everyone is already dancing the same way, so copying doesn't stand out as much.

The Robot Dance: Good at Guessing, Bad at Understanding
The researchers then asked: Do our computer models see the dance the way humans do?

The Result: The computers were actually pretty good at detecting when people were mirroring each other. They could tell the difference between a "mirroring" conversation and a "non-mirroring" one with decent accuracy.
The Catch: However, the computers were prioritizing the wrong clues to make that decision. When the researchers peeked under the hood to see what clues the AI was using, they found a mismatch. The humans were mirroring specific words and switching styles, but the computers were often prioritizing features other than those most salient to human behavior and focusing instead on other, less salient details like tiny fluctuations in voice pitch or intensity.
The Analogy: Imagine a security guard trying to spot a thief. The thief is always wearing a bright red hat (the human clue). But the security guard's camera is so focused on the thief's shoe laces (the computer clue) that it misses the hat entirely. The guard still catches the thief (the model gets the right answer), but it's using the wrong logic.

Why This Matters
The paper suggests that while our AI can spot that a connection is happening, it doesn't truly understand the human way that connection is built. It's like a student who gets the right answer on a math test by memorizing the answer key rather than understanding the formula.

The authors point out that this is a problem for the future. If we want to build AI assistants that can chat naturally with people who mix languages, we need them to understand the real dance, not just the fake one. If the AI relies on the wrong clues, it might fail when it meets a new group of people or a different language mix. The paper doesn't claim to have solved this yet; instead, it suggests that we need to teach our computers to pay attention to the same things humans do if we want them to be truly natural conversational partners.

Technical Summary: A Cross-lingual Comparison of Human and Classification Model Entrainment Behavior in Code-switched Speech Settings

Problem Statement

Conversational entrainment—the subconscious adaptation of communication styles (lexical, acoustic-prosodic, and gestural) to an interlocutor—is well-documented in monolingual and written contexts but remains underexplored in spoken code-switching (CSW). While prior work has examined CSW entrainment, it has largely been restricted to the Spanish-English language pair and written modalities. Furthermore, it is unclear whether machine learning classifiers, often used to detect or generate such stylistic phenomena, rely on the same features that drive human entrainment behavior. This paper addresses two primary questions:

RQ1: To what extent do entrainment patterns in spoken CSW generalize across distinct language pairs (specifically Mandarin-English, Hindi-English, and Spanish-English) with varying typological distances and cultural contexts?
RQ2: To what extent do classifier models attend to the same entraining features used by humans during spontaneous code-switched communication?

Methodology

The study employs a multi-stage approach involving corpus analysis, statistical modeling of human behavior, and machine learning evaluation.

Corpora

The authors analyze three corpora of informal multilingual speech:

Bangor Miami (BM): Spanish-English spontaneous conversations (35 hours, 56 dialogues).
SEAME: Mandarin-English spontaneous dialogues (192 hours, 256 dialogues).
MaSaC: Hindi-English situational comedy television speech (13 hours, 1,190 dialogues).
Note: MaSaC is scripted but treated as a proxy for naturalistic phenomena due to a lack of genre-matched open Hindi-English speech data. The authors manually annotated MaSaC with CSW strategy labels (Insertional, Alternational, Other) to match the granularity of BM and SEAME.

Human Entrainment Analysis

The authors replicate the analytical framework of Bhattacharya et al. (2024a) and Levitan and Hirschberg (2011) to measure entrainment at the turn-level and conversation-level.

Features:
- Lexical: Top-100/25 frequent words, conversation-level frequent words, affirmative cues, fillers, and perplexity-based measures of overall language use.
- Acoustic-Prosodic: Pitch, intensity, jitter, shimmer, harmonics-to-noise ratio (HNR), and speaking rate.
- CSW Style: Presence of CSW, CSW ratio, and CSW strategy type.
Metrics: Entrainment is quantified via proximity (similarity between speakers), convergence (movement toward similarity), and synchrony (temporal coordination). Statistical significance is determined via p-values.

Model Evaluation

The authors evaluate a suite of binary classifiers (both classical ML and Transformer-based) to detect whether a conversation exhibits significant entrainment.

Models: Includes SVC, Decision Trees, Random Forests, MLPs, XGBoost, and various pre-trained embeddings (XLM-R, mBERT, WavLM, mHuBERT) fused with MLPs.
Task: Binary classification (Entraining vs. Non-entraining) at both turn-level and conversation-level granularities.
Analysis:
- Feature Importance: Using built-in attributes and Shapley values to determine which features drive model predictions.
- Ablation Studies: Systematically removing features to measure the impact on model accuracy ( $\Delta$ acc).
Comparison: The models' feature reliance is compared against the statistically salient features identified in the human entrainment analysis.

Key Results

1. Cross-lingual Human Entrainment Patterns (RQ1)

Lexical Entrainment: Generally consistent across language pairs. Mandarin-English and Spanish-English show robust lexical entrainment across all word classes. Hindi-English (MaSaC) shows lexical entrainment for frequent words but lacks it for conversation-level frequent words, affirmative cues, and fillers. The authors attribute the latter to the scripted nature of MaSaC and shorter conversation lengths, noting that these categories may not appear in the dialogue.
Acoustic-Prosodic Entrainment: Shows significant variation.
- Mandarin-English (SEAME): Limited acoustic-prosodic entrainment; significant only in a minority of features.
- Hindi-English (MaSaC): Strong evidence of turn-level proximity and convergence across most acoustic-prosodic features.
- Spanish-English (BM): Mirrors patterns of absence in conversation-level proximity/convergence.
- Hypothesis: The authors suggest Mandarin's unique tonal character and distinct intonation norms may explain the lack of acoustic entrainment compared to Hindi and Spanish.
CSW Style Entrainment:
- Mandarin-English & Spanish-English: Exhibit entrainment over most CSW features (presence, ratio, strategy) via proximity and synchrony.
- Hindi-English: Differs markedly, exhibiting only turn-level convergence and less prominent entrainment overall. The authors hypothesize that because Hindi-English speakers already code-switch prevalently and independently (high M- and I-index), there is less "room" for stylistic influence, creating a ceiling effect.

2. Classifier Model Behavior (RQ2)

Feature Misalignment: While classifiers achieve reasonable accuracy in detecting entrainment, they consistently prioritize features other than those most salient to human behavior.
- Conversation-Level: Models often assign high importance to acoustic-prosodic features or single CSW metrics, ignoring the broader set of human-relevant features. In some cases, models rely on features that provide little discriminative signal for human entrainment.
- Turn-Level: Feature ablation studies reveal that removing specific entrainment features results in near-zero changes in accuracy. This suggests models rely on diffuse, redundant, or spurious signals rather than a coherent, human-grounded feature set.
Granularity Effect: Models perform better at the conversation level than the turn level. The authors suggest that turn-level dynamics are finer-grained and more variable, making them harder to capture via aggregate statistics.

Key Contributions

Cross-lingual Validation: The first study to validate spoken CSW entrainment patterns across Mandarin-English, Hindi-English, and Spanish-English, extending prior Spanish-English findings.
Human-Grounded Framework: A novel approach using human behavioral patterns as a benchmark to evaluate and interpret classifier decision-making in multilingual stylistic contexts.
Model-Feature Discrepancy: Demonstrates that while models can detect entrainment, they do so using different mechanisms than humans, highlighting a gap in understanding why models succeed.
Data Release: Release of a manually annotated version of the MaSaC corpus with new CSW style labels to support future research on spoken CSW.

Significance and Claims

The paper claims that empirically grounded human behavioral patterns provide a valuable lens for interrogating model decision-making. The findings suggest that high classification performance does not imply that a model has captured the underlying phenomenon of entrainment in a way that supports naturalistic generation.

The authors modestly conclude that:

Lexical entrainment generalizes robustly, but acoustic-prosodic and CSW style patterns vary based on linguistic character and cultural norms.
Current models may propagate feature-level misalignment if used to steer generative models, as they rely on signals distinct from human salience.
Future work must investigate whether generative models face similar challenges in code-switched speech and whether interventions (e.g., prompting or training strategies) can encourage human-like entrainment.

The study acknowledges limitations, including the reliance on English-involved CSW pairs, the use of scripted data for Hindi-English, and the simplification of entrainment into binary classification tasks. It emphasizes that statistical significance in corpus analysis is an imperfect proxy for cognitive salience and calls for future work using behavioral data (e.g., listener judgments) to better align models with human interaction.