💻 computer science

Acoustic and Facial Markers of Perceived Conversational Success in Spontaneous Speech

This study analyzes a large corpus of spontaneous Zoom conversations to demonstrate that acoustic and facial entrainment reliably correlates with higher perceived conversational success, identifying key multimodal markers of interaction quality in virtual settings.

Original authors: Thanushi Withanage, Elizabeth Redcay, Carol Espy-Wilson

Published 2026-04-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thanushi Withanage, Elizabeth Redcay, Carol Espy-Wilson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine two people sitting in a virtual living room, chatting on Zoom. Sometimes, the conversation feels like a smooth, jazz improvisation where the musicians are perfectly in sync. Other times, it feels like two people trying to play different songs on the same piano, resulting in a clunky, awkward mess.

This paper is a scientific investigation into what makes that "jazz session" feel successful, specifically when people are talking to strangers online. The researchers wanted to know: Can we measure the "magic" of a good conversation using computers?

Here is the breakdown of their study, translated into everyday language:

1. The Experiment: The "Zoom Date" Dataset

The researchers didn't just guess; they looked at a massive library of over 1,500 real conversations. These were 30-minute chats between strangers (aged 19–66) who had never met before.

The Setup: They were told to just chat and get to know each other (no specific tasks or games).
The Rating: Afterward, the participants filled out surveys rating how much they enjoyed the chat, how friendly they felt, and how well they connected.
The Groups: The researchers split these chats into two groups:
- The "Home Run" Chats (High Success): People who felt a strong connection and had a great time.
- The "Strikeout" Chats (Low Success): People who felt awkward, bored, or disconnected.

2. The Detective Work: What Did They Measure?

The researchers acted like digital detectives, looking for clues in three main areas:

The Rhythm (Turn-Taking & Pauses): Who spoke when? How long did they talk? How long did they wait before speaking?
The Voice (Acoustics): Did their voices match in volume (loudness) or pitch (high vs. low notes)?
The Face (Facial Expressions): Did they smile at the same time? Did they raise their eyebrows together?

3. The Big Discoveries (The "Secret Sauce")

A. The "Longer is Better" Rule (Turns and Pauses)

You might think a successful conversation is a fast-paced ping-pong match with very short sentences. Surprise! The study found the opposite.

The Finding: In "Home Run" chats, people took longer turns to speak and had shorter pauses between them.
The Analogy: Think of a successful conversation like a dance. In a good dance, partners hold the rhythm together and don't stop moving to check their shoes. In a bad conversation, it's like two people trying to dance but constantly tripping over each other's feet (long pauses) or refusing to let the other person finish a step (short, interrupted turns).
The Result: Successful chats had more total turns, but each turn was longer, and the silence between them was brief and comfortable.

B. The "Mirror Effect" (Facial Synchrony)

Did the participants mimic each other's faces?

The Finding: Yes! In successful chats, when one person smiled, the other person's smile often matched in timing.
The Analogy: Imagine two people watching a funny movie. If they are having a great time together, they will both laugh at the exact same moment. If they are disconnected, one might laugh while the other looks confused. The study found that synchronized smiling was a huge marker of a good connection.
The Twist: Interestingly, when people were not having a good time, they tended to synchronize their negative expressions (like frowning) more than the successful groups did.

C. The "Voice Echo" (Acoustic Entrainment)

Did their voices get closer in tone and volume?

The Finding: In successful chats, the speakers' voices naturally drifted closer together. If one person started speaking a bit softer or with a lower pitch, the other person tended to follow suit.
The Analogy: This is like two hikers adjusting their pace. If one hiker slows down to look at a view, the other slows down too so they can walk side-by-side. If they are out of sync, one is sprinting while the other is walking, and they lose each other. The study found that "Home Run" chats had voices that naturally harmonized, while "Strikeout" chats had voices that stayed far apart.

4. Why Does This Matter?

The researchers call this phenomenon "Entrainment." It's the unconscious act of syncing up with another person.

The Takeaway: A successful conversation isn't just about what you say; it's about how you sync up. It's about the rhythm, the shared smiles, and the matching voices.
The Future: The researchers hope to use these "markers" to help people who struggle with social interaction (like those with autism or social anxiety) learn how to "tune" their conversations. Imagine an app that gently nudges you: "Hey, you're speaking too fast, try slowing down to match your friend," or "You haven't smiled in a while, try mirroring their expression."

Summary

In short, this paper proves that great conversations feel like a well-rehearsed duet. The best chats happen when people naturally fall into step with each other—speaking longer, pausing less, smiling together, and matching their voices. When that "sync" is missing, the conversation feels awkward and disconnected.

1. Problem Statement

While entrainment (the phenomenon where interlocutors align their speech patterns, prosody, and non-verbal cues) is well-documented in task-oriented dialogues, there is a significant gap in understanding how these dynamics function in naturalistic, non-task-oriented, and virtual settings.

Context: Most contemporary peer interactions (e.g., job interviews, social networking) occur remotely via platforms like Zoom, yet prior research has largely focused on in-person interactions or specific task-based scenarios (e.g., collaborative problem solving).
Goal: The study aims to identify multimodal markers (acoustic, facial, and turn-taking) that predict Perceived Conversational Success (PCS) in spontaneous, unstructured dyadic conversations between adult peers.

2. Methodology

A. Dataset

Source: The CANDOR Corpus (Conversation: A Naturalistic Dataset of Online Recordings), collected by BetterUp Labs and the University of Pennsylvania.
Composition: >1,500 spontaneous, 30-minute dyadic video/audio conversations conducted over Zoom.
Participants: Previously unacquainted adults (ages 19–66) representing diverse gender, ethnic, and educational backgrounds.
Data Quality: Sessions were filtered to exclude background noise or interruptions. Separate audio channels were recorded for each speaker, eliminating the need for speaker diarization.

B. Defining Perceived Conversational Success (PCS)

Survey Data: Participants completed pre- and post-conversation surveys (229 items). The study focused on 21 constructs related to affect, enjoyment, friendship, and common ground.
Dimensionality Reduction: Principal Component Analysis (PCA) was performed on these constructs. Two latent dimensions were identified, but only PCA1 was selected as the PCS metric due to its superior discriminative utility regarding feature-level entrainment.
Grouping: To create high-contrast groups, the researchers selected conversations at the extreme ends of the PCS distribution (excluding the middle 50%):
- High-Successful Conversations (HSC): PCS $\ge$ 0.9 ( $N=91$ ).
- Low-Successful Conversations (LSC): PCS $\le$ 0.6 ( $N=35$ ).

C. Feature Extraction & Analysis

The study analyzed three primary modalities:

Turn-Taking and Pauses:
- Turns: Defined as contiguous speech bounded by floor changes (excluding backchannels). Metrics included min, max, mean, total duration, and total turn count.
- Pauses: Inter-turn silences > 0.6s. Metrics included min, max, mean, and total pause duration.
Acoustic Features:
- Pitch (F0): Extracted using the PENN (Pitch estimating Neural Network) and normalized to reduce gender variability.
- Intensity: Computed using Praat.
- Entrainment Metric (Proximity): Calculated adjacent-turn distance ( $|f_{current} - f_{partner\_next}|$ ) vs. non-adjacent distance (randomly paired turns). The hypothesis was that in successful conversations, adjacent turns would be closer in value than non-adjacent ones.
Facial Expressions:
- Tool: OpenFace toolkit to extract 17 Facial Action Units (FAUs).
- Entrainment Metric (Synchrony): Unlike turn-based analysis, facial synchrony was measured using non-overlapping 5-second windows across the entire conversation.
- Calculation: Pearson correlation of the same FAU between speakers, converted to Fisher z-transformed values ( $z_f$ ) to normalize distribution.

D. Statistical Analysis

Group Comparisons: Mann-Whitney U tests were used for turn/pause statistics and effect sizes (Cliff's delta) due to non-normal data distributions (Shapiro-Wilk test).
Synchrony: Welch's t-tests were used for Fisher-transformed facial correlations.
Correction: Benjamini-Hochberg False Discovery Rate (FDR) correction was applied to p-values.

3. Key Contributions

Validation of Entrainment in Virtual Settings: Demonstrated that entrainment is reliably detectable in spontaneous, unstructured Zoom conversations, not just task-oriented or in-person settings.
Multimodal Marker Identification: Identified specific features (facial action units, pitch, intensity, turn count, pause duration) that correlate with perceived success.
Methodological Framework: Established a pipeline for analyzing temporal dynamics (turn-based vs. window-based) to distinguish between proximity (magnitude alignment) and synchrony (temporal alignment).

4. Key Results

A. Turn-Taking and Pauses

Turn Duration: HSCs exhibited significantly longer maximum, mean, and total turn durations compared to LSCs.
Pause Duration: HSCs had significantly shorter and less frequent pauses.
Turn Count: Contrary to the idea that "more talking" is always better, HSCs had a lower total turn count than LSCs. This suggests that successful conversations involve fewer, longer, more substantial turns (higher engagement/reciprocity) rather than rapid, fragmented exchanges.
Statistical Significance: Strong class effects were found (e.g., Turn Count: $p = 1.18 \times 10^{-8}$ ).

B. Facial Synchrony

Positive Affect: Synchrony in FAUs associated with smiling (e.g., Lip Corner Puller, Cheek Raiser) was significantly higher in HSCs.
Negative Affect: Synchrony in negative affect markers was low and, when present, tended to be higher in LSCs.
Conclusion: Interpersonal synchrony of positive facial expressions is a robust non-verbal marker of conversational success.

C. Acoustic Proximity (Pitch and Intensity)

Entrainment Detection: In HSCs, the distance between adjacent turns (speaker A's pitch/intensity vs. speaker B's next turn) was significantly smaller than the non-adjacent baseline.
Effect Size: Negative Cliff's delta values indicated stronger entrainment in HSCs.
Specifics: Minimum pitch and minimum/maximum/mean intensity showed significant differences between HSC and LSC groups, confirming that acoustic alignment is a reliable predictor of success.

5. Significance and Future Work

Practical Application: The findings provide actionable markers for developing targeted interventions and training tools (e.g., for individuals with speech difficulties or in professional coaching) to foster more effective and engaging communication.
Theoretical Impact: The study bridges the gap between task-oriented dialogue research and naturalistic social interaction, proving that entrainment mechanisms persist and predict quality in virtual, unstructured environments.
Future Directions: The authors plan to extend this framework to mixed-neurotype interactions (e.g., autistic vs. neurotypical adults) using continuous PCS modeling to better understand communication barriers and facilitators across different cognitive profiles.