Quantifying the perceptual value of lexical and non-lexical channels in speech

Technical Summary: Quantifying the Perceptual Value of Lexical and Non-Lexical Channels in Speech

Problem Statement
Human language is modeled as a predictive processor where expectations of upcoming signals are conditioned on contextual cues. Speech functions as a multi-channel signal comprising a lexical channel (the words spoken) and a non-lexical channel (how they are spoken, primarily prosody). While evidence suggests both channels encode information, the relative effect of the non-lexical channel on human expectations—particularly when the lexical channel is uninformative or ambiguous—remains unclear. Previous attempts to disentangle these effects relied on conspicuous acoustic manipulations (e.g., low-pass filtering, pitch flattening) or required lexically equivalent response sets, which severely limited the ability to study variable lexical content and natural dialogue structures.

Methodology
The authors propose a generalised turn discrimination paradigm to quantify the perceptual value of non-lexical information across unconstrained lexical content.

• Stimuli Construction: Using the Switchboard Telephone Corpus, the authors constructed 120 stimuli, each consisting of three contiguous context turns and five potential responses.
• Dialogue Language Model (LM): To overcome the limitation of requiring lexically equivalent responses, a state-of-the-art BERT-based cross-encoder (fine-tuned on Switchboard) was used to sample plausible, lexically diverse responses. This allowed the creation of stimuli where responses were similarly plausible but varied in wording.
• Experimental Conditions: Participants (native English speakers from North America) rated the plausibility of responses (1–4 scale) under two conditions:
  1. Lexical Condition: Context and responses presented as text transcripts.
  2. Acoustic Condition: Context presented as text, but responses presented as audio recordings spliced onto the final context turn.
• Data Filtering: A subset of 63 "ambiguous" stimuli was identified where participants could not reliably identify the true response in the lexical condition alone.
• Metrics:
  • Accuracy: The frequency of the true response being rated highest (standard and weighted).
  • Entropy Reduction: To account for the inherent optionality in dialogue, the authors employed Ordinal Entropy (measuring dispersion among scores) and Permutation Entropy (measuring agreement at the stimulus level). Lower entropy indicates higher consensus among participants.
• Analysis: Bayesian multilevel regression models (Zero-One Inflated Beta Regression) were used to analyze entropy reduction while controlling for factors such as response length, mean surprisal, and dialogue acts.

Key Results

• Ambiguous Lexical Content: When the lexical channel was uninformative, the acoustic condition yielded significantly higher accuracy (0.25 vs. 0.08) compared to the lexical condition. This indicates that participants successfully leveraged non-lexical cues to constrain expectations and discriminate true continuations.
• Informative Lexical Content: When the lexical content was informative enough to allow discrimination, the addition of non-lexical information (acoustic condition) actually reduced accuracy (0.36 vs. 0.54). Participants were less likely to select the true turn when provided with acoustic cues.
• Consensus (Entropy): Despite the drop in accuracy for informative stimuli, entropy decreased in the acoustic condition for both ambiguous and informative stimuli. This means that while participants might have been less accurate in the acoustic condition for informative turns, they were more consistent in their (incorrect) judgments.
• Cognitive Load: The authors observed that weighted accuracy for attention check questions dropped in the acoustic condition, suggesting a higher cognitive load when processing audio compared to text.

Significance and Claims
The paper claims that its generalised paradigm enables the investigation of non-lexical information effects across a broader, more natural range of lexical content than previously possible. The primary findings challenge the assumption that multi-channel communication always improves discriminative performance. Instead, the authors demonstrate that:

1. Non-lexical information is crucial for constraining expectations when lexical content is ambiguous.
2. Non-lexical information can hinder discriminative performance when lexical content is already informative, likely due to conflicting cues or increased cognitive load.
3. Crucially, the non-lexical channel exerts a consistent perceptual effect regardless of lexical informativeness. Even when leading to incorrect judgments, non-lexical cues cause listeners to converge on similar interpretations (lower entropy).

The authors suggest these findings have implications for learning perceptually-motivated representations of spoken communication and for speech generation, specifically regarding the acceptable production variability across both lexical and non-lexical channels. They conclude that quantifying channel value via entropy reduction offers a novel perspective, revealing that non-lexical information shapes expectations in consistent ways even when it does not improve task accuracy.