VowelPrompt: Hearing Speech Emotions from Text via Vowel-level Prosodic Augmentation

Technical Summary: VowelPrompt: Hearing Speech Emotions from Text via Vowel-level Prosodic Augmentation

Problem Statement
Speech Emotion Recognition (SER) is a complex multimodal task requiring the integration of linguistic content and vocal expressivity, particularly prosodic features like fundamental frequency (F0), intensity, and temporal dynamics. While Large Language Models (LLMs) have shown promise in reasoning over textual transcriptions, they typically neglect fine-grained prosodic information, limiting their effectiveness and interpretability. Existing approaches face a trade-off: audio-language models (e.g., AudioPaLM, Emotion-LLaMA) integrate acoustic encoders but rely on opaque, black-box embeddings that hinder interpretability and text-only deployment; conversely, text-only prompting methods that augment transcripts with prosodic descriptions often use coarse, utterance-level summaries that blur fine-grained emotional cues. Furthermore, phonetic evidence suggests that vowels, as voiced nuclei with stable acoustic properties, are primary carriers of affective prosody, yet modern SER pipelines often extract features uniformly across all phonemes, potentially diluting the discriminative power of vowel-specific cues.

Methodology
The authors propose VowelPrompt, a linguistically grounded framework that augments LLM-based emotion recognition with interpretable, fine-grained vowel-level prosodic cues. The methodology consists of three core components:

1. Vowel-Level Acoustic Feature Extraction:

   • Forced Alignment: Given an utterance and its transcript, a standard forced-alignment pipeline (e.g., Montreal Forced Aligner) obtains precise temporal boundaries for phonemes.
   • Vowel Selection: The system isolates vowel segments (monophthongs and diphthongs) based on the International Phonetic Alphabet (IPA), excluding consonantal regions to focus on voiced, resonance-rich nuclei.
   • Descriptor Extraction: For each vowel segment, the system computes Low-Level Descriptors (LLDs): average pitch (F0) and slope, pitch variation (standard deviation), average intensity (RMS energy) and variation, and segment duration.
   • Normalization and Discretization: Features undergo speaker-level z-normalization and vowel-type normalization to ensure cross-speaker and cross-vowel comparability. Continuous values are then discretized via quantile-based binning into ordinal categories (e.g., "very low," "high").
   • Natural Language Conversion: These discretized values are deterministically mapped into concise natural language descriptions (e.g., "high F0, rising, loud, lengthened") for each vowel.

2. Prompt Construction:
   The generated natural language descriptors are appended to the textual transcript. This allows a text-only LLM to jointly reason over lexical semantics and localized, segment-level prosodic variation without requiring raw audio input at inference.

3. Two-Stage Model Adaptation:
   To adapt LLMs to this enriched input, the authors employ a two-stage training pipeline:

   • Supervised Fine-Tuning (SFT): A cold-start phase where the LLM is fine-tuned on data augmented with vowel-level descriptors. The model learns to generate both the correct emotion label and an interpretable reasoning trace (enclosed in <thought> tags) conditioned on the text and prosodic features.
   • Reinforcement Learning with Verifiable Reward (RLVR): Following SFT, the model is refined using Group Relative Policy Optimization (GRPO). This stage optimizes for two verifiable rewards: accuracy (matching the ground-truth label) and format adherence (ensuring the output contains valid reasoning and answer blocks). A KL penalty constrains updates relative to the SFT reference to prevent drift.

Key Contributions

1. Interpretable Vowel-Level Augmentation: The paper introduces a method to extract pitch, intensity, and duration features specifically from time-aligned vowel segments, converting them into natural language. This bridges the gap between opaque acoustic embeddings and human-readable prompts, enabling LLMs to reason over fine-grained prosodic structure.
2. Two-Stage Training Pipeline: The authors design a specific adaptation regimen combining SFT for alignment and RLVR (via GRPO) to enhance reasoning quality, enforce structured output, and improve robustness across domains and speakers.
3. Multilingual and Cross-Domain Generalization: By anchoring vowel selection to IPA standards and applying language-level normalization, the framework extends to multilingual settings (English, German, French) and demonstrates robustness in cross-domain transfer scenarios.

Experimental Results
Extensive evaluations were conducted across five benchmark datasets: IEMOCAP, MELD, CaFE (French), EmoDB (German), and ASVP-ESD (Multilingual).

• Zero-Shot Performance: On IEMOCAP and MELD, VowelPrompt consistently outperformed both transcript-only baselines and sentence-level prosodic augmentation methods (e.g., SpeechCueLLM). For instance, on IEMOCAP with GPT-4o, VowelPrompt improved Unweighted Accuracy (UACC) by up to 7.80% over the transcript-only baseline.
• Fine-Tuned Performance: Under supervised fine-tuning (SFT) and SFT with GRPO, VowelPrompt achieved state-of-the-art results. On IEMOCAP with LLaMA-3-8B-Instruct, it reached a Weighted F1 of 73.46% (SFT) and 73.02% (SFT & GRPO), surpassing baselines like InstructERC and SALMONN.
• Cross-Domain and Multilingual: In cross-domain transfers (e.g., IEMOCAP to MELD), VowelPrompt showed superior generalization, improving by 5.12% over sentence-level baselines under SFT & GRPO. In multilingual zero-shot settings (CaFE, EmoDB) and supervised multilingual adaptation (ASVP-ESD), the method consistently outperformed competing approaches, demonstrating effective transferability without language-specific supervision.

Significance and Claims
The paper claims that VowelPrompt offers a practical and scientifically transparent approach to prosody-aware emotion recognition. By leveraging well-established phonetic evidence regarding the role of vowels in affective prosody, the framework enables LLMs to achieve high accuracy without relying on raw audio or black-box acoustic encoders. The authors emphasize that the system provides interpretable intermediate representations, explicitly linking acoustic-prosodic patterns to emotional categories. This allows for human-verifiable reasoning processes, addressing the "opacity" problem of deep learning-based SER systems while maintaining the flexibility of text-only deployment. The work demonstrates that fine-grained, linguistically grounded features can significantly enhance the reasoning capabilities of LLMs in multimodal tasks.