Technical Summary: CBT-Audio

Problem Statement

Current research in applying Artificial Intelligence to Cognitive Behavioral Therapy (CBT) is predominantly limited to text-based interactions. While Large Language Models (LLMs) have been evaluated on therapeutic response generation and knowledge classification, these evaluations rely on text transcripts, creating a significant "blind spot." In clinical practice, therapists rely heavily on non-verbal vocal cues—such as tone, pace, hesitation, tremor, and pitch—to assess patient distress, often interpreting the mismatch between what a patient says and how they say it. Existing datasets and benchmarks (e.g., Cactus, DiaCBT, RealCBT) are either synthetic or text-only, failing to capture the emotional variability and reactivity patterns found in real spoken therapy. Furthermore, while speech-based mental health research exists (e.g., DAIC-WOZ), it largely focuses on diagnostic classification rather than turn-level distress estimation within a therapeutic session. There is a lack of resources to evaluate whether Audio Language Models (ALMs) can effectively utilize vocal cues to estimate patient distress intensity in CBT contexts.

Methodology

Dataset Construction: CBT-Audio

The authors introduce CBT-Audio, a dataset designed to evaluate patient distress estimation from spoken CBT sessions.

• Source Material: The dataset comprises 1,802 patient turns extracted from 96 publicly available CBT educational recordings (approximately 20 hours total) hosted on YouTube. These recordings feature role-plays and case-study walkthroughs where the "patient" is played by an actor, trainee, or clinician, ensuring no real patient identifiers are present.
• Processing Pipeline:
  1. Turn Extraction: WhisperX (using Whisper-large-v3) was used for transcription and forced alignment, while pyannote speaker-diarization-3.1 assigned speaker labels. Consecutive same-speaker segments separated by less than five seconds were merged.
  2. Role Classification: Transcripts were processed by Claude Sonnet 4.5 to distinguish between therapist and patient turns based on conversational patterns.
  3. Filtering: Only patient turns with audio duration $\ge$ 2 seconds were retained, filtering out single-word backchannels.
• Labeling Strategy:
  • Scale: Distress intensity is rated on a 1–5 ordinal scale adapted from the Positive and Negative Affect Schedule (PANAS), ranging from "Calm" (1) to "Overwhelmed" (5).
  • Reference Labels: Generated using a Semantic Similarity Rating (SSR) approach. GPT-audio 1.5 was prompted with the patient's audio clip and the preceding three turns of context to generate an emotional state description. This description was embedded using BGE-m3 and compared via cosine similarity against five anchor definitions. The anchor with the highest similarity determined the label. The process was repeated three times, with the majority vote (median for ties) serving as the final reference label.
  • Validation: A subset of 194 clips was annotated by five independent experts with clinical and research backgrounds. The expert consensus agreed with the reference labels within $\pm$1 level on 81.4% of clips, with a moderate inter-rater agreement (Krippendorff's $\alpha$ = 0.439).

Experimental Setup

• Models: Ten open-source Audio Language Models (ALMs) were evaluated, including SALMONN-7B, MiniCPM-o 2.6, Phi-4-Multimodal, Qwen2.5-Omni, Qwen3-Omni, Kimi-Audio, Ultravox, Audio Flamingo 3, Voxtral-Mini, and Gemma-4-E4B. These models represent diverse architectures (e.g., Whisper-based encoders, custom audio encoders, MoE structures).
• Input Conditions: Each model was tested under three conditions:
  1. Audio-Only (AO): Patient audio clip + 3-turn context (transcript).
  2. Transcript-Only (TO): Patient transcript + 3-turn context.
  3. Audio+Transcript (AT): Patient audio clip + patient transcript + 3-turn context.
• Task: Models were prompted to output a single integer (1–5) estimating the patient's distress intensity.
• Metrics: Primary metric was Mean Absolute Error (MAE). Secondary metrics included 1-off accuracy ($A_1$), Spearman rank correlation ($\rho$), Quadratic Weighted Cohen's $\kappa$ (QWK), and RMSE. Statistical significance was assessed using paired Wilcoxon signed-rank tests on per-turn absolute errors.

Key Results

Overall Performance

• Top Performers: Gemma-4-E4B and Qwen3-Omni-30B-A3B achieved the lowest MAE. Gemma-4-E4B performed best with Audio+Transcript input (MAE 0.720), while Qwen3-Omni performed best with Audio-Only (MAE 0.785).
• Model Variability: Performance varied significantly across model families. For instance, SALMONN-7B was a clear outlier with high error rates (MAE > 1.6), whereas Voxtral-Mini (3.6B parameters) performed comparably to models twice its size. Parameter count did not show a clear correlation with performance within the 7–8B parameter range.

Impact of Input Modality

• Audio vs. Text: Audio-only input did not consistently outperform transcript-only input. In fact, for some models (e.g., Ultravox, SALMONN), transcript-only was the superior setting.
• Audio + Transcript: Combining audio with transcripts yielded the most consistent improvements. In 8 out of 10 model families, adding audio to the transcript improved distress estimation over using the transcript alone. This improvement was statistically significant ($p < 0.05$) for four models: Audio Flamingo 3, Kimi-Audio-7B, Gemma-4-E4B, and Qwen3-Omni-30B-A3B.
• Case Studies: Analysis of specific cases revealed that audio provides the most value when verbal content and vocal delivery diverge.
  • Underestimation by Text: When a patient expressed hopelessness verbally but with a near-tearful, hesitant voice, text-only models underestimated distress, while audio-informed models corrected this.
  • Overestimation by Text: When a patient described anxiety symptoms verbally but with a steady, controlled voice (or laughter), text-only models overestimated distress, while audio cues helped models recognize the lack of immediate acute distress.
  • Alignment: When words and voice aligned, all input conditions performed similarly.

Significance and Claims

The paper claims that CBT-Audio addresses a critical gap in AI for mental health by making spoken patient behavior measurable for evaluation. The authors do not claim to have built a clinical diagnostic system; rather, they demonstrate that:

1. Audio provides complementary information: Vocal cues contain distress signals not captured by text, particularly when there is a mismatch between content and delivery.
2. Current ALMs can leverage these cues: While audio alone is not a universal replacement for text, combining audio with transcripts allows current open-source models to better estimate patient distress intensity.
3. Future Direction: The work supports the argument that future CBT models should be designed and evaluated as models of spoken therapeutic interaction, not just text generation. CBT-Audio provides a foundation for developing safer, audio-aware systems for mental health interaction and evaluating models on their ability to detect the nuances of patient distress.

Limitations

The authors acknowledge several limitations:

• Data Source: All recordings are in English and feature actors/trainees rather than real patients seeking treatment, which may limit the generalizability of emotional ranges compared to real-world therapy.
• Automation: Turn extraction relied on automatic diarization and transcription, which can introduce errors, particularly with overlapping speech.
• Scope: The labels measure perceived distress from observable interaction, not clinical diagnosis or patient self-report. The dataset is intended for research evaluation, not clinical assessment.