⚡ electrical engineering

TRI-DEP: A Trimodal Comparative Study for Depression Detection Using Speech, Text, and EEG

This paper presents TRI-DEP, a systematic trimodal study demonstrating that combining EEG, speech, and text modalities with pretrained embeddings and carefully designed fusion strategies achieves state-of-the-art performance in subject-independent depression detection.

Original authors: Annisaa Fitri Nurfidausi, Eleonora Mancini, Paolo Torroni

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Annisaa Fitri Nurfidausi, Eleonora Mancini, Paolo Torroni

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out if a friend is feeling down. You wouldn't just look at their face; you'd listen to their voice, pay attention to what they say, and maybe even notice if they seem unusually quiet or restless. You are using three different clues to get the full picture.

This paper, titled "TRI-DEP," is about teaching a computer to do exactly that, but for detecting depression. The researchers built a system that acts like a super-observant detective, using three specific "senses":

EEG (The Brain's Electrical Hum): Like a microphone picking up the electrical static of a brain.
Speech (The Voice's Tone): Listening to how someone speaks, not just what they say.
Text (The Words): Reading the actual transcript of what they said.

Here is a simple breakdown of what they did and what they found.

The Problem: The "One-Clue" Trap

Before this study, most computer systems tried to detect depression using only one clue (like just the voice) or two clues (voice + brain waves).

The Issue: It's like trying to solve a mystery with only half the evidence. Also, previous studies were messy—they didn't always test fairly, sometimes letting the computer "cheat" by memorizing the answers instead of learning the patterns.

The Solution: The "Three-Legged Stool"

The researchers created a new system called TRI-DEP. Think of it as a sturdy three-legged stool. If you remove one leg (one type of data), the stool wobbles. But with all three, it stands tall and stable.

They took a dataset of 38 people (some with depression, some healthy) and ran them through a rigorous test. To make sure the computer wasn't cheating, they used a strict rule: The computer had to learn from some people and be tested on completely different people it had never seen before. This ensures the system actually learns to spot depression, not just memorize faces.

The Tools: Old Maps vs. GPS

The researchers tested two ways to process the data:

Handcrafted Features (The Old Map): This is like manually measuring the pitch of a voice or counting the number of times someone pauses. It's done by humans with specific rules.
Pre-trained Embeddings (The GPS): This uses massive AI models that have already "read" millions of books or listened to thousands of hours of brain waves. They already understand the deep patterns.

The Result: The "GPS" (Pre-trained AI) crushed the "Old Map." The AI models that had already learned from huge amounts of data were much better at spotting the subtle signs of depression than the manually measured rules.

The Big Discovery: The "Weak Link" that Saves the Day

Here is the most surprising part of the story:

Speech was the superstar. It was the best single clue.
Text (what they said) was a close second.
EEG (the brain waves) was the weakest. On its own, it was barely better than guessing.

However, when they combined all three, the result was amazing.
Think of it like a football team. You have a star striker (Speech) and a great midfielder (Text). You also have a defender (EEG) who isn't great at scoring goals on their own. But when the defender steps in to block a shot that the other two missed, the whole team wins.

Even though the brain waves (EEG) were weak alone, they provided extra information that the voice and text missed. When the computer combined all three, it reached a state-of-the-art performance (scoring about 86% accuracy), which is the best anyone has achieved on this specific dataset.

The "Fusion" Strategy: How They Combined the Clues

The researchers tried different ways to combine the three clues:

Voting: Letting each clue vote, and taking the majority. (A bit messy).
Weighted Averaging: Giving the strongest clues (Speech and Text) more say, and the weaker clue (EEG) a smaller say.
Bayesian Fusion: A mathematical way of updating the probability of depression as new clues come in.

The Winner: The "Weighted Averaging" and "Bayesian" methods worked best. They gave the most weight to the voice and text, but kept the brain waves in the mix to catch the subtle stuff.

The Catch: Small Sample Size

The study had a limitation: only 38 people. In the world of statistics, this is a tiny group. Because the group was so small, they couldn't prove with 100% mathematical certainty that their new method was statistically better than the old ones (the "p-value" didn't reach the magic number).

However, the trend was clear: Every time they added a third clue, the system got better and more stable. It's like seeing a plant grow every time you water it; even if you can't measure the exact millimeter of growth, you know it's working.

The Takeaway

This paper is a blueprint for the future. It tells us:

Don't rely on just one sense. Combine voice, words, and brain activity.
Use big AI models (pre-trained embeddings) rather than manual rules.
Even a "weak" clue (like EEG) is valuable when combined with strong ones.

The researchers have released their code and data so other scientists can build on this "three-legged stool" to create even better, more reliable tools for helping people with depression.

1. Problem Statement

Depression is a leading global health burden, yet automatic detection remains challenging due to the complexity of clinical symptoms. While multimodal approaches (combining speech, text, and neural signals) show promise, existing research suffers from three critical limitations:

Limited Scope: Most studies focus on bimodal systems (e.g., EEG+Speech or Speech+Text), neglecting the potential of trimodal integration.
Inconsistent Evaluation: Many studies lack transparent data-splitting protocols, often using segment-level splits that cause data leakage (where data from the same subject appears in both training and testing sets), leading to inflated performance metrics.
Feature Ambiguity: There is a lack of systematic comparison between traditional handcrafted features and modern pre-trained embeddings, particularly for EEG and clinical text.

The paper aims to address these gaps by establishing a rigorous, reproducible benchmark for trimodal depression detection using EEG, speech, and text.

2. Methodology

Dataset

Source: The Multi-modal Open Dataset for Mental-disorder Analysis (MODMA).
Cohort: 38 subjects (24 with Major Depressive Disorder, 14 Healthy Controls) who provided both resting-state EEG and structured clinical interview recordings.
Data Modalities:
- EEG: 128-channel resting-state recordings (250 Hz).
- Speech: Audio from 29 question-answer interview segments per subject (~25 mins total).
- Text: Since MODMA lacks transcripts, the authors generated them using WhisperX (ASR) on the Chinese speech data.

Experimental Design & Preprocessing

Validation Protocol: To prevent data leakage, the authors implemented stratified 5-fold subject-level cross-validation. This ensures that all data from a specific subject is either entirely in the training set or entirely in the test set.
EEG Preprocessing: Two distinct pipelines were tested:
1. Standard: 29 channels, 250 Hz, 10s windows (for handcrafted features and LaBraM).
2. CBraMod: 19 channels, 200 Hz, 5s windows (replicating the MUMTAZ dataset preprocessing).
Speech Preprocessing: Resampled to 16 kHz, denoised, and segmented into 5s overlapping windows (50% overlap).
Text Preprocessing: Raw Chinese transcripts tokenized directly.

Feature Extraction

The study compares Handcrafted Features vs. Pre-trained Embeddings:

EEG:
- Handcrafted: Statistical, spectral, and entropy descriptors (10 per channel).
- Pre-trained: LaBraM (Large Brain Model) and CBraMod (fine-tuned on MUMTAZ).
Speech:
- Handcrafted: MFCCs (40) and Prosody+MFCCs (46).
- Pre-trained: XLSR-53 and Chinese HuBERT Large.
Text:
- Pre-trained: Chinese BERT Base, MacBERT, XLNet, and MPNet Multilingual.

Architectures & Fusion

Unimodal Encoders: Various combinations of CNNs, LSTMs, GRUs, and Attention mechanisms were tested to find the optimal architecture for each modality.
Fusion Strategy: The authors employed Decision-Level (Late) Fusion. Each modality is processed independently to generate a prediction probability. These probabilities are then fused using three strategies:
1. Weighted Averaging: Optimized weights via grid search.
2. Bayesian Fusion: Combining likelihood ratios.
3. Majority Voting.

3. Key Contributions

First Systematic Trimodal Study: This is one of the first works to systematically integrate EEG, speech, and text for depression detection, moving beyond the dominant bimodal approaches.
Rigorous Benchmarking: The use of strict subject-level cross-validation and the release of fixed data splits ensures reproducibility and eliminates data leakage issues common in prior literature.
Feature Comparison: A comprehensive evaluation showing that pre-trained embeddings consistently outperform handcrafted features across all three modalities.
Open Source: The authors released the code, model checkpoints, and specific data splits to foster transparency.

4. Results

Unimodal Performance

Speech: The strongest single modality. Chinese HuBERT embeddings with a BiGRU+Attention encoder and CNN classifier achieved F1 = 0.809.
Text: Chinese MacBERT with an LSTM achieved F1 = 0.784.
EEG: The weakest modality in isolation. CBraMod with a CNN achieved the best result but only reached F1 = 0.446 (below random chance for some metrics, though better than others).

Multimodal Performance

Trimodal Superiority: Combining all three modalities yielded the best results.
- Weighted Averaging (EEG+Speech+Text): F1 = 0.864 (±0.095).
- Bayesian Fusion (EEG+Speech+Text): F1 = 0.864 (±0.095).
Bimodal Comparison: The Speech+Text combination was the strongest bimodal pair (F1 = 0.839), outperforming the best unimodal speech model by ~3 percentage points.
Role of EEG: Despite poor unimodal performance, EEG provided complementary information that boosted the trimodal system's performance by ~2.5% over the best bimodal (Speech+Text) system.

Statistical Significance

Due to the small sample size (N=38), no pairwise comparison reached statistical significance at $\alpha=0.05$ (the minimum p-value for the Wilcoxon test with 5 folds is 0.0625). However, the consistent direction of improvement across fusion methods and the reduced variance (stability) in the Speech+Text and Trimodal models provide strong converging evidence for the efficacy of multimodal integration.

5. Significance and Conclusion

The TRI-DEP framework establishes a new state-of-the-art benchmark for multimodal depression detection with an F1-score of 0.864.

Clinical Implication: The study demonstrates that while EEG alone is not a strong predictor, it adds valuable complementary information when combined with speech and text. This supports the clinical reality that depression manifests across multiple biological and behavioral channels.
Methodological Impact: The paper highlights the necessity of subject-level splits to avoid data leakage and proves that pre-trained models (like HuBERT and MacBERT) are superior to handcrafted features for this task.
Future Work: The authors propose an experimental framework for future research, suggesting that future studies should focus on improving ASR accuracy for clinical speech, exploring early/intermediate fusion strategies, and scaling to larger, more diverse cohorts.

In summary, TRI-DEP provides a robust, reproducible, and high-performing foundation for the next generation of automated mental health diagnostic tools.