Cognitive-Linguistic Indicators of Depression in Online Communities: Analysed by DistilBERT and Holographic Reduced Representation

Technical Summary: Cognitive-Linguistic Indicators of Depression in Online Communities

Problem Statement
Depression remains a leading cause of global disability and suicide, yet traditional diagnostic methods (clinical interviews, self-reports) are resource-intensive, difficult to scale, and often inaccessible due to stigma. While social media offers a vast corpus of self-generated text for early detection, existing computational approaches face a dichotomy: deep learning models (e.g., BERT) achieve high accuracy but function as "black boxes" lacking psychological grounding, while traditional psycholinguistic methods (e.g., LIWC) are interpretable but often fail to capture semantic nuance, negation, and context. This study investigates whether combining cognitively grounded linguistic features with transformer-based contextual embeddings can improve automated depression detection while maintaining interpretability.

Methodology
The study utilizes a subset of 1,937 posts from the Kaggle Reddit Suicide and Depression Detection dataset, comprising 1,000 depression-class posts and 937 control posts (after preprocessing to remove URLs, HTML, and short texts). The research compares two classification pipelines:

1. Baseline Model (TF-IDF + Naïve Bayes):

   • Features: Term Frequency–Inverse Document Frequency (TF-IDF) vectors with a vocabulary of 5,000 features and bigram ranges.
   • Classifier: Multinomial Naïve Bayes.
   • Purpose: Serves as a standard, non-contextual text classification benchmark.

2. Hybrid Model (DistilBERT ⊕ HRR + Logistic Regression):

   • Contextual Embeddings: 768-dimensional [CLS] embeddings from DistilBERT (pre-trained, zero-shot, no depression-specific fine-tuning) to capture semantic nuance, negation, and narrative structure.
   • Cognitive-Linguistic Features: Based on Beck's Cognitive Theory of Depression, four specific features are extracted: first-person pronoun density, absolutist word density (e.g., "always," "entirely"), negative emotion ratio, and total word count.
   • Holographic Reduced Representation (HRR): These four features are encoded into 256-dimensional vectors using HRR. Each feature is assigned a random "role" and "filler" vector; the numeric value of the feature scales the filler. These are combined via circular convolution and superposition to create a single structured vector representing the cognitive state.
   • Integration: The 768-dim DistilBERT vector and the 256-dim HRR vector are concatenated into a 1,024-dimensional hybrid representation.
   • Classifier: Logistic Regression.

Key Results
The hybrid DistilBERT ⊕ HRR model significantly outperformed the TF-IDF baseline across all metrics:

• Macro F1 Score: Improved from 0.80 (Baseline) to 0.94 (Hybrid).
• 5-Fold Cross-Validation F1: Improved from 0.83 to 0.92, confirming the stability of the gains.
• AUC (Area Under Curve): Increased from 0.958 to 0.981, with notable improvements in the top-left region of the ROC curve, indicating better sensitivity while maintaining low false positives.
• Precision-Recall: The hybrid model maintained higher precision as recall approached 1.0 compared to the baseline.

Unexpected Findings

• Zero-Shot Transfer: DistilBERT embeddings generated without depression-specific fine-tuning transferred effectively to the task, suggesting that full end-to-end fine-tuning may not be necessary for this specific application, thereby reducing computational costs.
• Token Length Bias: 12% of depression-class posts exceeded DistilBERT's 512-token limit, compared to only 4% of control posts. This indicates that truncation disproportionately affects the target class, potentially introducing systematic bias against longer, more narrative depression posts.

Significance and Contributions
The paper posits that the primary contribution is the integration of two previously disconnected research streams: data-driven deep learning and theory-driven cognitive science.

• Interpretability: The HRR component provides a transparent link between model predictions and specific cognitive constructs (e.g., absolutist thinking), addressing the "black box" limitation of pure transformer models.
• Complementary Signal: The results demonstrate that theory-driven features provide complementary information to contextual embeddings, validating the hypothesis that cognitive grounding enhances model robustness.
• Scalability: By using lightweight DistilBERT embeddings rather than full BERT fine-tuning, the study offers a computationally efficient pathway for scalable mental health screening.

Limitations and Future Directions
The authors acknowledge that the study is limited by the dataset size and the 512-token truncation issue. Future work is motivated to address this bias through chunking strategies or long-context models. Additionally, the authors suggest extending this methodology to larger, longitudinal datasets, other languages, and other cognitive conditions such as anxiety and PTSD, aiming to shift the field toward cognitively grounded, explainable AI in healthcare.