Towards Consistent Detection of Cognitive Distortions: LLM-Based Annotation and Dataset-Agnostic Evaluation

Technical Summary: Towards Consistent Detection of Cognitive Distortions

Problem Statement
Automated detection of Cognitive Distortions (CDs)—biased thinking patterns associated with mental health conditions like depression and anxiety—faces a fundamental challenge: subjectivity. Existing datasets often suffer from low inter-annotator agreement (IAA) among human experts, leading to noisy, unreliable "gold standard" labels. Models trained on such inconsistent data struggle to generalize, often yielding low performance metrics (e.g., weighted F1-scores between 0.2–0.4). Furthermore, comparing models trained on datasets with different characteristics (sizes, label distributions, or annotation qualities) using traditional metrics like F1-score is problematic, as absolute performance values are not directly comparable across heterogeneous benchmarks.

Methodology
The authors propose a two-pronged approach: an LLM-based annotation framework to generate consistent labels and a dataset-agnostic evaluation metric to compare model performance fairly.

1. LLM-Based Annotation Framework:

   • Hypothesis: Instead of relying on a single LLM prediction, the authors hypothesize that multiple independent runs of an LLM on the same text will reveal stable labeling patterns. Consistent recurrence of a label across runs is treated as a signal of internal reliability, serving as a proxy for ground truth in the absence of human consensus.
   • Procedure: The study utilizes the publicly available "Therapist Q&A" dataset (2,530 user-response pairs). The authors employ GPT-4 and GPT-4o models via API with two temperature settings (0.5 and 0.7) to control randomness.
   • Prompting: Two prompt types are used: a Multi-Label Prompt (MLP) allowing unconstrained assignment of CDs, and a Ranked-Label Prompt (RLP) constraining the model to select the most dominant CD (plus an optional secondary one).
   • Consistency Filtering: Each input is processed through five independent runs per configuration. Labels appearing in at least 4 out of 5 runs are selected as final annotations. Data points failing this threshold are marked as ambiguous and excluded to ensure robustness.

2. Dataset-Agnostic Evaluation:

   • Challenge: Standard metrics (e.g., F1-score) cannot fairly compare models trained on datasets with different label distributions or sizes.
   • Solution: The authors introduce a normalized effect size metric based on Cohen's Kappa ($\kappa_{F1}$).
   • Calculation:
     • A Random Baseline ($F1_{random}$) is calculated theoretically based on the dataset's class distribution, representing the expected performance of a model that merely mirrors class imbalance without learning.
     • The Model Performance ($F1_{calculated}$) is the weighted F1-score of the trained model.
     • The metric is computed as: $\kappa_{F1} = \frac{F1_{calculated} - F1_{random}}{1 - F1_{random}}$.
   • This normalizes performance relative to chance, allowing for fair cross-dataset and cross-study comparisons.

Key Contributions

• Reliable Annotation via Repetition: The paper demonstrates that multiple independent LLM runs can produce internally consistent annotations (Fleiss' Kappa = 0.78 for GPT-4 at 0.5 temperature), outperforming the consistency of human annotations on the target dataset.
• Improved Downstream Performance: Models trained on these LLM-generated, consistency-filtered labels consistently achieve higher performance (both in raw F1 and normalized $\kappa_{F1}$) compared to models trained on the original human-labeled "golden" data.
• Evaluation Framework: The introduction of the $\kappa_{F1}$ metric provides a principled method for evaluating and comparing predictive models across datasets with differing characteristics, addressing a gap in NLP evaluation where benchmarks are unstable or non-standardized.

Results

• Annotation Consistency: GPT-4 with a temperature of 0.5 achieved the highest inter-run agreement (Fleiss' Kappa = 0.78), significantly higher than GPT-4o or higher temperature settings. Over 90% of data points had at least one label repeated in all five runs under optimal conditions.
• Downstream Task Performance: In binary, multi-class, and multi-label classification tasks using MentalRoBERTa, models trained on LLM-generated labels (both RLP and MLP) consistently outperformed those trained on the original human-labeled dataset. For instance, in the multi-label task with GPT-4-0.7, the LLM-trained model showed a 33.1% improvement over the random baseline, compared to only 16.1% for the human-labeled model.
• Human Verification: A verification study involving three psychology experts on a subset of disagreeing samples yielded inconclusive results. Experts showed no strong preference for LLM labels over human labels, and inter-expert agreement was low (Fleiss' Kappa = 0.20). This suggests the inherent ambiguity of the task and the dataset itself, rather than a failure of the LLM method.

Significance and Claims
The paper claims that for subjective NLP tasks where ground truth is unattainable and human agreement is low, reliability (internal consistency) is a necessary prerequisite for validity. By treating annotation as a process of repeated querying rather than single-shot prediction, LLMs can generate internally coherent, low-noise training data that improves downstream model generalization.

Furthermore, the authors argue that the proposed $\kappa_{F1}$ metric offers a scalable solution for the broader NLP community to compare models across heterogeneous datasets without relying on a single, potentially flawed benchmark. The study concludes that while the specific dataset used has limitations (e.g., user inputs often describe events rather than explicit distorted thoughts), the methodology of leveraging LLM consistency and effect-size-based evaluation provides a robust foundation for advancing research in mental health NLP and other subjective domains.