💬 NLP

Multi-Level Narrative Evaluation Outperforms Lexical Features for Mental Health

This paper introduces a three-level narrative framework and demonstrates that macro-level LLM evaluation of narrative structure significantly outperforms traditional lexical and semantic embedding features in predicting mental health states across 830 Chinese therapeutic texts, highlighting the critical predictive value of global discourse organization over word-level statistics.

Original authors: Yuxi Ma, Jieming Cui, Muyang Li, Ye Zhao, Yu Li, Yixuan Wang, Chi Zhang, Yinyin Zang, Yixin Zhu

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuxi Ma, Jieming Cui, Muyang Li, Ye Zhao, Yu Li, Yixuan Wang, Chi Zhang, Yinyin Zang, Yixin Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: It's Not What You Say, It's How You Tell the Story

Imagine you are trying to understand a person's mental health by reading a diary entry they wrote about a difficult day.

For a long time, scientists tried to figure this out by counting words. They looked for "sad" words, "angry" words, or words about "death." This is like trying to understand a movie just by counting how many times the word "explosion" appears. It tells you something, but it misses the plot.

More recently, computers got smarter. They started looking at the meaning of sentences (like checking if sentence A flows smoothly into sentence B). This is like checking if the movie scenes are edited together nicely.

This paper argues that both of those methods miss the most important part. The researchers found that to truly understand someone's mental state, you need to look at the architecture of the story itself. You need to see if the story has a beginning, a middle, an end, a clear setting, and a logical reason for why things happened.

The Three Layers of Analysis

The researchers built a "three-story building" to analyze 830 Chinese therapeutic writing samples (stories written by people dealing with depression, anxiety, or trauma).

The Ground Floor (Micro-Level): The Word Count
- What it does: It counts specific types of words (e.g., "I," "sad," "yesterday").
- The Analogy: This is like looking at the bricks in a house. You can count how many red bricks there are, but that doesn't tell you if the house is a stable home or a pile of rubble.
- The Result: It helps a little, but it's not the main predictor of mental health severity.
The Second Floor (Meso-Level): The Sentence Flow
- What it does: It checks if one sentence makes sense after the previous one.
- The Analogy: This is like checking if the hallway in the house is straight. If you walk from the kitchen to the bedroom, does the path make sense?
- The Result: Surprisingly, this layer did almost nothing on its own. Just because sentences flow smoothly doesn't mean the whole story is organized.
The Penthouse (Macro-Level): The Story Structure
- What it does: This is where the researchers used a powerful AI (a Large Language Model) to act like a literary critic. Instead of just counting words, the AI asked:
  - "Does this story have a clear setting?"
  - "Is there a problem, a struggle, and a resolution?"
  - "Is the narrator making sense of their experience, or are they just rambling?"
- The Analogy: This is looking at the blueprint of the house. Is the roof on straight? Are the rooms connected logically? Is the foundation solid?
- The Result: This was the winner. The way the story was organized was the strongest signal for predicting depression, anxiety, and trauma.

What the AI Found in the Stories

The researchers didn't just get a score; they found specific "signatures" for different conditions:

For Depression: The stories often lacked time. The writers struggled to put events in a clear order (past, present, future). It was like a movie where the scenes are shuffled randomly.
For Anxiety: The stories often lacked place. The writers struggled to say where things were happening or felt unsure about the details. It was like a movie where the camera is shaky and the location is never clear.
The "Cognitive Bias" Signal: The most powerful predictor wasn't a specific sad word. It was how the writer used logic. People with higher distress tended to tell stories where they blamed themselves for everything or saw the world in "all-or-nothing" terms. The AI could spot these patterns in how the story was built, not just what words were used.

The Main Takeaway

The paper challenges the old idea that we need to count "sad words" to understand mental health.

Instead, it suggests that mental health is about how we organize our experiences.

A healthy mind builds a coherent story with a clear beginning, middle, and end.
A distressed mind often builds a story that is fragmented, confusing, or lacks a clear structure.

The researchers found that if you want to predict how someone is feeling, you shouldn't just ask, "Did you use the word 'sad'?" You should ask, "Did you tell a story that makes sense?"

What This Means (According to the Paper)

For Research: We need to stop treating AI like a simple word counter. We need to use AI as a "structural engineer" to see how the story is built.
For Future Studies: The paper suggests that if we want to help people, we might need to teach them how to structure their stories (e.g., helping a depressed person organize time, or helping an anxious person ground their story in a specific place) rather than just telling them to write more.
The Limitation: The paper notes that this study looked at cross-sectional data (a snapshot in time). It cannot prove that bad stories cause depression, only that they are strongly linked. It also notes that these findings are based on Chinese writing, so we don't know yet if the exact same rules apply to English or other languages.

In short: The structure of the story matters more than the vocabulary. A well-organized story is a sign of a well-organized mind.

1. Problem Statement

Current computational approaches to analyzing therapeutic writing for mental health assessment are theoretically fragmented and often fail to capture the hierarchical nature of narrative construction.

Limitations of Lexical Methods: Dictionary-based tools (e.g., LIWC) capture word frequencies but ignore discourse structure and semantic relationships, which are central to therapeutic change.
Limitations of Embeddings: Distributional semantic embeddings quantify local coherence (sentence-to-sentence similarity) but often conflate this with global narrative organization, failing to distinguish between local flow and the macro-structural construction of meaning.
The Gap: No existing framework maps computational techniques onto the established three-level hierarchy of discourse processing (surface representation, textbase, and situation model) to systematically analyze how narratives are constructed in mental health contexts.

2. Methodology

Dataset

Sample: 830 Chinese therapeutic writing samples (>100 words each) from 830 participants (ages 9–50; 76.4% female).
Context: Data drawn from six diverse interventions (clinical, post-disaster, school-based, online) conducted between 2018–2024.
Conditions: Participants were assessed for Depression (16.5%), Anxiety (12.0%), and Probable PTSD (38.9%) using various standardized instruments (e.g., BDI-II, PHQ-9, GAD-7, PSSI-5). Scores were normalized to a 0–1 range and converted to ordinal severity levels.

Multi-Level Analytical Framework

The authors propose a three-layer framework grounded in discourse processing theory (Kintsch & Van Dijk, 1978):

Layer 1: Micro-level (Lexical Features)
- Method: Uses Simplified Chinese LIWC.
- Features: Captures psycho-affective states via specific word categories: first-person singular pronouns (self-focus), negative emotion words, certainty/discrepancy markers (absolutist thinking), social words, past-tense focus (rumination), death-related terms, and negation.
Layer 2: Meso-level (Semantic Coherence)
- Method: Uses OpenAI's text-embedding-3-small model to generate 1536-dimensional vectors for sentences and the full text.
- Features:
  - Local Coherence: Sentence-to-sentence (s2s) cosine similarity (mean, min, std) to measure propositional flow and thematic shifts.
  - Global Coherence: Sentence-to-document (s2d) similarity to measure thematic alignment and "wandering."
Layer 3: Macro-level (Narrative Evaluation via LLM)
- Method: GPT-4o acts as a structured evaluator (zero-shot) with deterministic sampling ( $T=0$ ) and JSON output to ensure reproducibility.
- Sub-component L3a (Formal Structural Features):
  - Propositional Logic: Decomposes text into semantic units (Actions, Sensory, Emotion, Cognition) to measure cognitive processing and narrative grounding.
  - Rhetorical Structure Theory (RST): Rates rhetorical relations (e.g., Cause, Contrast) on a 0–5 scale for global coherence.
  - Labov's Story Grammar: Rates six structural components (Abstract, Orientation, Complicating Action, Evaluation, Resolution, Coda) to assess macro-structural integrity.
- Sub-component L3b (Clinically-Grounded Dimensions):
  - Evaluates 15 dimensions based on CBT and cognitive therapy models, categorized into: Structural Trauma Processing (avoidance vs. exposure), Cognitive Processing (biases, sense-making), Affective/Agentic Integration, and Global Structural Coherence.
  - Key Distinction: These measure how psychological states are organized in discourse (e.g., the structure of cognitive bias), not just the presence of symptoms.

Evaluation Protocol

Tasks: Continuous symptom regression (Depression, Anxiety) and Multi-class severity classification (Depression, Anxiety, Trauma).
Models: ExtraTreesRegressor for regression; Gradient Boosting and ExtraTrees for classification (with class-balanced weights).
Validation: Stratified 5-fold cross-validation across 7 feature combinations (Baseline $\to$ L1 $\to$ L2 $\to$ L3 $\to$ Combinations).
Metrics: $R^2$ , RMSE, MAE for regression; AUC, Balanced Accuracy, Macro-F1 for classification.
Interpretability: SHAP (SHapley Additive exPlanations) analysis used to determine feature importance.

3. Key Results

Macro-Level Dominance: Layer 3 (Macro-level) alone was the strongest single predictor, approaching the performance of the full model.
- Depression Regression: $R^2 = 0.295$ (L3) vs. $0.207$ (L1) and $-0.014$ (L2).
- Trauma Classification: AUC = $0.692$ (L3) vs. $0.617$ (L1) and $0.484$ (L2).
Ineffectiveness of Isolated Meso-Level: Layer 2 (Semantic Embeddings) showed negligible independent utility, yielding negative $R^2$ scores and near-chance AUC ( $\leq 0.566$ ). This suggests that local semantic similarity does not capture meaningful variance in therapeutic writing without macro-structural context.
Incremental Gains in Integration: While L2 performed poorly alone, adding it to L1 and Baseline improved classification performance incrementally. However, regression performance only saw substantial improvement when Layer 3 was added.
Feature Importance (SHAP):
- Top Predictors: Clinically-grounded narrative dimensions (L3b), specifically cognitive_bias_score and self_reported_distress_score, were the most influential features.
- Structural Predictors: Formal structural features (L3a) like Labovian evaluation and resolution scores were negatively correlated with symptom severity (better structure = lower distress).
- Condition-Specific Signatures:
  - Depression: Favored cognitive composition and temporal organization (consistent with rumination).
  - Anxiety: Prioritized uncertainty markers and spatial grounding (consistent with hypervigilance).

4. Key Contributions

Theoretical Framework: Introduces a unified, three-level computational framework that aligns NLP techniques with established discourse processing theory (Surface, Textbase, Situation Model), bridging the gap between symbolic, distributional, and generative methods.
Empirical Evidence: Demonstrates that narrative organization (macro-structure) is the primary carrier of mental health signals in therapeutic writing, significantly outperforming traditional lexical counting and semantic embeddings.
Methodological Innovation: Proposes using LLMs not as "black-box" classifiers but as structured evaluators to extract theoretically grounded, interpretable dimensions (e.g., RST coherence, Labovian grammar) with external auditability.
Condition-Specific Insights: Identifies distinct narrative signatures for different disorders (temporal disorganization in depression vs. spatial grounding deficits in anxiety), generating testable hypotheses for targeted interventions.

5. Significance and Implications

Paradigm Shift: Challenges the field's historical reliance on word-counting (LIWC), suggesting that how a story is structured is more diagnostic than what words are used.
Clinical Utility: The framework identifies specific structural deficits (e.g., lack of temporal coherence in depression) that can be targeted in therapeutic writing interventions. For instance, scaffolding temporal organization could be a specific intervention for depression.
Interpretability: By moving away from opaque deep learning classifiers to structured LLM evaluation, the study provides a transparent link between linguistic form and psychological state, making AI tools more viable for clinical settings.
Future Research: Highlights the need for longitudinal studies to determine causality (does narrative fragmentation cause distress or vice versa?) and suggests that distributional metrics may be more useful in unconstrained corpora (e.g., social media) than in structured clinical narratives.