Technical Summary: PsyProbe

Problem Statement

While recent advances in Large Language Models (LLMs) have enabled the development of mental health dialogue systems, existing approaches remain predominantly reactive. They often lack systematic user state modeling, which limits their ability to support proactive therapeutic exploration. In counseling practice, the exploration phase is critical for deepening understanding of a client's core issues through open-ended questioning. However, standard LLMs struggle to move beyond initial descriptions to elicit deeper psychological insights because they do not explicitly track evolving user states or identify information gaps. Furthermore, prior work in Korean counseling contexts has been limited by a reliance on simulator-based synthetic data and a lack of systematic frameworks for tracking diverse personal information as it emerges during dialogue.

Methodology

The authors introduce PsyProbe, a proactive dialogue system designed specifically for the exploration phase of counseling. The system operates through a modular architecture that integrates structured clinical formulation with cognitive error detection.

Core Framework: PPPPPI + Cognitive Errors

PsyProbe models user psychological states using the PPPPPI framework (Presenting, Predisposing, Precipitating, Perpetuating, Protective, Impact), augmented with cognitive error signals. This formulation allows the system to represent and update core issues throughout the conversation, moving beyond simple topic tracking to clinical case formulation.

System Architecture

PsyProbe consists of four primary modules:

1. State Builder: Extracts structured psychological profiles from unstructured user utterances through three stages:

   • Cognitive Error Extraction: Identifies maladaptive thinking patterns (e.g., catastrophizing, overgeneralization, personalization, and selective abstraction via "should/must" statements) based on Beck's and Lefebvre's theories.
   • PPPPPI Alignment: Performs evidence-grounded slot filling to map content to the six clinical dimensions.
   • Theory-of-Mind (ToM) Reasoning: Infers the user's underlying beliefs, desires, and intentions (categorized into Motivational Interviewing phases: Engaging, Focusing, Evoking, Planning) to guide response alignment.

2. Memory Construction: Maintains a two-layer memory system:

   • Turn History: A lightweight record of recent dialogue keywords, events, and emotions.
   • Overall Summary: A clinical snapshot containing the current PPPPPI analysis, core narrative, core emotion, and recurring themes. This summary is updated conservatively based on new evidence.

3. Strategy Planner: Guides the next system turn by:

   • Label Prediction: Predicts Motivational Interviewing (MI) behavioral codes (e.g., Open Question, Complex Reflection) using few-shot prompting with the KMI corpus.
   • Strategy Generation: Converts predicted codes into concrete dialogue strategies, specifying therapeutic goals, key content points, and stylistic constraints.

4. Response Generator: Produces the final response through a three-stage pipeline:

   • Question Ideation: Generates candidate probing questions based on gap scores calculated for each PPPPPI slot. Slots with missing, weak, or outdated information are prioritized. Protective factors are queried conservatively to avoid premature problem-solving.
   • Draft Generation: Composes an initial response integrating selected speech acts with dialogue context, maintaining an empathic and concise tone.
   • Critic and Revision: Evaluates the draft for contextual appropriateness, redundancy, and therapeutic soundness. It iteratively refines the question component (keeping, adding, replacing, or removing questions) to ensure the output supports exploration without repetition.

Key Contributions

1. Structured User State Modeling: The paper proposes a novel approach for exploratory counseling based on the PPPPPI framework, augmented with cognitive error signals to track evolving psychological core issues.
2. Proactive Question Generation Mechanism: The authors develop a mechanism combining gap-based Question Ideation with an iterative Critic-and-Revision process. This supports the generation of contextually appropriate, non-redundant, and proactive questions.
3. Empirical Validation in Real-World Scenarios: The study conducts human evaluations with 27 participants in real-world Korean counseling scenarios, alongside expert evaluation by a certified counselor. This addresses the limitation of prior work relying on synthetic data.

Results

The evaluation compared PsyProbe against a baseline GPT model (prompted with client-centered principles) and a professionally certified human counselor.

• Automatic Evaluation: The full PsyProbe model consistently outperformed the baseline and ablation modes (removing State Builder, Strategy Planner, or Question Ideation/Critic) across lexical overlap (ROUGE, BLEU) and embedding-based similarity (BERT-F1) metrics. Ablation studies indicated that the State Builder and Question Ideation/Critic modules contributed most significantly to performance gains.
• User Evaluation: Participants rated PsyProbe significantly higher than the baseline in Naturalness (approaching human levels) and Engagement Intention (PsyProbe: 1.19 vs. Baseline: 0.33 on a 3-point scale). While empathy scores improved slightly, they remained below human levels, suggesting that simple reflection strategies are insufficient to fully capture human empathic depth.
• Expert Evaluation: A certified counselor evaluated the system on Core Issue Understanding (CIU), Probing Question Quality (PQQ), and Question Rate (QR).
  • Question Rate: PsyProbe achieved a rate of 0.815, comparable to the human counselor (0.830) and significantly higher than the baseline (0.263).
  • Core Issue Understanding: PsyProbe scored 3.37 (on a 5-point scale), a substantial improvement over the baseline (1.153), validating the effectiveness of systematic state modeling.
  • Probing Question Quality: Questions generated by PsyProbe were rated as "natural and appropriate," falling above the "acceptable" threshold.

Significance and Claims

The paper claims that systematic user state modeling and proactive questioning are effective strategies for therapeutic exploration. PsyProbe demonstrates that an LLM-based system can achieve question rates and core issue understanding comparable to professional counselors in the exploration phase. The authors assert that the integration of the PPPPPI framework with cognitive error detection allows the system to ground conversations in explicit user states, thereby deepening therapeutic exploration.

The study acknowledges limitations, including the reliance on multiple LLM calls which may impact scalability, the lack of standardized evaluation protocols for proactive counseling, and the constraints of the 20-minute session duration. The authors conclude that while the system shows promise in naturalness and questioning strategy, future work must focus on enhancing empathic responsiveness and extending the framework to the insight and action phases of counseling.