"You tell me": A Dataset of GPT-4-Based Behaviour Change Support Conversations

1. Problem Statement

While Large Language Models (LLMs) are increasingly used for mental health and behavior change interventions, current research is predominantly system-focused. Existing studies often evaluate LLM capabilities in isolation or use static datasets (e.g., EMPATHETICDIALOGUES) to train models, neglecting the unpredictability of user utterances and the dynamic nature of real-world interactions.

• Gap: There is a lack of resources to study how real users interact with LLM-based agents in sensitive contexts (like counseling) and how user behavior impacts the generation of LLM texts.
• Challenge: Rule-based conversational agents lack the flexibility of human counselors, while unguided LLMs pose safety risks. Understanding the interplay between user input and LLM output in behavior change scenarios is critical for designing effective, safe systems.

2. Methodology

The authors conducted a preregistered online user study to collect a dataset of interactions between users and two distinct GPT-4-based conversational agents.

A. Study Design & Participants

• Participants: 164 German native speakers recruited via Prolific (ages 19–72; mean age 32.28; 41.4% female).
• Task: Participants were assigned one of three target behaviors to discuss: Healthier Nutrition, Sustainable Living, or Decreased Procrastination.
• Procedure:
  • Pre-Test: Participants completed questionnaires measuring readiness to change, identification with the target behavior, and demographic data.
  • Interaction: Users engaged in a 12-turn conversation with the system.
  • Post-Test: Participants rated the system's helpfulness, empathy, and therapeutic alliance, and completed a post-interaction readiness-to-change assessment.
• Conditions: Participants were randomly assigned to one of two system conditions:
  1. Standard GPT-4: The bot generates responses based solely on the conversation history.
  2. MI-adapted GPT-4: The bot is guided by Motivational Interviewing (MI) principles. A classifier analyzes the user's utterance (Valence, Topic, Reason Type) to select a specific MI strategy (e.g., Reframe, Affirmation). This strategy is injected into the system prompt to guide the GPT-4 generation.

B. System Architecture & Data Collection

• Conversation Structure:
  • Phase 1 (Engaging/Focusing): Rule-based introduction and scaling questions.
  • Phase 2 (Evoking): 5 turns where the bot generates responses (either Standard or MI-adapted).
  • Phase 3 (Conclusion): Rule-based summary and next steps.
• User Feedback Loop: After every bot turn, users rated the response as Helpful, Unhelpful, or Harmful, with an optional text explanation.
• Data Processing:
  • Classification: User utterances were classified by valence (Change, Sustain, Follow/Neutral), topic (Commitment, Taking Steps, Reason), and reason type.
  • Translation: All German data (conversations and explanations) were translated to English using machine translation for broader accessibility.
  • Quality Control: Attention checks were applied; technical errors were logged but retained for analysis.

3. Key Contributions

The paper presents a novel, multi-modal dataset and a framework for analyzing human-LLM interaction in behavior change:

1. The Dataset: A collection of 2,149 conversational turns across 185 chats. It includes:
   • Raw conversation logs (German and English).
   • User language analysis (valence, topic, reason type classifications).
   • User ratings and qualitative feedback on bot turns.
   • Psychometric measures (Therapeutic Alliance, Empathy, Engagement, Readiness to Change).
   • Demographic data.
2. MI-Adapted Prompting Strategy: A technical implementation of using classifiers to dynamically inject Motivational Interviewing strategies into LLM system prompts, bridging the gap between rule-based rigidity and LLM flexibility.
3. User-Centric Analysis: The dataset specifically captures the user's reaction to LLM outputs, allowing researchers to study how user behavior (e.g., "sustain talk" vs. "change talk") influences the interaction flow and perceived efficacy.

4. Results & Observations

While the paper is primarily a resource paper (focusing on the dataset rather than a specific hypothesis test), it reports several key observations from the collected data:

• User Expectations vs. System Design: Despite being informed that the system would not give advice, a significant number of users in the MI-adapted condition expressed frustration ("You tell me") when the bot did not provide direct solutions. This highlights a mismatch between MI principles (user-generated solutions) and user expectations for AI assistants.
• Implicit Information Needs: Users frequently expressed implicit information needs (e.g., needing simple, cheap, tasty food options) without directly asking for them. The dataset captures these nuances, suggesting opportunities for hybrid systems that combine emotional support with information retrieval.
• Adherence to Personas: Although participants were asked to adopt a specific persona, most interacted from their own perspective, indicating that high-fidelity role-play is difficult to enforce in short-term LLM interactions.
• Technical Robustness: The dataset includes instances of technical errors (e.g., duplicate messages), providing a unique resource for studying how users handle LLM failures.

5. Significance and Future Applications

This dataset addresses a critical gap in Human-Computer Interaction (HCI) and Natural Language Processing (NLP) by shifting the focus from system generation quality to user-system interaction dynamics.

• Research Applications:
  • Training Classifiers: Improving models that detect user readiness to change or classify "sustain talk."
  • Evaluating LLM Safety: Analyzing how users perceive and react to potentially harmful or unhelpful LLM outputs in sensitive contexts.
  • Hybrid System Design: Informing the development of systems that can detect implicit information needs and switch between emotional support (LLM) and factual retrieval.
• Design Implications: The findings suggest that future behavior change agents must better manage user expectations regarding advice-giving and be designed to handle the unpredictability of human input, rather than assuming users will strictly follow therapeutic protocols.

In summary, the "YOU TELL ME" dataset provides a foundational resource for moving beyond isolated LLM evaluation toward a holistic understanding of how humans and AI collaborate on complex behavioral goals.