Technical Summary: ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions

Problem Statement

Conversational AI systems have achieved massive scale, yet existing datasets and research primarily capture observable user utterances rather than the underlying cognitive processes driving them. Current approaches treat conversation transcripts as the primary unit of observation, limiting the understanding of user intent to what is explicitly verbalized. This creates a "surface-level" view of human-AI interaction that misses the latent motivations, constraints, and reactions that shape user behavior. While recent work has explored machine Theory of Mind (ToM) and user simulations, these efforts often rely on structured, limited mental variables or synthetic data, failing to capture the dynamic, unspoken cognitive context of real-world interactions.

Methodology

The authors introduce ThoughtTrace, a large-scale dataset and framework designed to pair real-world multi-turn human-AI conversations with users' self-reported latent thoughts.

Data Collection

• Participants & Scale: The study involved 1,058 unique users engaging in 2,155 conversations, generating 17,058 interaction turns and 10,174 thought annotations.
• Models: Interactions occurred across 20 different language models, ranging from frontier models (e.g., GPT-5.4, Gemini 3.1 Pro Preview, Claude Opus 4.6) to smaller open-weight models. Users were unaware of the specific model they were interacting with.
• Procedure: Participants completed two open-ended, self-defined tasks (e.g., planning, problem-solving) within a 10-minute window. Crucially, alongside natural chatting, they privately annotated their thoughts in real-time:
  • Reasons: Explanations for why they sent a specific user message (e.g., task motivation, context grounding).
  • Reactions: Internal responses to the assistant's previous message (e.g., satisfaction, dissatisfaction regarding style or content).
• Demographics: The dataset includes rich metadata on age, gender, education, occupation, and AI usage frequency, ensuring a representative spectrum of frequent AI users.

Thought Taxonomy

The authors categorized thoughts into two main forms with specific subtypes:

1. Reasons (7 types): Task Motivation & Goal, Task Continuation, Task Reorientation, Context Grounding & Constraints, Content Expectation, Style Expectation, and Social/Others.
2. Reactions (5 types): Explicit Affirmation, Partial Satisfaction, Content Relevance, Presentation Style, and Scope Fit.

Key Contributions

1. New Data Modality: ThoughtTrace is the first large-scale dataset pairing naturalistic multi-turn conversations with rich, first-person cognitive traces (reasons and reactions) and demographic metadata.
2. Characterization of Thoughts: The paper establishes that thoughts are a distinct data modality with four key properties:
   • Distinct from Messages: Thoughts capture substantial latent information not verbalized in the conversation. Embedding analysis shows significant semantic shifts between messages and their corresponding reasons/reactions.
   • Difficult to Infer: Frontier LLMs (GPT-5.4, Gemini 3.1, Claude Opus 4.6) struggle to infer these thoughts from context alone, achieving low semantic similarity scores (averaging ~2.9 for reasons and ~2.5 for reactions on a 1-5 scale) compared to human annotations.
   • Diverse Content: Thoughts span a wide spectrum of intents, from high-level motivations to specific dissatisfaction signals.
   • Stage-Dependent: Thought dynamics shift across conversation stages (e.g., Task Motivation dominates early turns, while Task Continuation and context-grounding reasons prevail later).
3. Demonstrated Utility: The paper validates the practical value of thoughts for downstream modeling tasks.

Experimental Results

1. Predicting User Behavior

The authors evaluated whether access to thought annotations improves the ability of LLMs to predict a user's next message.

• Setup: Three frontier models were tasked with predicting the next user message given either (a) conversation history only, or (b) conversation history augmented with the user's annotated reasons and reactions.
• Outcome: Thought-augmented prediction significantly outperformed the history-only baseline. The average semantic similarity score increased from 21.6 to 30.6, representing a 41.7% relative gain. This indicates that latent thoughts provide actionable signals for anticipating user needs that are not recoverable from surface dialogue.

2. Improving Model Alignment

The authors tested if thoughts provide better signals for training personalized assistants compared to standard message-based feedback.

• Setup: They generated "rewrites" of unsatisfactory assistant responses.
  • Message-guided: Rewrites based on the user's follow-up message (standard approach).
  • Thought-guided: Rewrites based on the user's explicit dissatisfaction reaction and the underlying reason.
  • These were used to train a Qwen3.5-4B model via Direct Preference Optimization (DPO).
• Outcome: Models trained with thought-guided rewrites achieved a 47.9% win rate on the Arena-Hard benchmark, outperforming the base model (+25.6%) and the WildChat baseline (+6.6%). Notably, thought-guided rewrites outperformed message-guided rewrites on the same dataset by +4.5%, suggesting that thoughts encode richer dissatisfaction signals than users explicitly articulate in follow-up messages. Additionally, thoughts surfaced 2.2× more dissatisfaction instances than messages alone.

Significance and Claims

The paper positions ThoughtTrace as a foundational resource for studying the cognitive dynamics behind human-AI interaction. Its significance lies in three areas:

• User Modeling: It enables systematic study of how conversational context shapes latent thoughts and how these thoughts subsequently drive user utterances, moving beyond surface-level behavior to internal intent.
• Model Training: It provides a new supervisory signal that allows models to learn from and align with users' latent goals, preferences, and reactions, potentially leading to more proactive and personalized assistants.
• Evaluation: It supports benchmarks for thought prediction and offers thought-centered measures of user satisfaction, shifting evaluation from surface-level correctness to latent intent and subjective experience.

The authors acknowledge limitations, including the potential reactivity of asking users to externalize thoughts (which may polarize reasoning) and the fact that the dataset captures only consciously accessible cognition, leaving subconscious processes unobserved. However, they argue that these consciously articulated thoughts represent a valuable, previously unrecorded signal for improving AI assistance.