Technical Summary: SHARECHAT

Problem Statement

Current academic benchmarks for evaluating Large Language Models (LLMs) rely on uniform, text-only interfaces that obscure how the unique designs and affordances of commercial platforms shape real-world user behavior and system performance. Existing datasets, whether synthetic or real-world (e.g., WildChat, LMSYS-Chat-1M, ShareGPT), suffer from four primary limitations:

1. Artificial Uniformity: They force diverse models into a single generic interface, stripping away platform-specific tools (e.g., citations, thinking traces) that guide user interaction.
2. Loss of Non-Textual Context: They often discard rich structural elements like source citations and reasoning traces, preventing the study of how these features influence prompting strategies.
3. Limited Interaction Depth: Average conversation depths are relatively short, failing to capture the complexity of multi-turn goal evolution and the degradation of model reliability over extended contexts.
4. Observer Bias: Many datasets rely on participants aware they are being monitored (e.g., via consent-based gateways), potentially altering behavior toward higher social desirability compared to authentic, unobserved usage.

Methodology

The authors constructed SHARECHAT, a large-scale corpus of authentic user-LLM conversations collected via a post-hoc public sharing model.

• Data Collection: The dataset comprises 142,808 conversations (660,293 turns) from five major platforms: ChatGPT, Perplexity, Grok, Gemini, and Claude. Data was harvested from publicly shared URLs found on the internet using the Internet Archive (Wayback Machine) to ensure scalability and avoid direct scraping of private accounts.
• Extraction Pipeline: Automated browser control (Selenium) was used to render HTML and parse it into structured JSON. The pipeline captured platform-specific metadata, including:
  • ChatGPT/Grok: Turn timestamps and model versions.
  • Perplexity/Grok: Source citations.
  • Claude/Grok: "Thinking blocks" (intermediate reasoning).
  • Claude: Code artifacts and analysis blocks.
• Privacy Preservation: A rigorous de-identification pipeline using Microsoft's Presidio framework removed Personally Identifiable Information (PII) across multiple languages. An automated LLM-based evaluator (GPT-OSS-120B) and manual review validated the removal process.
• Temporal Scope: Data spans from April 2023 to October 2025, covering over 30 months of usage.
• Analysis Framework: The paper employs three case studies to demonstrate utility:
  1. Conversation Completeness: Using Qwen3-8B to extract user intentions and classify their resolution (complete, partial, incomplete).
  2. Source Grounding: Analyzing citation strategies and domain distributions for search-augmented systems (Perplexity and Grok).
  3. Temporal Analysis: Correlating turn positions with response latencies using turn-level timestamps.

Key Results

Dataset Statistics

• Scale & Depth: SHARECHAT contains 142,808 conversations with an average depth of 4.62 turns, significantly exceeding existing corpora (e.g., WildChat at 1.61, LMSYS-Chat-1M at 2.02).
• Linguistic Diversity: The dataset covers 95 distinct languages, with English (61.8%) and Japanese (18.0%) being dominant.
• Toxicity: SHARECHAT exhibits lower toxicity rates than prior benchmarks. User toxicity is 2.9% (vs. 6.05% in WildChat), and LLM toxicity is 3.2% (vs. 5.18% in WildChat). A moderate-to-strong positive correlation exists between user toxicity and LLM response toxicity across all platforms.
• Topic Distribution: "Seeking Information" is the primary intent (39.6%). However, platform-specific patterns emerge: Perplexity is heavily concentrated on information seeking (63.3%), while Claude shows a higher share of technical help (17.0%).

Case Study Findings

1. Conversation Completeness: Performance varies by platform. Claude (88%) and ChatGPT (83%) show the highest rates of fully satisfying user intentions. Perplexity shows the lowest full-completion rate (67%) but the highest rate of "partial" verdicts, suggesting its role as a search engine providing intermediate steps rather than final answers.
2. Source Grounding:
   • Grok: Heavily relies on social data, with 40,624 references to X (Twitter), nearly triple the citations for Wikipedia.
   • Perplexity: Utilizes a diversified, encyclopedic strategy, with Wikipedia as the leading source, followed by community forums (Reddit) and authoritative repositories (NIH, GitHub).
3. Temporal Dynamics:
   • Latency Trends: ChatGPT shows a negative correlation between turn index and LLM response time (faster responses as conversations lengthen), likely due to caching or context optimization. Conversely, Grok shows a positive correlation (slower responses as context accumulates), indicating increasing computational overhead.
   • User Response Time: While user response time increases with LLM output length, the correlation is weak (near zero), suggesting length is only one factor among many (e.g., task type, time of day) driving user engagement time.

Significance and Claims

The paper claims SHARECHAT is the first share-URL-based corpus to preserve native interaction affordances across five platforms simultaneously. Its significance lies in three areas:

1. Platform-Aware Evaluation: It enables stratified evaluation across product ecosystems, revealing that users treat services as specialized tools (e.g., using Perplexity for search, Claude for coding) rather than interchangeable assistants. This challenges the "universal assistant" assumption in general benchmarks.
2. Rich Metadata for Downstream Tasks: The preservation of citations, thinking traces, and timestamps provides natural benchmarks for Retrieval-Augmented Generation (RAG) accuracy, interaction pacing, and latency-aware evaluation—dimensions absent in existing datasets.
3. Longitudinal and Depth Analysis: The extended conversation depth allows researchers to study phenomena that emerge only over long contexts, such as model reliability degradation and the evolution of user goals, offering supervision signals for reinforcement learning that go beyond binary single-turn correctness.

The authors position SHARECHAT as complementary to, not a replacement for, existing corpora like WildChat. While WildChat mitigates self-selection bias through monitored gateways (introducing observer bias), SHARECHAT avoids observer effects but introduces self-selection bias (visible in lower toxicity and denser outputs). Together, they offer a more complete triangulation of user-LLM interactions.

Limitations

The authors acknowledge four limitations:

1. Self-Selection Bias: Users who share conversations likely filter for interactions they deem valuable, skewing the corpus toward specific interaction types.
2. Platform Imbalance: ChatGPT accounts for over 70% of the data, while Claude is less than 1%. Collection windows also differ, potentially conflating platform differences with temporal changes.
3. LLM-Based Evaluation: Topic and completeness analyses rely on LLMs as proxies for human intent, with validation currently limited to English conversations.
4. Illustrative Scope: The presented analyses are illustrative; future work is needed to fully explore longitudinal dynamics as platform features evolve.