Technical Summary: Expectation, Backlash, Recovery, and Excitement

Problem Statement

Conversational AI Systems (CAISes) are not static products; they continuously evolve through model releases, feature updates, safety interventions, and access-policy shifts. While prior research has extensively mapped user perceptions of AI through static snapshots (surveys, cross-sectional social media analysis), these approaches fail to capture the dynamic nature of user sentiment in response to specific system interventions. Existing literature often treats model changes as purely technical updates, overlooking their social and relational impact on users. This study addresses the gap in understanding how user perceptions of CAISes shift dynamically before and after specific model release events across multiple providers.

Methodology

The authors conducted a long-term, large-scale analysis of Reddit discussions from October 2022 to December 2025, covering 20 subreddits and 668,063 posts. The methodology is structured around a "release-centered design" treating model releases as intervention points.

1. Data Construction and Normalization:

• Corpus: Submissions were filtered from 20 relevant subreddits (e.g., r/ChatGPT, r/ClaudeAI, r/grok) focusing on six major providers: OpenAI, Anthropic, Google, xAI, Meta, and DeepSeek.
• Mention Extraction: A taxonomy-guided LLM pipeline was developed to identify and normalize model mentions into a four-level hierarchy: <provider, family, generation, tier>. This taxonomy, derived from the LLM Arena leaderboard, mapped 189 entries across 9 providers. The extractor achieved high precision (F1 > 0.95) in mapping raw text to this schema.
• Filtering: The analysis focused on "single-mention" posts to ensure attribution clarity, resulting in a dataset of 363,546 posts containing 505,250 model mentions.

2. Perception Measurement:

• Sentiment Classification: A human-validated LLM classifier assigned posts to positive, neutral, or negative categories based on explicit textual evidence. The classifier achieved a weighted F1 of 0.82 against human majority labels.
• Thematic Concept Induction: Adapting the LLooM framework, the authors induced 236 interpretable "Canonical Concepts" (e.g., "Coding Productivity," "Expectation Gap," "Forced Upgrade Frustration") from the corpus. These concepts were scored against posts using inclusion criteria, allowing for the tracking of thematic prevalence without relying on pre-defined taxonomies.

3. Intervention Analysis:

• Window Design: For each model release, symmetric pre- and post-release windows were defined based on provider-specific release gap statistics (ranging from 25 days for DeepSeek to 104 days for Google).
• Delta Calculation: The study computed sentiment deltas (change in share of positive/negative posts) and concept deltas (change in concept prevalence) between the pre- and post-release windows. Statistical significance was assessed using chi-squared tests and two-proportion z-tests with Benjamini-Hochberg FDR correction.

Key Results

1. Long-Term Sentiment Trends:

• OpenAI and Google: Both exhibited a long-term shift from neutral to negative sentiment. For OpenAI, neutral sentiment declined by ~19 percentage points (pp) while negative sentiment rose by ~19 pp over the observation period.
• Anthropic: Stood out as the only provider showing a positive trend, with positive sentiment rising from ~20% to ~35% and negative sentiment declining significantly.
• Meta: Showed a neutral-to-positive shift, while xAI and DeepSeek showed no clear long-term directional trends due to shorter observation windows or high volatility.

2. Release-Specific Dynamics:

• Anthropic (Claude 4): Demonstrated the clearest positive intervention profile (+5.3 pp positive, -10.5 pp negative). This was driven by the public release of "Claude Code," which aligned model capability with user workflows (coding/automation), shifting discourse from general productivity to specific coding productivity.
• OpenAI (GPT-5): Triggered the strongest adverse shift (-3.5 pp positive, +10.3 pp negative). The backlash was characterized by "Forced Upgrade Frustration" and "Feature Removal Frustration," where users felt a loss of a "personal companion" and experienced broken workflows.
• OpenAI (GPT-5.1): Showed a partial recovery, reducing the volume of backlash concepts but not fully restoring enthusiasm, indicating a shift from platform-level revolt to narrower product complaints.
• DeepSeek R1: Caused a "demand shock" with a 19-fold increase in post volume. Sentiment was mixed: high praise for engineering capability ("Model Comparison & Evaluation") coexisted with severe frustration regarding access, reliability, and censorship ("Reliability Availability Frustration").
• xAI (Grok 3): Produced a divided reception with simultaneous increases in both positive and negative sentiment. The discourse was heavily politicized, linking model performance to the provider's identity (Elon Musk) and political agendas, alongside standard reliability concerns.
• OpenAI (GPT-4o): Characterized by a massive "Expectation Gap" (+19.4 pp). Users reacted to the disparity between the "omni" demo capabilities (real-time voice, video) and the limited features available at launch, leading to mixed sentiment driven by access constraints rather than model quality alone.

Significance and Claims

The paper claims that model releases are not merely technical updates but "user-facing interventions" that reshape public discussion, sentiment, and expectations. The study demonstrates that:

1. Perceptions are Dynamic: Static snapshots are insufficient; user trust and sentiment are highly sensitive to specific intervention types (e.g., forced upgrades vs. new feature availability).
2. Provider Identity Matters: The reception of a release is deeply influenced by the provider's brand, political stance, and relationship with the user (e.g., the "relational attachment" to GPT-4o vs. the "engineering admiration" for DeepSeek).
3. Product-Model Fit is Critical: Successful releases (like Claude 4) align technical capabilities with clear user workflows, whereas failures often stem from misaligned expectations (GPT-4o) or forced changes that disrupt established user habits (GPT-5).

The authors conclude that providers must treat releases as consequential socio-technical events, aligning capability announcements with actual access, maintaining model continuity during transitions, and monitoring user reactions over time rather than treating reception as a one-time event.

Limitations

The study acknowledges several limitations:

• Data Source: Analysis is restricted to text-based Reddit posts, excluding multimedia content and potentially over-representing technically engaged early adopters.
• AI-Generated Content: The dataset may contain AI-generated or AI-assisted posts, which are difficult to filter.
• Methodological Constraints: The use of LLMs for extraction and classification introduces potential errors, and the batch-based concept induction may introduce redundancy.
• Window Sensitivity: While robustness checks show stability, the choice of symmetric windows may smooth out immediate, short-lived reactions.