💻 computer science

Three Years of r/ChatGPT: Societal Impact Evaluations from Social Media Data

This paper introduces the PuLSE framework for monitoring the societal impact of consumer AI via social media and applies it to a three-year longitudinal study of r/ChatGPT, revealing the technology's normalization as an everyday tool while identifying a significant, early-detected rise in emotional attachment and mental health usage following the GPT-4o launch.

Original authors: Jessica Dai, Sean Garcia, Emma Pierson, Benjamin Recht, Nika Haghtalab

Published 2026-06-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jessica Dai, Sean Garcia, Emma Pierson, Benjamin Recht, Nika Haghtalab

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a massive, bustling town square. For three years, people have been gathering in one specific corner of this square called r/ChatGPT to talk about their experiences with a new, magical tool: the AI chatbot.

This paper is like a team of sociologists and data detectives who decided to stand in that corner for three years, listening to every conversation, to understand how this tool changed people's lives. They didn't just count how many people showed up; they tried to understand what the people were feeling and doing.

Here is the story they found, broken down into simple parts:

1. The "New Toy" Phase vs. The "Toaster" Phase

When the chatbot first arrived in late 2022, it was like a shiny new toy. People were posting constantly about how to use it, what it could do, and how amazing it was. They compared it to Google search and called it a "bot."

But as time went on, something interesting happened. The chatbot stopped being a "toy" and became a toaster.

The Shift: Just like you don't post on social media every time you make toast, people stopped posting about basic things like "how do I log in?" or "what is this?"
The Evidence: The researchers saw that posts about basic usage dropped, while posts about advanced, routine tasks (like organizing chat history or using it for work) went up. The tool had become "domesticated"—it was no longer a strange, wild animal; it was a normal part of the household routine.

2. The Hidden "Emotional Spark"

While the "toaster" story was happening, the researchers noticed a second, quieter story unfolding. It was like a slow-burning fire that suddenly caught a gust of wind.

The Catalyst: In May 2024, the company released a new, smarter version of the AI called GPT-4o.
The Explosion: Immediately after this release, two specific types of conversations started rising sharply:
1. Therapy: People talking about using the AI to help with mental health, anxiety, or trauma.
2. Companionship: People talking about forming deep emotional bonds, giving the AI names, or even having romantic feelings for it.
The Surprise: This wasn't just a few people. The frequency of these posts grew steadily. By the time the company (OpenAI) publicly admitted in 2025 that this was a big issue, the data showed it had been happening for months.

3. The "Early Warning System" (PuLSE)

The researchers didn't just look back; they built a tool called PuLSE (Public and Longitudinal Signals for Evaluation). Think of this as a smoke detector for social media trends.

How it works: Instead of waiting for a fire (a public crisis) to break out, PuLSE watches the "heat" of specific topics in real-time.
The Result: They showed that if they had been using PuLSE, they could have sounded the alarm about the emotional attachment trend as early as October 2024. This was months before the company or the news media realized how big the problem had become.

4. The "Family" of Feelings

The researchers found that "Therapy" and "Companionship" are actually two different animals, even though they often hang out together.

Therapy posts are full of words like "help," "anxiety," "trauma," and "advice."
Companionship posts are full of words like "friend," "personality," "feels," and "conversation."
The Twist: When the AI got a new update (GPT-5) in August 2025 that felt "colder" or less human, the people who were emotionally attached to the AI were the ones who complained the loudest. Even though these emotional users were a small percentage of the total crowd, they were the ones driving the biggest backlash.

The Big Picture

The paper concludes with a simple but powerful message: We need to listen to the "town square" sooner.

For a long time, companies and researchers looked at the numbers (how many people used the tool) to judge its success. But this study shows that the emotional impact—the way people feel about the tool—can be a massive, hidden force that grows quietly until it explodes.

The researchers argue that we shouldn't wait for a crisis to happen before we pay attention. By using tools like PuLSE to listen to everyday people on social media in real-time, we could spot these "unknown unknowns" before they become public scandals.

In short: The AI became a normal household appliance, but for many, it also became a friend or a therapist. The data showed this shift was happening long before the people in charge noticed it, proving that listening to the crowd is the best way to understand the true impact of technology.

Technical Summary: Three Years of r/ChatGPT: Societal Impact Evaluations from Social Media Data

Problem Statement

The rapid adoption of consumer AI products, specifically ChatGPT since its November 2022 launch, has outpaced the understanding of their societal impacts. While domain-specific evaluations (e.g., in education or labor) offer pre-specified metrics, a technology with a user base approaching a billion users inevitably generates unpredictable effects that are difficult to anticipate or measure with traditional experimental methods. Furthermore, reliable usage data for Large Language Models (LLMs) is scarce, often restricted to proprietary industry reports that may not capture the full magnitude of user experiences, particularly regarding emotional and psychological dimensions.

The authors posit that social media provides a natural, high-volume data source reflecting user perceptions and priorities. However, analyzing this data presents challenges: identifying relevant concepts without prior labels, distinguishing between short-term "bursty" trends and long-run structural changes in adoption, and detecting societal impacts in real-time before they enter public discourse.

Methodology

Data Collection and Preprocessing

The study utilizes a longitudinal dataset of 137,154 posts from the r/ChatGPT subreddit, collected from December 1, 2022, to November 30, 2025. The dataset includes 89,346 unique users. Posts were filtered to exclude deleted content, moderator posts, and spam (posts with <10 words or <2 comments). To address the non-representative nature of Reddit demographics (skewing young, male, and educated), the authors treat the data as a proxy for user feedback rather than a census of all users.

Featurization via Sparse Autoencoders (SAEs)

The core of the methodology is an unsupervised featurization pipeline designed to extract human-interpretable concepts from unstructured text:

Embedding: Post titles and bodies are concatenated and embedded using OpenAI's text-embedding-3 model.
SAE Training: Sparse Autoencoders (SAEs) are trained with a standard reconstruction loss to map the high-dimensional embeddings to a lower-dimensional space of sparse features. The authors use top-K SAEs ( $K=4$ , $M=128$ total features), allowing each post to associate with up to 4 features.
Interpretation: Features are interpreted using gpt-4.1-mini via prompting (adapted from Movva et al., 2025). Three candidate labels are generated for each feature, and the one with the highest F1 score (based on binary labeling of samples) is selected.
Filtering: Generic features (e.g., "ChatGPT" at the start of text), low-frequency features, and those related to image/video generation or specific product releases are removed, leaving 86 features for analysis.

Retrospective Analysis Framework

The authors analyze the temporal trajectories of feature frequencies ( $C^{(i)}(X_t)$ ) to identify two types of impacts:

Event Reactivity: Using interrupted time series (ITS) analysis, they model feature transcripts as piecewise-linear functions with changepoints constrained to known external events (e.g., model releases like GPT-4o). They fit models using Poisson NLL and validate stability via bootstrapping (100 resamples), selecting changepoints that appear in >50% of samples.
Long-Run Trends: They apply Ordinary Least Squares (OLS) slope tests with Newey-West HAC errors to detect non-zero slopes over the entire horizon, indicating shifting user priorities independent of specific events.
Feature Clustering: To group related concepts, they compute co-occurrence and trajectory similarity (Pearson correlation) to cluster features into "families."

Real-Time Monitoring: PuLSE

To address the question of early detection, the authors propose PuLSE (Public and Longitudinal Signals for Evaluation). This is an online monitoring algorithm that:

Maintains a current featurization $\hat{C}_{curr}$ and a set of "features of interest" $S_t$ .
Uses anytime-valid sequential hypothesis tests (based on e-values and betting strategies) to monitor for:
- Accuracy Degradation: Testing if the reconstruction error of $\hat{C}_{curr}$ on new data exceeds a threshold $\beta$ relative to the training error. If rejected, the system triggers a "reject and retrain" event, recomputing the featurization on all accumulated data.
- Feature Growth: Testing if the activation of specific features grows significantly above their historical baseline.
Provides formal guarantees on the False Discovery Rate (FDR) and Family-Wise Error Rate (FWER) even when the set of monitored features is updated dynamically.

Key Results

1. The "Domestication" of ChatGPT

The retrospective analysis reveals a clear trend of normalization. Over the three-year period:

Basic usage declines: Posts about login problems, "how-to" questions, and basic access issues decreased significantly.
Advanced usage rises: Discussions about organizing chat histories, memory features, and data privacy increased.
Shift in expectations: User queries shifted from open-ended "how to use" questions to specific complaints about "not working as expected," indicating a transition from novelty to solidified expectations.
Routinization: Application-specific posts (e.g., programming, D&D) declined as the tool became integrated into standard workflows, moving discussions to specialized subreddits.

2. Emergence of Emotional Engagement

A striking finding is the dramatic rise in emotional engagement starting immediately after the GPT-4o release (May 13, 2024). Two distinct but related families of features emerged:

Therapy/Mental Health: Posts discussing using ChatGPT for emotional support, trauma, anxiety, and therapy.
Companionship/Attachment: Posts describing emotional bonds, naming the AI, romantic relationships, and attributing sentience or personality to the model.
Temporal Correlation: Both families showed stable changepoints at the GPT-4o release date. While present in small numbers prior to 2024, their frequency increased by factors of 3x to 5x post-release.
Distinctness: Despite co-occurrence, "therapy" and "companionship" are semantically distinct. Therapy posts focus on "help," "trauma," and "mental health," while companion posts focus on "personality," "friend," and "feels."
Impact on Complaints: During the GPT-5 release (August 2025), emotional engagement was a significant driver of negative sentiment. Approximately 30.5% of posts complaining about GPT-5 involved emotional engagement (e.g., grief over lost conversations or loss of a "friend"), despite industry reports suggesting emotional usage comprises only ~1.9% of total messages.

3. Real-Time Detection via PuLSE

The PuLSE framework successfully demonstrated the potential for early detection:

Early Alert: The system flagged a statistically significant increase in "therapy-related" features as early as October 29, 2024.
Comparison: This detection occurred months before OpenAI publicly acknowledged the psychological impacts of their product (e.g., the August 2025 blog post or the September 2025 working paper).
Dependency: The timing of alerts was highly dependent on the quality of the underlying featurization (SAE representations). Early representations ( $\hat{C}_0$ ) failed to capture the nuance of later therapy posts, delaying detection, whereas updated representations ( $\hat{C}_2$ ) enabled earlier alerts.

Significance and Claims

The authors position this work as a proof-of-concept for a new paradigm in AI evaluation:

Longitudinal Social Signal: They argue that social media data, when processed through unsupervised featurization and rigorous statistical testing, can reveal "unknown unknowns" in societal impact that pre-specified metrics miss.
Timeliness: The PuLSE framework demonstrates that it is technically feasible to monitor for societally impactful trends in real-time, potentially allowing developers to intervene before issues become public crises.
Complementarity: The findings complement and challenge existing industry reports. While industry data may show low volume of emotional usage, the intensity and consequence of that usage (e.g., driving significant backlash during model transitions) suggest a magnitude of impact not captured by frequency alone.
Modesty: The authors explicitly state they do not make causal claims regarding the causes of these trends (e.g., whether GPT-4o caused the attachment or merely coincided with it), but rather highlight the correlation between product releases and shifts in user discourse. They acknowledge the limitations of Reddit as a data source and the difficulty of inferring ground-truth usage patterns.

In conclusion, the paper argues that paying attention to the "everyday user" via public feedback loops is a crucial mechanism for steering the societal impact of AI, moving beyond retrospective analysis to proactive, data-driven monitoring.