💻 computer science

You See It, They Don't: An Exploratory Study of User-to-User Variation in Instagram Comments

This exploratory study reveals that Instagram's new AI-driven comment ranking system produces less variation in visible comments for news posts than non-news posts, with differences driven more by post metrics than by user attributes like gender, political leaning, or location.

Original authors: Brahmani Nutakki, Manon Lilott Kempermann, Ingmar Weber

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Brahmani Nutakki, Manon Lilott Kempermann, Ingmar Weber

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine Instagram as a massive, bustling town square. For years, everyone standing in that square saw the same bulletin board: the top comments on a post were the same for everyone, like a fixed list of the loudest voices.

But in March 2025, the town mayor (Meta) installed a new, invisible AI librarian. This librarian doesn't just show you the same list; they curate a "personalized" list of top comments based on who you are. The worry was: "If the librarian shows a conservative person one set of comments and a liberal person a completely different set, will we stop hearing each other and start living in separate realities?"

This paper is a small-scale experiment to see if that librarian is actually doing that.

The Experiment: Four "Ghost" Shoppers

To test this, the researchers created four "sock-puppet" accounts (fake user profiles that act like real people). They gave these ghosts specific personalities:

A Female Democrat
A Female Republican
A Male Democrat
A Male Republican

They also sent these ghosts to two different "locations" (using digital tunnels called VPNs in New York and Texas) to see if geography changed what they saw.

Then, they sent these ghosts to look at 200 different posts: 100 from news accounts (like CNN or Fox News) and 100 from fun, non-news accounts (like cat lovers or sports fans). They asked: "When these four different ghosts look at the same post, do they see different top comments?"

The Surprising Results

The researchers expected the ghosts to see very different lists, especially on news posts. They thought the AI would aggressively sort comments to match the user's political views.

Instead, they found the opposite:

The "Ghost" Effect was Small: On average, the four different ghosts saw the same top comments 88% of the time. Only about 12% of the comments changed depending on who was looking.
News is More Stable than Fun: Surprisingly, the comments on news posts were less likely to change than the comments on fun posts (like cats or food). The AI seemed to show a more consistent list for news than for entertainment.
Who You Are Matters Less Than the Post: The researchers thought the user's gender or political leaning would be the main reason for the differences. They were wrong. The biggest factors were actually how popular the post was (how many followers the account has) and how many comments it already had.
- Analogy: Imagine a popular restaurant. Whether you are a vegetarian or a meat-eater, if the restaurant is packed and everyone is ordering the same dish, the menu the waiter shows you will look the same. The "crowd" (comment count) matters more than your personal taste.
The Order Changes, Even if the List Doesn't: While the list of comments was mostly the same, the order in which they appeared was very different.
- Analogy: Imagine a playlist. Everyone might hear the same 10 songs, but for the Democrat, the first song is a ballad, and for the Republican, the first song is a rock anthem. The content is there, but the emphasis is shifted.

What This Means (and What It Doesn't)

The paper concludes that while the AI is personalizing the order of comments, it isn't drastically hiding entire comments from different users yet. The "filter bubble" effect on the top comments is weaker than expected, at least for this specific setup.

However, the authors are careful to say this is just a "first look."

The "Newbie" Problem: The fake accounts were new. The AI might not know them well yet, so it's showing a "default" list. As the accounts get older and the AI learns more about them, the differences might grow.
The "Echo Chamber" Reality: The researchers noticed that on news posts, the top comments usually agreed with the news source anyway (e.g., right-wing comments on a right-wing news site). So, even without the AI, the comments were already polarized. The AI just kept showing the same "agreed-upon" comments to everyone.

The Bottom Line

Think of this study as checking the thermostat in a new house. The researchers wanted to know if the thermostat was set to "Freeze" for some people and "Heat" for others. They found that, for now, the temperature is mostly the same for everyone (only a 12% difference), but the fan speed (the order of comments) is definitely different.

The paper doesn't claim this is the final answer or that the system is perfect. It simply says: "We checked, and the personalization isn't as extreme as we feared yet, but we need to keep watching because the system is still learning."

1. Problem Statement

In March 2025, Meta introduced an AI-driven system to rank and personalize the order of comments shown to Instagram users (default setting: "For You"). While extensive research exists on how feed personalization contributes to political polarization, there is a significant gap in understanding how comment ranking specifically influences user perception and discourse.

The authors investigate whether this new system creates systematic differences in the visible comments shown to different users based on their attributes (gender, political leaning, location). The study specifically asks:

To what extent do visible comments vary across different users viewing the same post?
Is this variation higher for news-related content (a known polarizing domain) compared to non-news content?
Can user attributes (gender, political leaning, location) systematically explain the observed variation, or are other factors more dominant?

2. Methodology

The study employs a small-scale exploratory audit using "sock-puppet" accounts to simulate diverse user profiles.

A. Data Collection Pipeline

Sock-Puppet Accounts: Four new accounts were created with distinct personas:
- Female Democrat
- Female Republican
- Male Democrat
- Male Republican
- Setup: Uniform birth dates, gibberish usernames, and empty bios. Political leaning was inferred by manually following specific politicians from the respective parties. No other interactions (liking/sharing) were performed to isolate the effect of following behavior.
Target Accounts:
- 10 News Accounts: Selected across the political spectrum (Left to Right) using Media Bias/Fact Check ratings.
- 10 Non-News Accounts: Selected from niches like sports, entertainment, food, and pets.
- Control: Follower counts were matched roughly between news and non-news groups to reduce variability.
Posts: The 10 most recent posts from each of the 20 accounts were selected. Posts were at least 24 hours old to ensure comment counts were "saturated" (stabilized).
Collection Process:
- A Selenium crawler collected the top visible comments (without scrolling) for each post.
- Locations: Data was collected from two distinct VPN endpoints (New York and Texas) to test location effects.
- Total Crawls: 8 distinct crawls (4 accounts × 2 locations).
- Scope: Top 10 text-based comments per post were analyzed.
- Dataset Size: 200 posts total, resulting in ~980 unique comments from news accounts and ~1,037 from non-news accounts after deduplication.

B. Analytical Approach

Descriptive Analysis: Calculated the variation proportion ( $V(p)/T(p)$ ) between pairs of crawls for each post. This measures the symmetric difference (comments appearing in one crawl but not the other) normalized by the total comments collected.
Post-Level Regression: A Bayesian Generalized Linear Mixed Model (Beta-Binomial) was fitted to model variation.
- Outcome: Variation proportion per post.
- Fixed Effects: Location, Gender, Political Leaning (derived from the pair being compared), comment count, follower count, and account type (News vs. Non-News).
- Random Effect: Post ID (to account for repeated observations).
Comment-Level Regression: A Bayesian Bernoulli (Logistic) Model analyzed individual comments to see if their sentiment (Supportive, Against, Neutral) predicted whether they were missing from one of the two crawls.

3. Key Results

A. Overall Variation is Low

Contrary to the hypothesis that personalization would lead to high divergence, the study found that only ~12% of visible comments varied between different user crawls on average.

B. News vs. Non-News Variation

Counter-intuitive Finding: Comments on news posts exhibited less variation (avg. 0.06 predicted variation) than those on non-news posts (avg. 0.17 predicted variation).
This suggests the algorithm is more consistent in what it shows for news content, possibly due to stronger engagement signals or different ranking heuristics.

C. Drivers of Variation

Account Metrics (Strongest Predictors):
- Comment Count: Positively associated with variation. Posts with more comments showed higher variability (likely due to the system "exploiting" rich engagement signals to personalize).
- Follower Count: Negatively associated with variation. Accounts with fewer followers showed higher variability (possibly due to the system needing to "explore" more to find relevant content).
User Attributes (Weak Predictors):
- Gender, Political Leaning, and Location had modest effects (only a few percentage points on the probability scale).
- While some 95% Highest Density Intervals (HDI) excluded 1 (indicating statistical significance), the practical magnitude was small. User attributes were not the primary drivers of what comments were seen.

D. Comment Sentiment

Supportive comments (aligned with the post caption) were more consistently present across crawls (lower probability of non-overlap).
Against/Neutral comments were more likely to vary, suggesting the system may filter out dissenting views more aggressively or inconsistently.

E. Ranking Order vs. Presence

While the presence of comments varied little (12%), a preliminary analysis of ranking order showed substantial variation. The average variance proportion for ranking was 0.5 for news and 0.7 for non-news, indicating that while users see similar sets of comments, the order in which they see them differs significantly.

4. Key Contributions

First Systematic Audit: This is the first study to investigate personalized comment ranking on Instagram following the March 2025 AI update.
Methodological Framework: Provides a reproducible pipeline for collecting and analyzing comment visibility using sock-puppet accounts and Bayesian modeling.
Counter-Intuitive Insights: Challenges the assumption that personalization leads to high divergence in news contexts; finds that account metrics (engagement/followers) are stronger predictors of variation than user demographics.
Data & Code Availability: The authors commit to releasing the collection code and data to support further research.

5. Significance and Limitations

Significance: The study suggests that while comment personalization exists, its immediate impact on which comments are visible is limited compared to the impact of ranking order. It highlights that platform mechanics (account size, engagement volume) may currently drive visibility more than user identity, though this could evolve as the algorithm learns more about users.
Limitations:
- Small Scale: Only 4 sock-puppet accounts and 200 posts were used.
- New Accounts: The sock-puppets were newly created, meaning the algorithm had limited historical data on them, potentially underestimating personalization effects.
- Domain Specificity: The study focused on US-based accounts; results may differ globally.
- Content Type: The lack of variation in news comments might be due to the specific nature of the selected news posts (e.g., comments aligning with the account's bias).

Conclusion

The paper concludes that while Instagram's new comment ranking system does introduce variation, the effect is currently modest regarding comment presence but substantial regarding comment order. The findings motivate the need for larger, longitudinal audits to understand how these systems might shape online discourse and polarization as the algorithms mature and user data accumulates.