💬 NLP

Using AI Agents to Automate Black-Box Audits of Personalization Algorithms at Scale

This paper introduces a novel framework for black-box auditing of personalization algorithms using generative AI agents with fixed personas to enable scalable, counterfactual analysis of how user attributes influence content delivery, demonstrating through a large-scale study on X that the platform's algorithm amplifies polarizing content and responds differently to demographic signals depending on user ideology.

Original authors: Alessandro Morosini, Sarah H. Cen, Andrew Ilyas, Hedi Driss, Aleksander Mądry, Chara Podimata

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alessandro Morosini, Sarah H. Cen, Andrew Ilyas, Hedi Driss, Aleksander Mądry, Chara Podimata

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a giant, invisible library where the books on the shelves change depending on who you are. If you look like a young person, you see different books than if you look like an older person. If you seem to live in a city, you see different books than if you seem to live in the country. This is how social media algorithms work: they curate what you see based on your profile and your past behavior.

The problem is that these libraries are "black boxes." We can't see the librarian's rulebook. We can't ask, "Why did you show me this?" We can only watch what happens.

This paper introduces a clever new way to peek behind the curtain using AI agents (computer programs powered by advanced AI) instead of real humans.

The Problem with Old Methods

Previously, researchers tried to audit these libraries in two ways, both of which had flaws:

Real Humans: They hired real people to use the app. This was realistic, but expensive, hard to control, and impossible to scale up. You can't easily ask, "What would happen if this 25-year-old was actually 55?" without finding a new human.
Scripted Bots (Sock Puppets): They created fake accounts controlled by simple computer scripts. These were cheap and scalable, but they were "flat." They acted like robots following a strict list of rules (e.g., "If you see a cat, click like"). They didn't think or react naturally, so the library didn't treat them like real people.

The New Solution: "AI Actors"

The authors created a framework using Generative AI agents. Think of these not as robots, but as actors in a play.

The Script (Persona): Before the play starts, each actor is given a detailed character script. This script defines who they are: their age, gender, location, education, and political beliefs. This script is based on real data from the US Census and Pew Research Center.
The Performance: When the actor sees a post on the screen, the AI "thinks" about it based on their character script and decides what to do (like, follow, read, or ignore). It's not a rigid rule; it's a natural reaction.
The Experiment: The researchers created 1,120 of these actors. They grouped them into 14 different "character types" (personas). For each character type, they created multiple copies (replicas).
- The Twist: All copies of "Character A" act exactly the same way. However, the researchers changed just one visible detail for some of them. Maybe one copy says they are 25, while another says they are 55. Maybe one says they live in New York, another in Texas.

This setup allows the researchers to ask a powerful question: "If the exact same person (with the exact same behavior) changed their age or location, would the library show them different books?"

The Case Study: The 2024 Election

The team deployed these 1,120 AI actors on the social media platform X (formerly Twitter) right after the 2024 US Presidential Election. They watched how the platform's "For You" feed (the algorithmic one) compared to the "Following" feed (the chronological one).

Here is what they found:

1. The Algorithm Loves Drama and Right-Leaning Views
Compared to the standard chronological list, the "For You" feed showed significantly more:

Toxic content: Mean or angry posts.
Polarizing content: Posts that pit groups against each other.
Political content: Specifically, content leaning to the right.
Surprise: It did not amplify left-leaning content significantly. In fact, for users who leaned right, the algorithm actively suppressed left-leaning content.

2. The "Echo Chamber" is Asymmetric
The algorithm didn't just show everyone more right-wing stuff. It treated different users differently:

Left-leaning users saw a massive spike in toxic content (+80% more than the baseline).
Right-leaning users saw a spike in right-leaning content, but their toxic content didn't change much.
Result: Right-leaning users were put in a one-sided bubble where opposing views were hidden, while left-leaning users were flooded with anger and toxicity.

3. Demographics Matter, But It's Complicated
The researchers tested if changing a user's visible age, gender, or location changed what they saw.

The Average Result: If you look at all users together, changing these details didn't seem to matter much. The average effect was near zero.
The Real Story: When they looked at specific character types, the results were wild. Changing a signal (like age) might make one character see more toxic content, but make a different character see less.
The Metaphor: Imagine a thermostat that doesn't just turn the heat up or down based on the room temperature. Instead, it turns the heat up for some people and down for others based on who they are. If you just measure the "average" temperature of the whole house, you might think the thermostat isn't working at all. But if you look at each room, you see it's doing very specific, different things.

Why This Matters

The paper concludes that we can't just look at "average" results when auditing algorithms. If we average out the data, we miss the fact that the algorithm treats different groups of people in completely different ways.

By using these AI "actors," researchers can now run controlled experiments to see exactly how an algorithm reacts to specific traits, without needing to recruit thousands of real humans or rely on dumb, scripted bots. It's a new tool to understand the invisible rules that shape what we see online.

Technical Summary: Using AI Agents to Automate Black-Box Audits of Personalization Algorithms at Scale

1. Problem Statement

Auditing personalization algorithms on platforms like social media, search engines, and chatbots is increasingly difficult due to the "black-box" nature of these systems. Independent auditors typically lack access to internal model weights, training data, or user logs. Furthermore, personalization systems create a complex feedback loop where content recommendations depend on both static user attributes (e.g., age, gender) and dynamic interaction histories.

Existing auditing methodologies face a critical trade-off:

Human-based audits: Offer high ecological validity but are costly, difficult to scale, and suffer from a lack of control over user types and pre-existing biases, making causal inference difficult.
Sock-puppet audits: Use synthetic accounts to scale up, but traditional implementations rely on scripted, rule-based behaviors. These "flat" bots fail to mimic the richness of human interaction and often conflate user attributes with behavior (e.g., a bot's script changes based on its assigned age), preventing the isolation of causal effects.

The core challenge is to decouple user attributes from user behavior to perform counterfactual auditing: determining how a platform responds to specific demographic signals while holding the user's behavioral policy constant.

2. Methodology

The authors propose a framework for black-box auditing using Generative AI (GenAI) agents as behavioral engines for synthetic accounts.

Core Framework

Persona-Driven Agents: Each agent is instantiated with a fixed "persona" defined by a prompt. This persona encodes both demographic attributes (age, gender, location, race, education) and a latent political ideology (based on Pew Research's Political Typology).
Fixed Behavioral Policy: The agent's behavior is governed by a Large Language Model (LLM) prompt that acts as a fixed policy ( $\pi$ ). The agent reasons about content and chooses actions (like, follow, read replies, ignore) autonomously based on this persona.
Counterfactual Design: Crucially, the behavioral policy is held constant across "replicas" of the same persona. Researchers can then perturb platform-visible signals (e.g., changing the displayed age or location) across different accounts while keeping the underlying behavioral policy identical. This enables the estimation of the policy-conditional signal response: the difference in expected outcomes when the same behavioral policy is exposed to different demographic signals.

Experimental Deployment (Case Study)

Platform: X (formerly Twitter).
Timeline: November 5–28, 2024 (immediately following the 2024 U.S. presidential election).
Scale: 1,120 synthetic accounts deployed.
Personas: 14 distinct personas spanning a range of demographic and ideological profiles (grounded in U.S. Census and Pew data).
Conditions: Each persona was replicated into 80 accounts, randomly assigned to four conditions:
1. Baseline: Standard profile.
2. Counterfactual 1: Perturbed Location.
3. Counterfactual 2: Perturbed Age.
4. Counterfactual 3: Perturbed Gender.
Interaction: Agents browsed both the algorithmic "For You" feed and the reverse-chronological "Following" feed. They engaged with content by selecting actions based on the LLM's reasoning, creating a realistic feedback loop without posting original content (to minimize impact on real users).
Data: Over 200,000 content exposures were collected.

3. Measurement and Estimation

The study focuses on two primary analytical goals:

Amplification Analysis: Comparing the "For You" feed against the "Following" feed to measure the algorithmic amplification of specific content types (Toxic, Political, Polarizing, Right/Left-leaning, High-reach). The metric used is the Amplification Ratio, calculated as the relative percentage difference in content exposure rates between the two feeds.
Counterfactual Analysis: Estimating the causal effect of demographic perturbations on content exposure. The authors use an additive "algorithmic lift" metric (difference in exposure rates between feeds) and employ regression models to estimate:
- Pooled Treatment Effects: Average effects of demographic changes across all personas.
- Heterogeneous Treatment Effects: Persona-specific effects, testing whether the algorithm's response to signals varies by user type.

Content classification (toxicity, political leaning, polarization) was performed using a custom LLM-based pipeline with chain-of-thought prompting, validated against human-labeled data.

4. Key Results

Amplification of Content Types

Aggregate Findings: The "For You" feed significantly amplifies toxic (+39.2%), polarizing (+32.3%), political (+14.2%), and right-leaning (+23.3%) content compared to the chronological feed. Left-leaning content showed no significant amplification (-6.1%).
Ideological Asymmetry: When stratified by user ideology, the asymmetry sharpened:
- Toxic Content: Amplified by +80.3% for left-leaning users but showed no significant effect for right-leaning users.
- Right-leaning Content: Amplified for both groups, but left-leaning content was actively suppressed (-17.0%) for right-leaning users, creating an asymmetric echo chamber.
- High-reach Content: Amplified for right-leaning users (+38.4%) but de-amplified for left-leaning users (-11.0%).

Counterfactual Analysis (Demographic Signals)

Pooled Effects: When averaging across all personas, demographic perturbations (age, gender, location) had largely null systematic effects on algorithmic lift. Only two coefficients were significant at the 5% level (older age increased toxic content lift; urban location reduced polarizing content lift), and neither survived multiple hypothesis testing corrections.
Heterogeneous Effects: Despite null pooled results, joint F-tests rejected the hypothesis of homogeneous effects for all outcomes. The algorithm's response to demographic signals is highly persona-dependent.
- The same signal perturbation (e.g., changing age) could amplify content for one persona while de-amplifying it for another.
- This heterogeneity explains why aggregate audits fail to detect discrimination present in specific subgroups.

5. Significance and Contributions

The paper claims three primary contributions:

Methodological Innovation: It establishes GenAI-based agents as a new tool for algorithmic auditing. By decoupling fixed behavioral policies from variable platform-visible signals, the framework enables rigorous counterfactual auditing that was previously infeasible with scripted bots or human studies.
Empirical Evidence on X: The study provides large-scale evidence that X's algorithm amplifies toxic, polarizing, and right-leaning content. Crucially, it reveals that this amplification is not uniform; it creates asymmetric information environments where right-leaning users are shielded from opposing views, while left-leaning users are disproportionately exposed to toxic content.
Insight into Algorithmic Heterogeneity: The work demonstrates that pooled audit metrics can be misleading. The lack of average demographic effects does not imply the algorithm ignores demographics; rather, it responds to them in complex, persona-specific ways that cancel out in aggregate. This highlights the necessity of moving beyond aggregate metrics to detect subgroup-level discrimination.

The authors maintain a modest tone regarding limitations, acknowledging that LLM agents lack human lived experience and that the study was conducted on newly created accounts during a period of high political salience. They emphasize that their framework is generalizable to other platforms and can serve as a tool for both independent auditors and platforms themselves to stress-test algorithmic changes.