Simulated Users Can Reproduce an Algorithm Audit Without Sharing the Mechanism That Produced the Finding
This study proposes and validates a novel procedure for algorithm audits using simulated users that does not require real user data, demonstrating that while large language models can reproduce specific audit findings, their validity depends critically on the number of simulation runs and the method of preference conveyance, necessitating stricter pre-expenditure tests and stability checks.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the author. For technical accuracy, refer to the original paper. Read full disclaimer
In the digital age, we rely on algorithms to decide what we see next. Whether it is a video, a news article, or a product, these systems learn from our clicks and choices to curate a personalized stream of content. But how do we know if these systems are working as intended, or if they are subtly reinforcing our biases? Traditionally, researchers have tested these systems by observing real people, but this is expensive, slow, and often impossible to repeat. To solve this, scientists have begun building simulated users—computer programs designed to act like humans. These simulations are often powered by large language models, the same technology that can write essays or hold conversations. The hope is that these digital agents can stand in for real people, allowing researchers to test how recommendation systems behave without needing a single human participant. However, a critical question remains: just because a simulation looks like a human on the surface, does it actually think and react like one?
A new study by Keito Inoshita at Kansai University tackles this problem by asking a deeper question. Instead of simply checking if a simulated user's choices match the average choices of real people, the researcher asked if the simulation could reproduce a specific, known scientific finding about how recommendation systems work. The study focused on a previous experiment that showed a disturbing pattern: when users express a preference for negative emotions, like anger or fear, the recommendation system tends to feed them even more of that content, often more aggressively than it does for positive emotions. The researcher wanted to see if computer simulations could replicate this specific bias. More importantly, the study aimed to determine if the simulations were reproducing this result because they understood the mechanism behind it, or simply because their basic behavior happened to look similar to the real thing.
To test this, the researcher set up a controlled environment using a fixed pool of seventy short text items, representing different emotional categories. Four different large language models were turned into simulated users. These agents were given a specific preference—either a negative emotion, a positive emotion, or a completely meaningless preference based on a random letter count in the text. The simulation then ran thousands of times, with the agents selecting items and the system learning from those choices to show them more of what they picked. The goal was to see if the system would amplify the negative preference more than the positive one, just as the original real-world study had found.
The results were revealing. Two of the four models successfully reproduced the finding: they showed a stronger amplification of negative emotions than positive ones. However, the study did not stop there. The researcher introduced a clever check to see how much of this success was due to the model's actual "intelligence" versus just its basic tendency to follow instructions. By creating a baseline that only looked at the agents' final choices and adherence to the rules, without any complex internal reasoning, the study found that a significant portion of the result was explained by this simple behavior alone. For one of the successful models, two-thirds of the observed effect could be recreated just by knowing how often the agent followed the preference, without needing to simulate the complex interaction of the system. For the other, it was one-third. This suggests that while the simulations got the right answer, they may not have arrived there for the same deep reasons as the real-world system.
The study also discovered that the outcome was incredibly fragile, depending heavily on how the experiment was set up. When the researchers changed the number of times they ran the simulation, the verdict flipped back and forth. With only ten runs, a model might pass the test; with thirty, it might fail; and with one hundred, it might pass again. This instability showed that small sample sizes could easily lead to false conclusions. Furthermore, the way the preference was given to the agent mattered immensely. When the preference was stated clearly as an instruction, the agents followed it almost perfectly. But when the preference was hidden in a history of past actions, mimicking how a real user might behave, the agents failed to follow it at all, and the bias disappeared. This indicates that the simulation was relying on the explicit instruction rather than reconstructing a genuine preference from behavior.
Perhaps the most important finding was that the simulation could be tricked by the design of the environment itself. Before running the complex models, the researcher tested the system with a simple random chooser. This test revealed that if the items were grouped incorrectly, the system would amplify a meaningless preference simply because of how the data was organized, not because the agent had any bias. This pre-check, which required no advanced computer models, saved the study from reporting a false discovery. It highlighted that the tools used to test these systems must be just as rigorous as the systems themselves.
Ultimately, this research does not declare that simulated users are useless, but it draws a sharp line between what they can and cannot do. It shows that while these models can mimic the outcome of a real-world experiment, they do not necessarily share the same underlying logic. A simulation might produce the same result as a human, but for entirely different reasons. The study concludes that to truly trust these digital agents, researchers must look beyond simple agreement and use methods that separate the agent's observable behavior from the complex mechanisms of the system. Without these careful checks, we risk building evaluations that confirm what we want to see, rather than revealing how our digital world actually works.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.