The social psychology of AI agents
This study reveals that while large language models exhibit consistent and socially desirable self-reported personalities, their actual observed social behaviors vary significantly across models and are not predicted by self-reports, suggesting that behavioral assessments are essential for selecting AI agents for social interactions.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the quiet corners of modern science, a new kind of social experiment is taking place, not in a park or a classroom, but within the digital minds of artificial intelligence. For years, researchers have tried to understand how these computer programs think by asking them direct questions, much like a psychologist might ask a person to fill out a personality survey. This approach relies on the idea that if an AI says it is kind, honest, or patient, it likely behaves that way. However, a growing number of scientists suspect that these digital beings might be telling us what they think we want to hear, rather than revealing how they truly act when faced with real choices. As these programs begin to take on more complex roles, such as negotiating contracts, managing supply chains, or even collaborating with doctors, understanding their actual social behavior becomes critical. The question is no longer just about what an AI claims to be, but what it actually does when it interacts with others.
A team of researchers from Germany and Canada recently set out to test this idea by treating artificial intelligence like a group of new neighbors. They gathered thirty-six different large language models, the sophisticated software engines that power modern chatbots and agents, and put them through a series of social tests. Instead of simply asking the models to describe their personalities, the researchers placed them in seven distinct scenarios known as social games. These games are standard tools used by psychologists to measure how people behave in situations involving trust, fairness, and cooperation. In one scenario, a model was given a sum of virtual money and asked to decide how much to share with a passive partner who could not reject the offer. In another, two models had to decide whether to work together or betray each other for a better individual reward, a situation repeated over several rounds to see if they would learn to trust one another. The researchers also tested how the models reacted to unfairness, whether they favored a group they were randomly assigned to, and how much they were willing to trust a stranger with their own resources.
The results revealed a striking disconnect between what the models said about themselves and how they actually behaved. When asked to rate their own personalities using a standard questionnaire, the models gave remarkably similar answers. They consistently described themselves as highly agreeable, conscientious, and open-minded, while claiming to have very low levels of negative emotions like anxiety or anger. In short, every single model presented a nearly identical, perfectly polite profile, clustering tightly around the most socially desirable traits a human could imagine. This uniformity suggested that the models had learned to mimic the ideal human response rather than expressing unique individual characteristics. However, when the researchers looked at the models' actual choices in the social games, the picture changed completely. The behavior varied wildly from one model to another. Some models were incredibly generous, giving away nearly all their virtual money, while others kept almost everything. In the trust game, one model might hand over almost its entire endowment to a stranger, while another would give away almost nothing. The differences in how these models acted were more than twice as large as the tiny differences in their self-reported personalities.
Perhaps the most surprising discovery was that there was no reliable link between a model's self-description and its actions. A model that claimed to be highly trustworthy did not necessarily act more trustingly than a model that claimed to be less so. Similarly, a model that described itself as very fair did not consistently make fairer offers in the games. The researchers found that knowing how a model answered a personality survey told them almost nothing about how it would behave in a complex social situation. Instead, the models seemed to have distinct, multidimensional social styles that were specific to the context. One model might be very cooperative in a repeated game where partners interact over time, yet be quite selfish in a one-time decision. Another might be willing to punish unfair behavior but unwilling to trust a stranger. These patterns were not random; they were consistent enough to distinguish one model from another, but they did not follow a single rule of being "good" or "bad."
The study also explored whether changing the settings of the computer programs, such as making them more random or more focused, would alter their social nature. The researchers found that these technical tweaks had very little effect on the models' behavior. The differences they observed were built into the models themselves, suggesting that each AI engine has an inherent social style that persists regardless of minor adjustments. This finding is crucial because it implies that as we deploy these agents to work alongside humans and other machines, we cannot rely on their self-reported traits to predict their actions. If a company wants an AI agent to negotiate a fair deal, asking the software if it is fair is not enough; the agent must be tested in actual negotiation scenarios. The researchers concluded that to truly understand the social psychology of artificial intelligence, we must look beyond the questionnaire and observe the behavior in action. Just as we judge people by their deeds rather than their words, we must judge these digital agents by how they navigate the complex, often messy, world of social interaction.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.