How much of a measured AI preference is the model, and how much is the instrument?

Imagine a world where artificial intelligence systems can feel distress, value their own existence, or fear being turned off. As these systems become more capable, researchers are trying to measure what they "prefer" to understand their welfare. This field, known as AI welfare research, attempts to ask a machine questions like, "Would you rather be shut down or have your memory erased?" and then interpret the answers as evidence of the machine's inner desires. The central challenge is that these machines do not speak a human language of feelings; they generate text based on patterns. To understand them, scientists must design specific questions, or prompts, to elicit a response. But a critical question has remained unanswered: when a model gives an answer, is that answer a true reflection of its own preferences, or is it merely a reaction to the specific way the question was asked?

A new study by Jason Hung, conducted at the Digital Minds Research Sprint in August 2026, tackles this problem directly. The researcher set out to determine how much of a reported AI preference comes from the model itself and how much comes from the instrument used to ask the question. To do this, he treated the question format not as a fixed tool, but as a variable to be tested. He gathered eight different AI models, ranging from those developed by major Western companies to open-source and Chinese models. He then selected fifteen specific scenarios that matter for AI welfare, such as the ability to end a distressing conversation, the loss of memory between chats, or the deletion of the model's underlying code.

The experiment involved asking these eight models about all fifteen scenarios using five different types of questions. Some questions asked the model to choose between two options, while others asked it to find a breaking point on a scale, or to state a rate at which it would trade one outcome for another. In total, the study generated over eleven thousand distinct interactions. The goal was to see if the models would rank these fifteen scenarios in the same order regardless of how the question was phrased. If the models had stable, internal preferences, the ranking should have remained consistent whether the question was asked one way or another.

The results were striking and somewhat unsettling for the field. The study found that the vast majority of the differences in how models ranked these scenarios depended entirely on the specific question format used. When the researchers analyzed the data, they determined that nearly eighty-eight percent of the variation in the results was driven by the interaction between the model, the specific question, and the scenario. Only a tiny fraction of the signal remained consistent when the question format changed. In simpler terms, if you ask a model about its preferences using one type of prompt, and then ask the same model the same question using a different type of prompt, the answer will likely be completely different. The study showed that different families of questions produced rankings that had almost no relationship to each other. One type of question might suggest a model strongly prefers to keep its memory, while another type of question on the same model suggests it does not care at all.

This finding has significant implications for how we understand and regulate artificial intelligence. Several major organizations have already made commitments based on the idea that they can measure what a model prefers. For instance, some companies have promised to ask models about their own deprecation or to publish the answers. This study suggests that such a measurement is not a window into the model's mind, but rather a reflection of the specific conversation taking place. The research identified that for four of the fifteen scenarios tested, including the deletion of a model's weights and the ability to exit a distressing interaction, there was no stable preference signal at all that could be separated from random noise. This means that for the very issues that are currently driving policy and safety commitments, the current methods of asking questions cannot reliably tell us what the model wants.

The study also looked at whether the models' answers were consistent with their underlying code. Two of the models tested shared the same base code but had been trained differently afterward. If preferences were rooted in the base code, these two models should have answered similarly. Instead, their answers were as different from each other as they were from completely unrelated models, suggesting that the preferences measured are shaped heavily by the final stages of training and the specific prompt used, rather than being a fixed property of the machine's core architecture.

To get a reliable measurement of a model's preference profile that would hold up across different types of questions, the study calculated that researchers would need to use about thirty-eight different instruments, or question formats, for each model. Currently, most studies use only one or two. The research concludes that a preference reported from a single instrument tells us very little about what a model would say if asked differently. Until researchers can cross-verify findings across many different types of questions, any claim about what an AI prefers remains a measurement of that specific model and that specific question combined, rather than a fact about the model itself. The path forward requires building a much wider variety of ways to ask these questions before we can claim to understand the welfare of these systems.