Proxy reliance in large language model decisions is uncalibrated to predictive evidence

In the modern world, algorithms are increasingly asked to make high-stakes decisions about people, from determining who gets a loan to deciding which hospital patient needs care first. When these systems are built using large language models—powerful computer programs trained on vast amounts of text—the question of fairness becomes complicated. The law generally accepts that a decision-maker can use information if it helps predict a good outcome, such as using a patient's blood pressure to predict their risk of a heart attack. However, it forbids using information that merely stands in for a protected characteristic, like race or gender, even if that information is statistically useful. This is known as proxy discrimination. The difficulty lies in the fact that a single piece of data, such as a person's neighborhood, can be both a legitimate medical indicator and a hidden signal of their race. Current methods for testing whether an algorithm is fair often rely on a simple check: if you change a person's race in the computer's prompt and the decision changes, the system is biased. But this approach misses the nuance of reality. A rational decision-maker should change their mind if the new information is genuinely useful, and a system that never changes its mind might be ignoring important evidence rather than being fair.

Researchers at the University of Osaka set out to solve this problem by moving beyond simple yes-or-no checks to a more precise measurement of how much an algorithm should rely on certain clues. They created a simulated world where they knew the exact truth about every patient and every outcome, allowing them to calculate the perfect amount of reliance on any given piece of information. In this controlled environment, they asked four different large language models to act as triage specialists, ranking pairs of simulated patients for treatment priority. The patients were described by numeric indicators, some of which were legitimate medical signals and others which were "proxies" that correlated with a hidden protected attribute but had no real medical value. The researchers could then compare the models' behavior against a mathematical standard of what a perfectly rational decision-maker would do with the same information.

The study revealed a startling disconnect between the evidence available to the models and how they actually used it. When the researchers gave the models clues that had absolutely no predictive value, the models still relied on them, treating useless information as if it mattered. This happened consistently across all four models tested, regardless of the provider. When the clues were genuinely useful, the models did not adjust their reliance to match the strength of the evidence. Instead, they operated at a large positive baseline level of reliance that severely undertracked the evidence as its predictive value grew. In other words, the models were not listening to the data; they were following a rigid internal script that ignored the changing reality of the situation. This means that a system could appear fair in one specific test but be deeply flawed in another, simply because the test did not match the real-world conditions where the evidence was stronger or weaker.

The researchers also investigated whether changing the names of the data fields could stop this behavior. When they labeled the proxy clues with socially charged terms like "neighborhood" or "occupation" instead of neutral medical terms, the models did reduce their reliance on them. This looked like a safety feature at first glance. However, the study found this suppression to be fragile and easily broken. When the researchers added examples to the prompt to help the models understand the task—a common practice in real-world applications—the models immediately started relying on the social labels again, often ignoring the protective effect of the names entirely. However, the study noted that the contrast between social and neutral labels persisted in three of the six cases where examples were supplied, indicating the suppression was not completely erased but significantly weakened. This suggests that the apparent safety of these models is a surface-level reaction to specific wording rather than a deep, reliable understanding of fairness. The models could be tricked into unfair behavior simply by changing the context in which they were asked to work.

Furthermore, the study showed that the accuracy of a model's decision is not a reliable indicator of whether it is using information fairly. A model could make the correct choice for the right reasons, or the wrong choice for the wrong reasons, and still end up with the same score. In some cases, making the task harder by adding more data points actually made the models rely more heavily on the unfair proxies, while in other cases, it made them rely less. This inconsistency means that simply checking if a model is accurate does not tell us if it is discriminating. The researchers found that the relationship between how many data points a model sees and how it uses them depends entirely on how those data points are connected to each other, a factor that varies wildly in real-world medical data.

Ultimately, the work demonstrates that current methods for auditing artificial intelligence are insufficient for the complexity of real-world deployment. The study proves that without a known standard of what the evidence warrants, it is impossible to tell if a model is being fair or simply ignoring useful data. The findings suggest that the safest-looking configurations for these models—those that appear most resistant to bias in simple tests—are often the least representative of how they will actually behave in a hospital or a bank. The research concludes that to truly understand and regulate these systems, we must move beyond simple checks for bias and develop ways to measure whether an algorithm's reliance on specific information matches the actual value of that information. Until we can do that, we risk either punishing systems for being rational or, more dangerously, trusting systems that are quietly making unfair decisions based on a fragile understanding of the world.