Developing and assessing a scale for measuring human–AI alignment in hiring decisions

In the modern world, the gatekeepers of opportunity are changing. For decades, the decision to hire someone for a job rested entirely on human judgment, a process influenced by experience, intuition, and sometimes unconscious bias. Today, artificial intelligence is increasingly stepping into that role, screening thousands of resumes and making initial selections before a human ever sees a name. This shift raises a fundamental question: when a machine makes a hiring decision, does it think like a person? The answer is not as simple as asking if the computer is right or wrong. Instead, researchers are asking how closely the machine's choices mirror the messy, varied, and often inconsistent ways humans actually decide. To understand this, scientists have begun to treat hiring not just as an economic transaction, but as a test of alignment, measuring the distance between human reasoning and algorithmic logic.

A team of researchers set out to measure this distance with a new approach they call a Labour Market Turing Test. The classic Turing Test, proposed decades ago, was designed to see if a machine could convince a human it was also human. This new version flips the script. Instead of asking if a machine can fool a person, it asks how much a machine's decisions look like a person's. The researchers did not simply ask an AI to pick the best candidate. They created a controlled experiment in Ecuador involving 277 professional human recruiters. These real people were hired to review pairs of job applicants for ten different types of jobs, ranging from warehouse assistants to software developers. To ensure the test was fair and focused on decision-making style rather than specific qualifications, the applicants in each pair were matched on their skills, education, and experience. The only differences between them were identity markers, such as gender, sexual orientation, or immigration status—factors that should not logically influence who is hired for the job.

The human recruiters made their choices, and then the researchers built a digital mirror. They created four different teams of artificial intelligence agents, each powered by a large language model, to perform the exact same task. The first team, called "Robots," received only basic instructions to pick the best candidate. The second team, "Avatars," was told to pretend to be specific human recruiters, mimicking their personality traits, education, and professional background. The third team, "Clones," went a step further; they were not only told to pretend to be a specific human but were also restricted to seeing only the same information that the human had seen, in the exact same order. The final team, "Randomistas," made their choices by flipping a digital coin, serving as a baseline for pure randomness. The goal was to see which of these AI teams made decisions that looked most like the human recruiters.

To measure the results, the researchers developed a scale to place every recruiter, human and machine, on a single line. At one end of the line sits a reference point called "Skynet," which represents the average choice made by the basic Robot team. This is not a measure of fairness, but a measure of pure algorithmic consistency. A recruiter who always agrees with Skynet is acting in a highly machine-like way. A recruiter who disagrees often is acting in a more human way. The study found that the average human recruiter was significantly different from the average machine. In fact, the human choices were about 1.73 standard deviations away from the machine's typical behavior. This means that, on average, humans are much less predictable and less consistent than the basic AI.

The researchers discovered that the more they tried to make the AI act like a human, the closer it got to the human average, but it never quite arrived. Even the "Clones," which were programmed to mimic specific humans and restricted to the same information they saw, still produced a pattern of choices that was statistically different from the real people. The most surprising finding came from the "Randomistas." The AI agents that simply flipped a coin to make their decisions actually ended up with a pattern of choices that was closer to the human average than any of the sophisticated AI teams. This suggests that human decision-making in hiring contains a level of randomness or inconsistency that is difficult for even the most human-like AI to replicate. The humans were not just making rational, rule-based choices; they were making choices that included a degree of unpredictability that the machines, even when told to be human, could not fully capture.

The study also revealed that the gap between humans and machines widened depending on the complexity of the job. When the recruiters were choosing between candidates for simple jobs, like a janitor or a warehouse worker, humans and machines agreed more often. However, as the jobs became more demanding, requiring specialized skills or higher education, the agreement dropped. The machines and the humans started to diverge more sharply. Furthermore, when the two candidates in a pair were very similar, with only tiny differences in their age or years of experience, the humans and machines disagreed more often. The machines seemed to latch onto these minute details and make a decision based on them, whereas humans were more likely to treat the candidates as essentially equal. This indicates that the AI is hyper-vigilant about small differences, while humans are more willing to accept ambiguity.

The researchers also looked at whether they could explain why some humans were more "machine-like" than others. They examined the recruiters' personalities, their cognitive test scores, their education, and their years of experience. Surprisingly, none of these factors provided a clear explanation. A recruiter's intelligence or their years in the field did not predict whether they would make choices that looked like a machine's. The only traits that showed a weak connection were personality-based: recruiters with higher self-esteem tended to be slightly more machine-like, while those with higher neuroticism were slightly less so. But even with these factors, the vast majority of the difference between human and machine choices remained unexplained. This suggests that the way humans make these decisions involves a complex, hidden dimension that cannot be easily measured by standard surveys or tests.

Ultimately, the study suggests that perfect imitation of human judgment requires capturing not just the average tendency to pick a certain type of candidate, but also the full diversity of human reasoning, including its inconsistencies. The AI systems, even when tuned to be human-like, tend to be more consistent and rule-based than the people they are trying to mimic. This creates a paradox: the more an AI system tries to be perfectly consistent and rational, the less it looks like a human. The researchers argue that this difference is crucial for understanding how to use AI in hiring. If we want AI to be trusted, we need to understand that it will not naturally behave like a human. It will not naturally embrace the same level of ambiguity or randomness that humans do. The study concludes that before we hand over the keys to hiring to algorithms, we must recognize that these systems are not just faster humans; they are fundamentally different kinds of decision-makers, operating on a logic that is stable and rule-bound, but often alien to the messy reality of human choice.