Do Judges Behave Like Algorithms?

Imagine a world where every decision is made by a giant, invisible calculator. In the realm of science known as artificial intelligence and law, researchers are constantly asking: should we let computers make the big calls, like who goes to jail and who stays free? To understand this, you need to know about two ways of making choices. The first is like a strict recipe: "If you have three strikes, you're out." It's clear, fair, and everyone gets the same result. The second is like a chef tasting a soup: "It needs a little more salt, but maybe not if the tomatoes are too sour." This is flexible and personal, but it depends on the chef's mood and memory. For decades, people have argued whether judges should follow the strict recipe or the flexible tasting. Now, with computers getting smarter, the debate has shifted. Instead of asking if we should replace judges with robots, scientists are asking a wilder question: Are the judges already acting like robots? If a judge is just following a hidden recipe, we could find it and fix it. But if they are truly tasting the soup based on feelings we can't see, that's a much harder problem to solve.

This paper dives into that mystery by peeking into the minds of 21 magistrates (judges) in Harris County, Texas, who decide whether people accused of minor crimes should be released before their trial or held in jail. The researchers treated these judges like black boxes and tried to reverse-engineer their decision-making process. They asked: Can we write a simple, understandable formula that predicts what a specific judge will do? And if we can, do all the judges use the same formula, or does every judge have their own secret code?

The team started by cleaning up a massive dataset of over 22,000 cases. First, they had to deal with the "glitches." They found that about 2% to 33% of a judge's decisions couldn't be explained by any logical rule at all. When they looked closer, they realized these weren't mysterious acts of magic or bias; they were just errors. Sometimes the computer forms were filled out wrong, sometimes the judge knew about a crime in a different county that wasn't in the database, or sometimes they heard a detail in the courtroom that never made it into the written record. Once they removed these "noisy" cases, the real story emerged.

The researchers discovered that, surprisingly, most judges do behave like algorithms. For 16 out of the 21 judges, the team could build a tiny, simple decision tree (a flowchart with just a few steps) that predicted the judge's decision with 85% to 100% accuracy. These formulas relied on a few clear factors, like the defendant's age, how many warrants they had outstanding, and the specific type of crime they were charged with. It turns out that for many judges, the "soup tasting" is actually just following a very specific, repeatable recipe.

However, here is the twist: while the judges are acting like algorithms, they are all using different algorithms. The study found that judges are highly consistent with themselves—meaning Judge A will usually make the same choice for similar cases 76% of the time. But they are terrible at being consistent with each other. When the researchers tried to use Judge A's "recipe" to predict what Judge B would do, it failed miserably, performing no better than random guessing. In fact, for some pairs of judges, they disagreed on nearly every case involving young people (under 24). One judge might see a young person with a warrant and say "release them," while another sees the exact same profile and says "hold them."

The paper also looked at what variables mattered most. While age and warrant counts were important for almost everyone, the weight they gave to other factors varied wildly. Some judges cared deeply about whether a defendant was homeless or their gender, while others ignored those factors entirely. This suggests that the outcome of a bail hearing depends heavily on which judge you get, not just the facts of your case.

Ultimately, the authors suggest that we shouldn't try to replace judges with computers. Instead, we should use these findings to understand the "hidden recipes" judges are already using. If we can see that Judge X always treats young people differently than Judge Y, or that a judge's decision changes based on a missing piece of data, we can fix those inconsistencies. By making these invisible rules visible, we can help the justice system become more fair and predictable, ensuring that the "recipe" is the same for everyone, regardless of who is cooking.