Pairwise, Style-Controlled, Nonparametric Tests for Judge Bias under the ISU Judging System

In sports where human beings decide the winner, the line between a judge's personal style and actual unfairness is often invisible. Some judges are naturally generous, handing out high scores to everyone, while others are strict, reserving top marks for only the most perfect performances. Some judges avoid giving extreme scores, sticking safely to the middle of the range, while others use the full scale. These differences in style are not necessarily bias; they are simply the way different people see the world. The real challenge for sports officials is to figure out when a judge is not just being strict or lenient, but is specifically favoring one competitor over another in a way that changes the outcome. This is the central question behind the study of judging integrity: how do you separate a judge's general personality from a specific, unfair preference for a particular athlete?

A new study by independent researcher Michael Allman tackles this problem using a fresh statistical approach designed specifically for the International Skating Union's judging system. The system used in figure skating and ice dance relies on a panel of judges who score every single move and component of a performance. To prevent a single outlier from skewing the results, the highest and lowest scores for each element are discarded, and the average of the remaining marks is used. Allman's work focuses on the 2026 Olympic Winter Games in Milano-Cortina, specifically the ice dance free dance segment, but the method was tested across 142 events from 17 different competitions between 2022 and 2026. The goal was to build a tool that could look at a single judge's marks for two specific skaters and determine if that judge shifted the gap between them in a way that the rest of the panel did not.

The method developed in this paper works by comparing what a judge actually did against what they would have done if they had simply matched the middle ground of the other judges. Imagine a judge giving a score to a gold-medal contender and a silver-medal contender. The researchers first calculate how much that judge's specific marks changed the final point difference between those two skaters. Then, they create a "neutral" version of the event where that judge's marks are replaced by the median score of the other judges. By comparing the real result to this neutral version, the researchers can measure the exact number of points a specific judge added or subtracted from the margin between two skaters. This is not about whether the judge was generally high or low; it is about whether they were unusually high for one skater and unusually low for another, relative to the group.

To ensure this measurement wasn't just picking up on random noise or the natural difficulty of the skaters' programs, the researchers used a technique called a permutation test. This involves shuffling the data thousands of times to see how often a result as extreme as the one observed could happen by pure chance. Crucially, this shuffling process was designed to keep the judge's personal style intact. The test asked: if this judge had the same general tendency to be strict or lenient, but applied it randomly to different skaters, would we still see this specific gap between the gold and silver medalists? If the answer is no, then the specific favoritism is likely real. This approach allowed the researchers to calculate a precise "margin shift" in points for every judge and every pair of competitors, along with a statistical probability that the result was not a fluke.

When applied to the 2026 Olympic Ice Dance Free Dance, the method revealed a striking pattern involving one specific judge. For the pair of skaters who finished first and second, this judge's marks shifted the point difference by 1.19 points in favor of the gold medalist. The statistical analysis showed that the chance of this happening randomly was extremely low, with a probability of 0.0003. When the researchers adjusted for the fact that they were testing many different pairs of skaters, the result remained statistically significant. This 1.19-point shift was larger than the actual official margin between the first and second place teams, which was 0.97 points. In a counterfactual scenario where this judge's marks were replaced by the median of the other judges, the silver medalist would have won. This finding was not isolated; the researchers identified nine events across the database where a single judge's specific bias was strong enough to theoretically change the winner.

The study also looked at the broader landscape of judging across the 142 events analyzed. Out of more than 270,000 individual comparisons between judges and pairs of skaters, the researchers found over 20,000 instances where a judge showed a statistically significant competitor-specific differential. These were not spread evenly; they clustered heavily in certain competitions and disciplines. For example, the 2022 European Championships accounted for a large portion of these significant findings, suggesting that the composition of the judging panel in a specific event matters more than the discipline itself. The researchers also noted that while many judges showed these differentials, they did not find evidence of a single, universal bias shared by all judges, which would have been harder to detect with this relative method.

One of the most important aspects of this work is how it handles the concept of "style." The researchers explicitly ruled out the idea that a judge who is generally harsh or generally lenient is necessarily biased. Their method is designed to ignore those global tendencies. If a judge gives everyone a low score, that is a style difference, not bias. Bias, in this study, is defined strictly as a judge treating one competitor differently from another in a way that the rest of the panel does not. The analysis confirmed that the method is robust: when the researchers checked the results for skaters with very different skill levels, the statistical reliability held up, meaning the tool is not simply confusing a hard-to-judge performance with unfairness.

The findings from the 2026 Olympics were corroborated by a separate check called a "leave-one-judge-out" analysis. In this test, the researchers simply removed the specific judge's marks entirely and replaced them with the average of the remaining judges to see if the winner changed. This independent method confirmed that removing the same judge identified by the main test would indeed flip the gold medal to the silver medalist. This convergence of two different statistical approaches strengthens the conclusion that the observed shift was not an artifact of the math, but a real feature of how that specific judge scored that specific event.

Ultimately, the paper provides a way to measure the invisible weight of a single judge's opinion in a crowded field. It shows that while the system of trimming high and low scores is designed to protect against outliers, it does not always prevent a single judge from tilting the scales between two specific competitors. The study does not claim to prove malicious intent or collusion, nor does it suggest that the entire judging system is broken. Instead, it offers a precise, non-parametric tool that can identify when a judge's specific marks for a pair of skaters deviate from the panel's consensus in a way that alters the final standing. By focusing on the specific margin between competitors rather than overall scores, the research highlights that even in a system with nine judges and strict rules, the outcome of a competition can hinge on the specific, differential choices of one individual.