📊 statistics

Engine-Equal, Human-Unequal: A Reproducible Outcome Skew in Engine-Assessed Equal Chess Positions

This paper demonstrates that in chess positions evaluated as objectively equal by strong engines, human game outcomes exhibit stable, reproducible biases favoring one side over the other across different player groups and time periods, proving that engine evaluations alone are insufficient to predict human results.

Original authors: Jesung Park

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jesung Park

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Hidden Bias in a "Perfectly Balanced" Game

Imagine you are a referee in a massive, global sports league where millions of people play the same game every day. To keep things fair, you have a super-smart robot judge that can look at any moment in the game and tell you exactly how likely each player is to win. If the robot says, "This is a 50/50 split," you assume the game is perfectly balanced, like a seesaw with two kids of the exact same weight. But here's the twist: what if the robot is only judging the physics of the seesaw, while the kids playing are actually human? Humans get tired, they get nervous, they have favorite moves they've practiced a thousand times, and they sometimes make mistakes that the robot, who never gets tired, simply doesn't understand.

This paper dives into that exact mystery, but in the world of chess. It asks a simple question: When a super-computer says a chess position is "dead even," do human players actually treat it as dead even? The study uses data from millions of real games played online. It looks at positions where the computer is 100% sure the score is zero (meaning no advantage for either side) and then checks what actually happened when humans played them. The researchers aren't trying to prove that chess is broken; they are trying to see if the computer's "perfect score" tells the whole story about human behavior. If the computer says "equal," but humans keep winning or losing more often than they should, it means there is a hidden layer of difficulty or advantage that the robot's math can't see.

The Great "Equal" Chess Mystery

Meet the "Engine-Equal, Human-Unequal" discovery. The authors, led by Jesung Park, went on a detective hunt through over 16 million chess games played on Lichess in October 2025. They were looking for a very specific type of chess position: one where the world's strongest chess engine, Stockfish 18, looked at the board and said, "This is perfectly balanced. White and Black have exactly the same chance of winning." The engine is so smart that it can look 28 moves deep into the future to make this call.

The researchers found 1,661 of these "perfectly equal" positions. But when they looked at how humans actually played these specific spots, they found something weird. Even though the computer said the game was a coin toss, the humans didn't treat it like one. In some of these "equal" positions, White players won more often than they should have. In others, Black players had the upper hand. It wasn't a random fluke; it was a pattern.

The "Ghost in the Machine" Test
To make sure this wasn't just a glitch or a lucky guess, the researchers played a clever game of "split the difference." They took all the games for a specific position and split them into two groups: Group A (played by one set of people) and Group B (played by a completely different set of people who never played against Group A). If the "unfairness" was just a mistake in the data, Group A and Group B would show different results. But they didn't. If a position looked slightly better for White in Group A, it looked exactly the same way in Group B.

The connection between the two groups was incredibly strong. The researchers calculated a "replication slope" of 0.69. Think of this like a shadow: if you see a shadow in the morning (Group A), you can predict with high confidence what the shadow will look like in the afternoon (Group B). The fact that this shadow exists across two totally different groups of people proves that the "unfairness" is a real property of the chess position itself, not just a quirk of who happened to be playing.

The "Thinking Time" Clue
The paper didn't just look at who won; it looked at how they played. They checked the clocks. They found that when a player was on the "disfavored" side of these supposedly equal positions, they took longer to make their move. It was as if they were staring at the board, scratching their heads, and thinking, "Wait, this feels harder than the computer said." The side that the position secretly favored moved faster, while the other side paid a "thinking tax," spending about 23% more time on their move. This proves that the imbalance isn't just a number on a scoreboard; it's a real, felt difficulty that humans experience in the moment.

What This Paper Rules Out
The authors were very careful to knock down other possible explanations. They proved that:

It's not about skill: They adjusted for player ratings. Even when a 1500-rated player faced a 1500-rated player, the skew remained.
It's not just "bad openings": They looked at specific families of openings. Even within the same opening (like the "Italian Game"), some specific board setups were harder for one side than the other.
It's not a time-travel glitch: They checked games from eight months later (June 2026) and the pattern was still there.
It's not just the engine being wrong: They used a second, different engine (Leela Chess Zero) to verify the positions, and the results held up.

The Big "But"
The paper is very honest about what it doesn't know. It proves that the skew exists and is reproducible, but it doesn't prove why. The authors suggest two main possibilities, but they can't tell which one is the real culprit without a new, controlled experiment:

The Position is Harder: The board setup itself might be tricky for one side, even if the computer says it's equal.
The Players are Biased: Maybe the players who choose to enter these specific positions are secretly better at them than their rating suggests, or they have practiced these lines more.

The authors call this an "observational" result. They found a stable, reproducible pattern where humans struggle more on one side of a "perfectly equal" board, and they even found that the struggling side spends more time thinking about it. But they leave the final "why" for a future study to solve.

The Takeaway
The most important finding is that a computer's "perfect score" isn't the whole story. Just because a robot says a chess position is a 50/50 coin flip doesn't mean humans will treat it that way. There are hidden currents in the game—tiny, specific difficulties that make one side feel heavier than the other. And the best part? This isn't just about chess. It's a reminder that when we use algorithms to judge human decisions, we might be missing the messy, human reality that happens right under the robot's nose.

Technical Summary: Engine-Equal, Human-Unequal

Problem Statement

Modern chess engines, such as Stockfish, reduce complex board positions to a single scalar value (centipawn evaluation). When an engine evaluates a position as "dead even" (e.g., within ±10 centipawns of zero at high search depth), the natural assumption is that the position is balanced for human players as well. This paper challenges that assumption, asking whether positions deemed objectively equal by engines actually yield balanced outcomes (50% win rate for White) when played by humans.

The authors posit that a scalar engine evaluation is not a sufficient statistic for human outcomes. While engine self-play models show that centipawn values map to different win/draw/loss distributions depending on material, the paper investigates whether "engine-equal" positions ( $SF \approx 0$ ) harbor reproducible, directional outcome skews (systematic advantages for White or Black) that persist after controlling for player ratings and opening families.

Methodology

Data Source and Selection

Dataset: The study utilizes the Lichess standard-rated games dump from October 2025 (26.4M games, 16.1M analyzed occurrences) and an out-of-sample month from June 2026 (25.0M games).
Position Filtering: The authors identified positions that are:
1. Reached by humans in sufficient volume (discovery half occurrence $\ge$ 50).
2. Evaluated by Stockfish 18 (depth 12, MultiPV 1) as $|eval| \le 50$ cp.
3. Deep Verified: A subset of 7,465 positions was re-evaluated at increasing depths (12, 16, 20, 25, 28). A position is included in the final panel ( $N=1,661$ ) only if its depth-28 evaluation is within $\pm 10$ cp of zero and remains stable (last three depths agree within 50 cp).
Constraints: Games must have both players rated $\ge$ 1000, occur at ply $\le$ 40, and be tagged with ECO codes.

Experimental Design

The core of the methodology is a reproducibility test designed to rule out five mundane explanations for observed skews: sampling noise, rating artifacts, account-specific quirks, family-level opening effects, and time-control artifacts.

Account-Disjoint Split: The primary test splits the data into two groups (A and B) based on completely disjoint sets of player accounts. A game is retained only if both players fall into the same group.
Rating Calibration: To isolate position-specific effects from player strength, the authors fit a fractional-logit GLM to predict White's expected score based on the rating gap, mean rating, and time control. The "skew" ( $\delta$ ) for a position is defined as the mean residual (actual score minus predicted score) across its occurrences.
Within-Family Fixed Effects: To rule out the possibility that the skew is merely a property of a specific opening family (e.g., "The Sicilian favors White"), the analysis subtracts the weighted mean skew of the opening family from each position's skew. The regression is performed on these demeaned values.
Replication Slope ( $\hat{\beta}$ ): The primary estimand is the slope of the regression of Group B's skew on Group A's skew (within opening families).
- $\hat{\beta} \approx 0$ : The skew is noise.
- $\hat{\beta} > 0$ : The skew is a stable property of the position, reproducible across disjoint player populations.
Inference: Significance is determined via permutation tests where position skews are shuffled within their opening families (preserving family structure but breaking position-specific links).

Secondary Measures

Clock Burden: The study analyzes per-move clock data to see if the "disfavored" side (the side with the negative skew) spends more time thinking than the favored side.
Robustness Checks: The analysis is repeated across temporal splits (early vs. late month), rating bands (Low, Medium, High), different time controls (Blitz, Rapid, Classical), and with stricter engine evaluation bands.

Key Results

1. Reproducible Outcome Skew

The primary finding is a statistically significant replication slope of $\hat{\beta} = 0.691$ (95% CI [0.646, 0.736], permutation $p=0.001$ ) on the primary account-disjoint axis.

Interpretation: A position's outcome skew measured in one group of players predicts the skew in a completely disjoint group of players.
Magnitude: The typical skew is small (median $|\delta| \approx 0.018$ , roughly a 2% deviation in White's score), but it is stable.
Direction: The skew is two-sided; some positions favor White, others Black.
Noise Correction: When corrected for measurement noise (using split-half reliability, ICC = 0.655), the disattenuated slope rises to $\approx 1.05$ , suggesting that if measurement were perfect, the skew would carry over at full strength.

2. Robustness and Existence

The existence of the skew is invariant across multiple perturbations:

Temporal: Replicates across calendar halves ( $\hat{\beta} = 0.811$ ) and an out-of-sample month (June 2026, $\hat{\beta} = 0.904$ ).
Rating Bands: Skews are rank-stable across Low (<1500), Medium (1500–1800), and High ( $\ge$ 1800) rating bands.
Time Controls: Replicates separately in Blitz and Rapid (though Classical data is too sparse for a standalone estimate).
Popularity: The slope increases monotonically (0.69 to 0.94) as the panel is restricted to more popular positions, indicating that the effect is not a "winner's curse" of noise mining but a real signal that becomes clearer with more data.

3. Behavioral Counterpart (Clock Time)

At engine-equal positions, the side to move that is "disfavored" by the position's skew spends significantly more time thinking.

Result: A 0.10 swing in skew corresponds to approximately 23% more think time for the disfavored side.
Specificity: This is a one-sided effect; the favored side moves faster than neutral controls, while the disfavored side moves slower. This suggests the imbalance is visible in the act of playing, not just in the final score.

4. Nature of High-Skew Positions

The 71 positions with replicated high skews ( $|\delta| \ge 0.05$ ) include both sharp tactical lines (e.g., Englund Gambit) and quiet positional structures (e.g., Giuoco Pianissimo).

Sharpness: High-skew positions tend to have lower candidate move entropy and less settled engine evaluations across search depths, suggesting they are tactically sharper or more volatile than the panel average.
Not Just Traps: The skew exists even in calm, non-tactical structures, indicating it is not solely a result of known opening traps.

Significance and Claims

What the Paper Establishes

Engine Scores are Not Sufficient Statistics: A machine evaluation of "zero" does not imply a balanced human decision problem. The gap between engine evaluation and human outcome is not random noise; it is a reproducible, directional property of specific positions.
Position-Specific Difficulty: The skew is a property of the "naturally-reached position" (the board state plus the players who choose to reach it). It survives within-opening-family controls, meaning it is not merely a result of one side of an opening family being stronger overall.
Behavioral Evidence: The skew manifests physically in the game via clock usage, confirming that the imbalance is felt by players in real-time.

Limitations and Causal Claims

The paper explicitly states that the result is observational and non-causal.

Selection Bias: The authors cannot rule out that the skew is caused by who selects into the position (repertoire selection). For example, players who choose a specific line might be systematically better at it than their rating suggests, or they might be better prepared.
Scope: The claim is about the "naturally-reached position," not the position in isolation.
Future Work: Establishing whether the position itself causes the difficulty (intrinsic difficulty) versus the players' preparation requires a randomized controlled trial (assigning players to lines), which is the subject of a separate pre-registered companion study.

Conclusion

The paper provides a large-scale, reproducible demonstration that "engine-equal" positions are often "human-unequal." This skew is a stable, measurable phenomenon that persists across disjoint player groups, time, and rating levels, and is accompanied by measurable cognitive costs (increased think time) for the disfavored side. The findings suggest that algorithmic ground truth in decision domains may not fully capture the complexity of human performance, even in highly structured environments like chess.