💻 computer science

A distilled value head breaks search, but search remains an effective teacher, not a move selector: a case study in 19x19 Go

This study demonstrates that while distilling a value head from a strong Go AI can result in a calibrated but non-discriminative evaluator that breaks MCTS search performance, the search process itself remains a highly effective teacher, as training the policy to mimic MCTS visit counts yields significant strength gains despite the search failing to select better moves during inference.

Original authors: Taiki Kojima

Published 2026-08-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Taiki Kojima

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a brilliant but stubborn student how to play a complex strategy game like Chess or Go. You have a super-smart teacher (let's call it "The Oracle") who can look at the board and tell you exactly who is winning. Usually, when you teach a student, you let them practice by playing against the teacher, or you let them think through a few moves ahead to see what happens. In the world of artificial intelligence, this "thinking ahead" is called Monte Carlo Tree Search (MCTS). It's like a simulator that runs thousands of tiny, imaginary games in a split second to figure out the best move.

The big idea in modern AI is that if you combine a smart "brain" (a neural network) with this "simulator" (MCTS), the result should be unbeatable. The brain guesses the move, and the simulator double-checks it. But what if the simulator starts making the brain worse? What if the tool meant to help you think actually confuses you? This is the strange puzzle a researcher named Taiki Kojima tackled. He wasn't trying to build a robot that wins the World Championship; he was trying to understand a glitch where the "thinking" part of the AI broke the "playing" part, even though the "thinking" part was still incredibly useful for teaching. It's a story about how a bad compass can still be a great map for a teacher.

The Glitchy Compass

In the world of computer Go (a game played on a 19x19 grid), researchers usually train AI by having it learn from a super-strong teacher. In this study, the teacher was a famous AI called KataGo. The researcher took a smaller, simpler AI and tried to copy the teacher's brain using a technique called knowledge distillation. Think of this like a student trying to memorize the teacher's homework answers.

Usually, once the student learns the answers, you let them use their own "thinking engine" (MCTS) to double-check their work before they make a move. The expectation was that this would make the student even stronger. But here is where the plot twists: When the researcher let the student use MCTS to pick its moves, the student actually got worse.

The student, playing without any thinking engine, won about 32% of the games against itself. But the moment it turned on the "thinking engine" to help choose moves, its win rate plummeted. It was as if a student who could solve math problems perfectly suddenly started failing every test the moment they were allowed to use a calculator.

The Broken Value Head

The researcher had to figure out why the calculator was broken. Was the calculator itself (the search algorithm) faulty? Or was the student's brain (the neural network) giving the calculator bad information?

To test this, the researcher swapped the student's brain with the teacher's brain for just the final step of the calculation. Suddenly, the win rate skyrocketed to 98.3%. This proved the "calculator" (MCTS) was working perfectly. The problem was entirely in the student's brain.

Specifically, the problem was in the value head. In AI terms, the "value head" is the part of the brain that says, "Is this position good or bad?" The student's value head was actually pretty good at judging the absolute score (it knew if a position was winning or losing overall). But it was terrible at comparing two similar moves side-by-side. It couldn't tell which of two sibling moves was slightly better.

Imagine a judge who can tell you if a painting is "good" or "bad," but when you show them two paintings that are both "good," they can't tell you which one is better. When the AI tried to use this confused judge to pick the best move, it kept making mistakes. The search engine dug deeper, found more confusing comparisons, and made the AI's decisions worse and worse.

The Fix: Stop Using the Compass, Start Using the Map

The researcher tried to fix the broken judge by retraining just that part of the brain. The judge got slightly better at comparing moves, but the AI still couldn't use the search engine to play better. The search engine was still broken as a move selector.

So, the researcher tried a completely different approach. Instead of trying to fix the AI's ability to use the search engine to pick moves, they decided to use the search engine only as a teacher.

Here is the magic trick:

The AI played a game using the search engine to generate a "visit distribution." This is a record of how much time the search engine spent looking at different moves. Even if the search engine picked the wrong final move, it might have spent a lot of time thinking about the right move.
The researcher then trained the AI's brain to copy this "thinking pattern" (the visit distribution) rather than just copying the final move.
They removed the "safety anchor" that usually keeps the AI from changing too much.

The result was shocking. By ignoring the search engine's final decision and only using its "thought process" as a lesson, the AI improved massively.

Generation 2 (using the old method): The AI played no better than its previous version.
Generation 3 (using the search engine as a teacher): The AI won 94.1% of games against the original baseline and 82.4% against the previous version.

This happened using less than half the amount of practice data (self-play games) that the previous generation used. The search engine was a terrible player (it picked the wrong moves 73% of the time), but it was an amazing teacher.

Why This Matters

The paper concludes that there is a huge difference between an AI's ability to make a decision and its ability to teach a lesson.

Think of it like a coach who is terrible at playing the sport themselves. If you ask this coach to play a match, they will lose. But if you ask them to watch a game and tell you where the players should focus their attention, they might be brilliant. The search engine in this study was like that coach: it couldn't pick the winner, but its "attention map" showed the student exactly where to look to learn.

The study also highlights a warning for anyone building AI: Don't trust your "scorecards" too much. The researchers found that standard metrics (like how well the AI ranked moves in a test) often failed to predict if the AI would actually get better at playing the game. Sometimes, the AI got better even when the test scores looked worse.

In the end, this isn't just about Go. The same problem happens in other AI fields, like training language models. Sometimes, the tool you use to generate answers is too flawed to be used directly, but the process of generating those answers holds the secret to making the AI smarter. The lesson? Don't throw away the teacher just because they can't play the game.

Technical Summary: A Distilled Value Head Breaks Search, but Search Remains an Effective Teacher

Problem Statement
In AlphaZero-style systems, Monte Carlo Tree Search (MCTS) typically serves a dual role: selecting moves at inference time and generating training targets for the policy network. It is generally assumed that a system where search improves inference-time play will also benefit from search-derived training targets. This paper investigates a counter-intuitive case in 19x19 Go where a distilled neural network, when combined with MCTS, performed significantly worse than the policy alone. The central research question is whether search can remain a useful "teacher" for policy learning even after it ceases to function as an effective "move selector" at inference time.

Methodology
The study utilized a 19x19 Go network (11.98M parameters, 192 channels, 18 residual blocks) distilled from KataGo. The experimental design was strictly pre-registered to avoid post-hoc reinterpretation of proxy metrics.

Baseline: A distilled network trained on KataGo's search outputs (policy, value, score, ownership) but deployed using only the policy head (argmax) without search.
Search Integration: MCTS (400 simulations, PUCT selection) was applied to the distilled network.
Defect Localization: The author isolated the failure by replacing the distilled network's leaf evaluations with KataGo's single-visit evaluations during search.
Iterative Generations:
- Generation 1 & 2: The policy was updated using a KL-divergence anchor to the previous generation's policy, attempting to improve the trunk while constraining policy drift.
- Generation 3: The training objective for the policy head was changed. Instead of a KL anchor, the policy was trained via cross-entropy to the MCTS visit-count distribution generated by the network itself.
Evaluation: All performance claims were validated via 272-game head-to-head matches from an empty board, adjudicated by KataGo at 100 visits, with 95% Wilson confidence intervals.

Key Results

Search Degrades Inference Performance: Contrary to expectations, adding MCTS to the distilled network reduced its win rate against the policy-only baseline to 0.317. Increasing simulation counts further degraded performance.
The Value Head is the Bottleneck: Replacing the distilled value head with KataGo's evaluation during search immediately restored win rates to 0.983. Analysis revealed the distilled value head was well-calibrated in absolute terms (mean residual -0.0091) but failed to rank sibling positions accurately (Spearman correlation $\rho = +0.005$ vs. +0.146 for the policy head).
Rebuilding the Value Head Was Insufficient: Retraining only the value head improved sibling ranking accuracy (from 0.54 to 0.64) but did not restore the utility of search for move selection (win rate vs. policy argmax remained 0.500).
KL-Anchored Training Failed: Generations 1 and 2, which used KL anchors to constrain policy updates, showed no distinguishable improvement over their predecessors (Generation 2 vs. Gen 1: 0.526 win rate).
Search as a Teacher Succeeded: Generation 3, which trained the policy via cross-entropy to the MCTS visit distribution, achieved a 0.824 win rate against Generation 2 and 0.941 against the original distillation baseline. This improvement was achieved using less than half the self-play data of Generation 2.
Proxy Metrics vs. Reality: Standard proxy metrics (e.g., sibling ranking accuracy, policy evaluation loss) failed to predict playing strength. In some cases, worsening proxy metrics coincided with significant Elo gains.

Key Contributions

Decoupling Inference and Teaching Roles: The paper demonstrates that the inference-time performance of an improvement operator (MCTS) and the teaching value of its output are distinct properties. MCTS failed to select better moves but provided a superior training signal for the policy.
Search-Derived Supervision: The study shows that a search-derived policy target (visit distribution) can substantially improve a student network even when the search procedure itself is too weak to improve inference-time play. This suggests the visit distribution captures structural information unavailable in the policy prior alone.
Methodological Rigor: The work highlights the unreliability of proxy metrics in complex RL systems and advocates for pre-registered decision criteria and direct head-to-head evaluation with confidence intervals.

Significance and Claims
The author claims this is a mechanistic case study rather than a benchmark. The primary significance lies in identifying a structural failure mode where a learned evaluator (value head) remains calibrated but loses ranking fidelity, causing an improvement operator to run "backwards" at inference time.

The paper draws a conceptual parallel to Reinforcement Learning from Human Feedback (RLHF) in Large Language Models (LLMs). It suggests that in both domains, an imperfect selector (MCTS or a reward model) can still generate valuable training data (visit distributions or ranked candidates) even if it cannot reliably select the optimal output during inference. The author concludes that future research on search-based learning systems must explicitly distinguish between whether an operator improves immediate decisions and whether its output improves subsequent learning, noting that the answers to these questions need not be the same.

Limitations
The author acknowledges that these findings are derived from a single architecture, a single random seed, and a specific game (19x19 Go). The self-play loop was not fully self-contained, as value supervision relied on KataGo labels due to games frequently hitting the move cap. The study establishes an empirical phenomenon without fully resolving the underlying mechanism of why the visit distribution contains useful signals despite the search's failure.

The Glitchy Compass

The Broken Value Head

The Fix: Stop Using the Compass, Start Using the Map

Why This Matters

Technical Summary: A Distilled Value Head Breaks Search, but Search Remains an Effective Teacher

More like this