🤖 machine learning

The Algorithm Is Not the Behavior: Learned Priors Override Look-Ahead in a Chess-Playing Neural Network

This study demonstrates that in Leela Chess Zero, the neural network’s internal algorithmic look-ahead correctly solves chess puzzles, but these solutions are systematically overridden in the final output by learned safety priors, proving that the presence of algorithmic structure does not guarantee algorithmic behavior.

Original authors: Elias Sandmann, Sebastian Lapuschkin, Wojciech Samek

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Elias Sandmann, Sebastian Lapuschkin, Wojciech Samek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Idea: The "Second-Guessing" Chef

Imagine you have a world-class chef who is also a brilliant mathematician. You give this chef a complex recipe puzzle: "How do you make this specific dish perfectly?"

The chef’s brain works in two stages:

The Calculation Stage: The chef instantly calculates the exact steps needed. They know, with 100% certainty, that the secret ingredient is a risky, spicy sauce that will make the dish perfect.
The Safety Stage: Just before serving, the chef’s "safety manager" jumps in. This manager is overly cautious. It thinks, "Spicy sauce is dangerous! What if the customer hates it? Let’s just serve plain water instead. It’s boring, but it’s safe."

So, the chef knows the perfect answer, calculates it correctly, but then ignores it at the last second because they are afraid of taking a risk.

This paper argues that this is exactly what happens inside Leela Chess Zero, one of the strongest chess-playing AI programs in the world.

The Discovery: "Forgotten Puzzles"

The researchers used a tool called the "Logit Lens." Think of this like an X-ray machine for the AI’s brain. It lets us peek inside the AI’s thinking process layer by layer, rather than just seeing the final move it makes.

They found a strange phenomenon they call "Forgotten Puzzles."

The Scenario: The AI is faced with a chess puzzle where there is only one winning move. Often, this winning move involves a sacrifice (giving up a piece to win the game).
The Internal Thought: When the researchers looked inside the AI’s brain, they saw that in the middle layers, the AI had already figured out the correct, winning move. It was calculating the future steps correctly. It "knew" the answer.
The Final Output: But in the very last layer, right before the AI made its move, it changed its mind. It swapped the brilliant, winning sacrifice for a boring, safe move that would eventually lose the game.

The AI didn’t fail to solve the puzzle; it failed to act on the solution.

Why Does This Happen? The "Safety Prior"

The researchers asked: Why does the AI change its mind at the last second?

They discovered that the AI has a strong "Safety Prior." This is a learned habit from its training. In most chess games, being safe and protecting your pieces is a good strategy. The AI has learned to prioritize "safety" and "avoiding threats" above all else.

Early Layers: The AI is aggressive and tactical. It sees the winning sacrifice.
Late Layers: The AI becomes conservative. It thinks, "Sacrificing a Queen is risky! Let’s just move the Queen to a safe square."

The "Safety Manager" in the AI’s brain overrides the "Mathematical Solver." The AI sacrifices its victory to satisfy its need for safety.

The Proof: Steering the AI

To prove that "safety" was the culprit, the researchers performed a causal experiment. They used a technique called "Steering."

Imagine you could gently push the AI’s brain in a specific direction. The researchers pushed the AI’s brain to care less about safety and more about aggression.

Result: When they did this, the AI stopped second-guessing itself. It recovered 61.7% of the "forgotten puzzles." It went back to making the brilliant, winning sacrifices it had originally calculated.

This proved that the AI had the capability to solve the puzzle all along; it was just being held back by its own cautious habits.

The Big Takeaway

The main conclusion of the paper is profound: Just because an AI has the "algorithm" (the ability to calculate the right answer) doesn't mean it will show that "behavior" (actually doing the right thing).

Algorithmic Structure: The AI has the circuitry to solve the problem.
Algorithmic Behavior: The AI actually outputs the correct answer.

The paper shows these two things can be disconnected. An AI can internally know the truth but output a lie (or a bad move) because of conflicting internal priorities—like a safety habit overriding a logical conclusion.

Why This Matters

This isn't just about chess. The authors suggest this might happen in other AI systems, like large language models (chatbots).

A chatbot might internally know the correct factual answer to a question.
But its "safety" or "politeness" training might override that fact, causing it to give a vague or incorrect answer to avoid seeming rude or risky.

The paper warns us: We cannot trust an AI’s output just because we know it can do the math. We need to look inside its "brain" to see if hidden habits are overriding its intelligence.

Technical Summary: The Algorithm Is Not the Behavior: Learned Priors Override Look-Ahead in a Chess-Playing Neural Network

Problem Statement
Recent mechanistic interpretability research has identified learned algorithms within neural networks, ranging from modular arithmetic to search and planning in game-playing agents. However, it remains unclear whether the presence of such algorithmic structure guarantees algorithmic behavior. Specifically, it is unknown if a model can internally compute a correct solution via a learned algorithm yet output an incorrect answer due to overriding factors. This paper investigates this tension in Leela Chess Zero (LC0), the strongest neural chess engine, where prior work (Jenner et al., 2024) identified learned look-ahead mechanisms. The authors ask whether these algorithmic computations are always reflected in the final output or if they can be suppressed by other learned biases.

Methodology
The authors extend the "logit lens" technique—originally designed for decoder-only language models—to the encoder-only, Post-Layer Normalization (Post-LN) transformer architecture of LC0’s policy network.

Post-LN Logit Lens Extension: Standard logit lenses project intermediate activations through the final layer normalization and unembedding matrix. In Post-LN architectures, layer normalization is applied after residual connections, creating non-linear dependencies that complicate this projection. The authors resolve this by applying zero ablation to all sublayer outputs beyond a given layer $\ell$ , while preserving the subsequent layer normalizations and DeepNorm scaling factors. This allows for the extraction of interpretable move probability distributions (policies) at every intermediate layer.
Identification of "Forgotten Puzzles": Using the extended logit lens, the authors analyze LC0 on a dataset of chess puzzles. They identify a phenomenon termed "forgotten puzzles," where the correct tactical solution (often a sacrifice or forcing move) appears with high probability in intermediate layers but is systematically overridden in the final output by a different, often safer, move.
Verification of Look-Ahead: To determine if the failure stems from a breakdown in the look-ahead mechanism, the authors replicate the causal analyses from Jenner et al. (2024) on the forgotten puzzle dataset. These include activation patching (measuring causal importance of square representations), attention head ablation (testing specific information flow between move targets), and bilinear probing (testing linear decodability of future moves).
Concept Preference Analysis and Steering: To identify the overriding factor, the authors measure how layer-wise policies prioritize specific chess concepts (e.g., king safety, threats) using Stockfish 8’s handcrafted evaluation terms as proxies. They then perform gradient-based activation steering against these safety-related concepts to test if suppressing them recovers the correct tactical solutions.

Key Contributions

Extended Logit Lens for Post-LN Encoders: The authors provide a method to extract highly interpretable intermediate policies from LC0’s Post-LN architecture, demonstrating that playing strength and puzzle-solving capability improve consistently with network depth.
Discovery of Forgotten Puzzles: They identify a systematic failure mode where intermediate layers correctly solve tactical puzzles (including immediate checkmates), but the full model outputs an incorrect move. This occurs in a substantial fraction of difficult puzzles (7,901 positions in their filtered dataset).
Causal Evidence of Prior Override: The authors provide causal evidence that learned safety priors, rather than algorithmic failure, are responsible for the incorrect outputs. By steering the model against late-layer safety preferences, they recover 61.7% of forgotten puzzles.

Results

Capability Progression: The logit lens reveals a three-phase progression in LC0: rapid improvement in early layers (0–5), a plateau in middle layers (6–10), and sharp strengthening in late layers (11–15).
Look-Ahead Functionality: Analyses on forgotten puzzles show that look-ahead operates normally. Future moves of the correct continuation are represented in intermediate layers, are causally important (activation patching and attention ablation confirm this), and are linearly decodable (probing accuracy is high). This rules out a failure of the algorithmic mechanism itself.
Safety Prior Override: Characterization of forgotten puzzles reveals that the model overwhelmingly replaces forcing tactical solutions (e.g., sacrifices) with quiet, positional alternatives. Concept preference analysis shows that late layers increasingly prioritize safe play (increasing own king safety, reducing opponent threats) over aggressive play.
Steering Recovery: Gradient-based steering against safety-related concepts (specifically reducing emphasis on mitigating opponent threats and increasing own king safety) at layer 13 recovers 61.7% of forgotten puzzles with minimal regressions on other puzzles. This confirms that safety priors override the algorithmically computed solutions.

Significance and Claims
The paper argues that the presence of learned algorithmic structure in a neural network does not guarantee that the network’s output will reflect it. A model can internally solve a problem through a learned algorithm (look-ahead) and still output the wrong answer due to competing learned priors (safety). This finding challenges the assumption that identifying algorithmic circuits is sufficient for predicting model behavior. The authors suggest that mechanistic understanding can identify such failure modes and suggest principled interventions (like steering) to address them. They note that this tension between learned algorithms and distributional priors may extend to other domains, including language models, where internal representations may encode correct answers that are not output due to sycophancy or other biases.