🤖 machine learning

Chessformer: A Unified Architecture for Chess Modeling

The paper introduces Chessformer, a unified encoder-only transformer architecture that integrates geometric attention bias and square-token representation to simultaneously achieve state-of-the-art performance in human move prediction, significantly enhance playing strength in Leela Chess Zero, and enable granular interpretability, demonstrating that aligning model design with domain geometry yields superior results across multiple objectives.

Original authors: Daniel Monroe, George Eilender, Philip Chalmers, Zhenwei Tang, Ashton Anderson

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniel Monroe, George Eilender, Philip Chalmers, Zhenwei Tang, Ashton Anderson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the game of chess as a giant, complex city. For years, AI researchers have been trying to build different types of "city guides" to navigate this city. Some guides are built to be the ultimate tour guides (winning every game), others are built to mimic human tourists (predicting what a regular person would do), and others are built to be transparent maps (so we can see exactly how they make decisions).

The problem, according to this paper, is that researchers have been building three separate, incompatible guidebooks for these three jobs. They are often clunky, inefficient, and don't quite fit the unique "geometry" of the chess city.

The authors introduce Chessformer, a new, all-in-one guidebook that does all three jobs better than the previous separate ones, using a clever new design.

Here is how they did it, broken down into simple concepts:

1. The New Map: "Squares as Tokens"

Most previous AI models looked at the chessboard like a long list of moves (like reading a story) or as a blurry photo. The authors argue this is like trying to navigate a city by reading a list of street names or looking at a satellite photo from space—it loses the specific layout.

Instead, Chessformer treats every single square on the 64-square board as its own unique "token" (a building block of data).

The Analogy: Imagine a city where every single street corner has its own dedicated GPS satellite. The AI doesn't just see "a move"; it sees the specific relationship between the corner where a piece is standing and the corner it wants to go to. This fits the natural shape of the chessboard perfectly.

2. The Secret Sauce: "Geometric Attention Bias" (GAB)

Standard AI models use a generic way to understand "where" things are (like knowing that "next to" means the same thing in a book as it does on a map). But in chess, "next to" means something very different depending on the piece. A knight's "next to" is an "L" shape; a rook's is a straight line.

The authors invented a new tool called Geometric Attention Bias (GAB).

The Analogy: Think of GAB as a dynamic, shape-shifting ruler.
- If the AI is looking at a Rook, the ruler stretches out in straight lines to see how far the Rook can move.
- If it's looking at a Knight, the ruler instantly reshapes itself into an "L" to see the Knight's jumps.
- It even changes based on the game stage. In the early game, it might look at the whole board; in the endgame, it might focus intensely on the King's safety.
- This allows the AI to "feel" the geometry of the board instantly, rather than having to learn it from scratch every time.

3. The Three Wins

The paper claims Chessformer wins in three specific areas, proving that one architecture can do it all:

A. Beating Humans at Predicting Humans (The "Psychic" Win)

The Goal: Predict what a human player will do next.
The Result: They created a model called MAIA-3. It predicts human moves with 57.1% accuracy.
The Magic: Previous models needed massive amounts of computer power (355 million "brain cells" or parameters) to get close to this. MAIA-3 did it with only 79 million parameters (less than a quarter of the size). It's like predicting a friend's move with a small, efficient notebook instead of a supercomputer.

B. Beating the Best Engines (The "Grandmaster" Win)

The Goal: Play chess better than the best existing engines.
The Result: They plugged Chessformer into Leela Chess Zero, a famous open-source engine.
The Magic: This upgrade added over 100 "Elo" points of strength (a measure of skill). To put that in perspective, the difference between a top human player and a world champion is often only 50 points. This upgrade was so strong that the new Leela engine actually defeated Stockfish (the long-time world champion engine) in major tournaments.

C. Seeing Inside the Brain (The "Transparent" Win)

The Goal: Understand why the AI made a move.
The Result: Because Chessformer treats every square as a specific token, we can look at its "attention" (what it is focusing on) and say, "Ah, it is looking at the square where the Knight is pinned."
The Magic: Previous models were like black boxes; you saw the input and the output, but the middle was a mystery. Chessformer is like a transparent window. The authors found the AI was learning specific chess concepts like "forks" (attacking two pieces at once) and "pins" (trapping a piece), and they could point exactly to the squares on the board where these patterns were happening.

The Big Takeaway

The paper's main lesson is that alignment matters. When you build an AI to understand a specific world (like chess), you shouldn't force that world into a generic box. Instead, you should build the AI's brain to match the shape of the world.

By building a model that respects the geometry of the chessboard (squares as tokens) and the rules of movement (dynamic GAB), the authors created a system that is stronger, more human-like, and easier to understand all at once, while using far less computing power than before.

Technical Summary: CHESSFORMER

Problem Statement

The field of chess modeling is currently fragmented, with disparate architectures optimized for specific, often conflicting objectives: maximizing raw playing strength, predicting human move choices, and enabling mechanistic interpretability. Existing approaches often misalign with the geometric structure of the chess domain. For instance, transformer-based models frequently employ tokenization schemes (such as move sequences or linearized board representations) or positional encodings (like rotary embeddings) that impose artificial one-dimensional structures on a two-dimensional board, potentially hindering performance. This fragmentation raises the question of whether a single, unified architecture can simultaneously achieve state-of-the-art performance in playing strength, human move prediction, and interpretability by aligning its internal representations with the domain's inherent geometry.

Methodology

The authors introduce Chessformer, an encoder-only transformer architecture designed to unify these objectives. The core design principles are:

Square-Token Representation: Unlike prior methods that tokenize move sequences or convolutional channels, Chessformer treats the 64 board squares as distinct tokens. Each square is represented by a 12-dimensional vector indicating the presence of one of the 12 piece types (or empty). The board is flipped to the perspective of the side to move. To capture history, the model concatenates representations of the current board state with the previous $n$ states (typically $n=7$ ), resulting in a fixed sequence of 64 tokens per time step.
Geometric Attention Bias (GAB): To address the permutation invariance of standard self-attention and the unique geometry of chess, the authors propose GAB. This is a novel, dynamic positional encoding that generates attention biases based on the board state itself.
- Mechanism: The board state tokens are compressed via linear projections and layer normalization into a global representation. This representation is then projected to generate a set of bias templates ( $h \times 4096$ ) for each attention head. These biases are reshaped to $h \times 64 \times 64$ and added additively to the dot-product attention logits before the softmax.
- Function: GAB allows the model to adapt attention patterns to the specific geometry of the position (e.g., piece mobility, pawn structures) rather than relying on static relative or absolute positions. It acts as a dynamic mixer of geometric templates.
Output Heads:
- Value Head: Predicts game outcomes (win, draw, loss) via mean pooling of the encoder output followed by a linear projection.
- Policy Head: Utilizes a "source-destination" formulation. Query vectors correspond to starting squares, and key vectors correspond to destination squares. The policy logits are computed via scaled dot-product, yielding a $64 \times 64$ matrix representing all possible moves. This design naturally supports the "from-to" structure of chess moves and enhances interpretability.
Training Regimes:
- Human Move Prediction (MAIA-3): Trained on a re-sampled dataset of Lichess blitz games to predict human moves given board states and player ratings. Ratings are incorporated via "soft embeddings" interpolated between weak and strong engine levels.
- Playing Strength (Leela-CF): Trained via supervised distillation from a search-augmented Leela Chess Zero (Lc0) oracle. The model learns to predict the policy distribution and value of the MCTS-augmented oracle without performing tree search during inference.

Key Contributions

Unified Architecture: Chessformer is the first architecture demonstrated to advance the state of the art simultaneously in raw playing strength, human move prediction, and interpretability.
Geometric Attention Bias (GAB): A new dynamic positional encoding that adapts to the board state, outperforming static absolute and relative biases. It enables the model to capture domain-specific geometric relationships (e.g., piece movement patterns) more effectively.
Square-Token Design: By treating squares as tokens, the architecture aligns with the visual and geometric nature of chess, reducing parameter load and enabling direct attribution of attention and activations to specific board squares.
Source-Destination Policy Head: A policy head that explicitly models the action space as a matrix of source-destination pairs, facilitating granular interpretability.

Results

1. Human Move Prediction (MAIA-3)

Performance: The 79M-parameter MAIA-3 model achieved a move-matching accuracy of 57.1% on the ALLIE test set.
Efficiency: This surpasses the previous state-of-the-art (ALLIE-ADAPTIVE-SEARCH, 55.9%) which required a 355M-parameter model with search enabled. MAIA-3 achieves this with less than a quarter of the parameters and without search.
Scalability: The 5M-parameter MAIA-3 model reached 55.4% accuracy, matching the previous state-of-the-art with 70 times fewer parameters.
Skill Levels: Gains were observed across all skill levels, with the most significant improvements occurring at higher ratings, suggesting the model better captures the consistency of strong play.

2. Playing Strength (Leela-CF)

Distillation: A 191M-parameter Chessformer (Leela-CF) was distilled from a search-augmented Lc0 oracle.
Elo Gains: In searchless settings, Leela-CF achieved an Elo of 2466 (using value maximization) and 2374 (using policy selection), significantly outperforming baseline models like AC-270M (2299 Elo) and Leela-CNN (2168 Elo) while using fewer FLOPs.
Tournament Victories: When integrated into full Leela Chess Zero configurations, the Chessformer-based engine defeated Stockfish (the perennial world champion engine) in multiple major tournaments, including the TCEC Cup 11 and TCEC Swiss 6 and 7. This represented a playing strength increase of over 100 Elo points over the previous best Lc0 configurations.

3. Interpretability

Attribution: The square-token design allows attention patterns and activations to be directly attributed to specific board squares.
Feature Analysis: Using cross-layer transcoders, the authors identified highly interpretable features corresponding to tactical motifs (forks, pins, checks) and strategic concepts (king safety, pawn structures).
GAB vs. Semantic Attention: Analysis revealed that GAB biases are relatively consistent across positions but vary significantly by query square (capturing local geometry), whereas dot-product attention varies significantly across positions but is consistent across query squares (capturing global semantic context). This suggests a complementary division of labor where GAB handles geometric constraints and dot-product handles semantic evaluation.

Significance and Claims

The paper claims that Chessformer demonstrates that aligning a model's tokenization, positional encoding, and output design with the underlying structure of a domain yields simultaneous gains in performance, human compatibility, and interpretability.

Efficiency: The architecture achieves superior results with significantly fewer parameters and computational resources compared to prior state-of-the-art methods.
Human Compatibility: By modeling human play without search and with high accuracy, the model bridges the gap between superhuman engine strength and human-like behavior, potentially aiding in teaching and collaboration.
Mechanistic Insight: The domain-aligned architecture makes mechanistic interpretability more natural, allowing researchers to directly map internal model states to chess concepts.

The authors conclude that while GAB is currently specialized to chess, the broader lesson is that adapting model architecture to a domain's geometric structure allows AI models to flexibly adapt to both task mastery and human compatibility. They acknowledge limitations, noting that GAB's benefits may depend on domains where geometric relations are central and that deeper mechanistic analysis is needed to fully understand the interplay between GAB and semantic attention in planning and evaluation.