🤖 AI

Chess\_db: A framework for working with large chess game datasets

This paper introduces Chess_db, a logic programming framework designed to efficiently process large chess datasets by converting PGN files into databases and leveraging open-source key-value stores to provide near-instant access to historical game statistics and position outcomes.

Original authors: Nicos Angelopoulos, Jan Wielemaker

Published 2026-07-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nicos Angelopoulos, Jan Wielemaker

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of chess not just as a game of kings and queens, but as a massive, living library of human strategy. For decades, this library was guarded by experts who memorized books and magazines, but today, the library has exploded. Thanks to the internet, millions of games are played every day, creating a data deluge that no single human brain could ever hold. This is the realm of "data science" applied to chess: the art of using computers to sift through these millions of moves to find patterns, predict outcomes, and help players get better. But here's the catch: standard computer tools often struggle to handle this sheer volume of information without getting bogged down or running out of memory. It's like trying to find a specific grain of sand in a beach that keeps growing every second. To solve this, researchers need a new kind of "shovel" that can dig through the data quickly, store it efficiently, and let players ask questions like, "What move did the best players make in this exact situation?"

This paper introduces Chess_db, a clever toolkit built by Nicos Angelopoulos and Jan Wielemaker that acts as a super-efficient librarian for these massive chess datasets. Think of Chess_db as a magical translator and a high-speed filing cabinet rolled into one. It takes the messy, text-based files where games are currently stored (called PGN files) and converts them into a structured, digital format that a computer can understand instantly. The authors built this using Prolog, a programming language that thinks in logic and rules rather than just following a linear list of commands, which makes it perfect for the complex rules of chess.

The paper's main discovery is that by using a specific type of database called a key-value store (specifically one named RocksDB), they can organize chess positions so that the computer can find information about them almost instantly, even when dealing with millions of games. They tested this by feeding the system 10 million games from the "Elite Lichess" database—a collection of high-level matches. They found that while older, simpler database methods started to slow down and struggle after a few hundred thousand games, their new system kept running, though with noticeable performance degradation as the dataset grew.

However, the authors are careful to note that this isn't a magic wand that solves every chess problem. They explicitly argue against the idea that older, simpler database methods (like standard SQLite) are suitable for storing the "position tables" needed for such huge datasets; their tests showed these older methods became unusably slow after just 1.3 million games. They also clarify that while their system can store any game, the most useful information for training players is usually found in the early parts of the game (the "opening"), where many different games often share the same starting positions. As games get deeper and more unique, the value of these pre-calculated tables drops off.

The team measured their success by timing how long it took to insert games into the database. They found that for the first 3 million games, the system was incredibly fast (under 3 minutes per 10,000 games). As the database grew to 10 million games, the speed slowed down significantly, taking about 8 to 10 minutes per 10,000 games. Furthermore, they discovered that restarting the insertion process into a large database incurred a heavy penalty, with delays of up to 5 hours before performance stabilized. They also compared their system to another database called Berkeley DB, which failed completely after 1.3 million games, taking over an hour for small tasks.

In short, Chess_db suggests that by using modern, high-performance storage tools, we can build a personal "chess brain" that holds millions of games and answers questions in a flash, provided we account for the time it takes to build and maintain such a large collection. This doesn't mean the computer will play the game for you, but it gives players and coaches a powerful way to study the past to master the future. The authors are confident in their results based on these specific experiments, but they also admit that real-world usage might involve smaller, more specialized collections of games, and they see future work in connecting these tools to visual interfaces so players can see the data in action.

Technical Summary: Chess_db: A Framework for Working with Large Chess Game Datasets

Problem Statement
The paper addresses the challenge of managing and analyzing the exponentially growing volume of chess game data, particularly in the context of player training and preparation. While the game of chess has seen a resurgence in popularity, fueled by online platforms and the availability of "objective truth" via strong AI engines, the sheer scale of available data (e.g., millions of games on Lichess) renders traditional methods of accessing and aggregating information inefficient. Specifically, the Portable Game Notation (PGN) format, while universal for storage, is impractical for rapid retrieval of aggregate statistics, such as win frequencies for specific positions or continuations. Furthermore, existing open-source tools often lack the ability to efficiently handle large datasets within a logic programming environment, necessitating a framework that can manipulate games in memory and persist them to scalable back-end databases.

Methodology
The authors present Chess_db, a suite of logic programming tools implemented as an installable SWI-Prolog pack. The methodology is divided into in-memory manipulation and database storage:

In-Memory Representation:
- Board Dictionary: Chess positions are represented as SWI-Prolog dictionaries. The board is encoded numerically (squares 1–64, column-wise), with pieces encoded as integers (1–6 for White, 7–12 for Black). The dictionary tracks critical state information including castling rights (cwk, cwq, cbk, cbq), en passant squares (eps), the half-move clock (hmv), full moves (fmv), and the side to move.
- PGN Parsing: The system implements a grammar to parse PGN files into Prolog terms (pgn(Info, Moves, Res, Orig)). It supports both full in-memory loading and incremental processing, allowing the system to handle files larger than available RAM by processing games one at a time.
Database Storage and Position Encoding:
- Position Key Generation: To enable near-instant access to position statistics, the authors employ a compact hexadecimal encoding for board positions. This method, adapted from the Stockfish engine, encodes square occupancy (16 nibbles) and piece identity (including castling rights, en passant availability, and side to move) into a unique string. This avoids the need to store half-move clocks or ply counters, which are deemed unnecessary for opening analysis where the position table is most useful.
- Back-Ends: The framework supports two database back-ends:
  - proSQLite: Used for storing game metadata (info), moves (move), and original PGN text (orig). It is noted as unsuitable for large-scale position tables due to performance degradation after hundreds of thousands of games.
  - RocksDB: A high-performance key-value store used specifically for the position table (posi). It maps the unique hexadecimal position key to a value containing continuation moves, win frequencies, and references to the top $N$ games reaching that position.

Key Contributions

Chess_db Framework: A modular SWI-Prolog pack that bridges logic programming with data science, providing predicates for parsing PGNs, manipulating board states, and interfacing with databases.
Efficient Position Encoding: A novel implementation of a compact hexadecimal position key within a Prolog environment, optimized for use in key-value databases to facilitate rapid lookup of position statistics.
Scalable Architecture: The system demonstrates the ability to process and store millions of games using incremental loading and RocksDB, overcoming the memory and performance limitations of standard SQLite for position-heavy datasets.
Open Source Integration: The work reinforces the open-source ethos of the chess community by providing free, declarative tools for data analysis, complementing existing resources like Lichess and Stockfish.

Experimental Results
The authors conducted experiments using the "Elite Lichess" dataset, comprising monthly PGN files from January 2023 to November 2025 (approximately 10 million games total).

Performance: When building a position database with a move depth limit of 25, the system processed 10 million games. Insertion times averaged under 3 minutes per 10,000 games for the first 3 million, rising to 8–10 minutes per 10,000 games for the final million.
Storage: The resulting position database for 10 million games occupied approximately 11.2 GB of disk space.
Comparison: The authors compared RocksDB against Berkeley DB. While RocksDB maintained performance, Berkeley DB deteriorated rapidly, becoming unusable after approximately 1.3 million insertions (taking over 30 minutes for 10,000 insertions).
Hardware: Experiments were conducted on a dated Linux desktop (AMD A8-6600K, 16GB RAM), suggesting the approach is viable even on modest hardware, though the authors note that typical user scenarios (individual players or coaches) would likely involve smaller, more curated datasets.

Significance and Claims
The paper claims that Chess_db provides a solid basis for future work with chess games within Logic Programming environments. Its significance lies in:

Promoting Logic Programming in Data Science: It demonstrates how declarative programming can effectively handle large-scale data manipulation and storage tasks.
Accessibility: By utilizing standard, open-source database technologies (SQLite and RocksDB) and a high-level interface, the tool lowers the technical barrier for players, coaches, and data scientists to analyze large game collections.
Utility: The framework allows users to narrow down large game collections into smaller, manageable PGN files compatible with standard visualization tools (e.g., Scid, Chessbase) that cannot handle massive datasets.
Openness: It aligns with the unique ecosystem of chess, where open-source tools and free access to data remain dominant, offering a transparent alternative to commercial database solutions.

The authors conclude that while the current implementation focuses on the opening phase (depth limit 25) where position repetition is most frequent, the framework establishes a foundation for further development, including integration with graphical interfaces and potential optimizations in move disambiguation.

More like this