Communicating Chess Strategies in Natural Language

Technical Summary: Communicating Chess Strategies in Natural Language

Problem Statement
While chess engines have achieved superhuman playing strength, their underlying strategies—typically represented as complex search trees and numerical evaluations—are difficult for human players (and even skilled ones) to comprehend. Existing approaches to explaining engine output, such as feature learning, concept extraction, or commentary generation, often focus on justifying a single move or expressing high-level themes. However, these methods frequently fail to communicate a complete, executable strategy that guides a player through future states and opponent responses. The authors identify a gap in "faithfully communicating chess engine-derived strategies in natural language" that is both informative enough to guide play and compact enough to be digestible.

Methodology
The paper proposes a framework for Chess Strategy Verbalization, which involves two main components: a pipeline for generating strategy descriptions and a robust evaluation framework.

1. Verbalization Pipeline:

   • Partial Strategy Tree Construction: To manage the exponential growth of the game tree, the authors construct a partial strategy tree. This tree includes the engine's best moves for the solving player but prunes the opponent's responses, retaining only the top $k_{strat}$ (e.g., 3) engine-ranked moves at each opponent node. This balances compactness with the retention of key strategic branches.
   • LLM-Based Verbalization: Large Language Models (LLMs) are used to convert the partial strategy tree (serialized as JSON) and/or high-level chess concepts (e.g., "fork," "king safety") into natural language descriptions.
   • Design Variations: The study examines three orthogonal design choices:
     • Strategy Guidance: Providing the LLM with the full partial strategy tree.
     • Concept Guidance: Providing the LLM with keyword concepts associated with the puzzle.
     • Self-Reflection: Using an LLM critic to evaluate and iteratively refine the generated description for clarity and completeness.

2. Evaluation Framework:

   • Downstream Utility: Instead of relying on surface-level metrics (e.g., BLEU) or subjective LLM-as-a-judge assessments, the authors evaluate descriptions based on their downstream utility. A "solving player" (either an LLM or a human) attempts to solve chess puzzles using the generated description as a guide.
   • Tree-Expanded Puzzle Playing: To address the limitation of evaluating only the "main line" (the most common engine continuation), the authors introduce an iterative evaluation procedure. At each opponent node, the system samples multiple possible opponent moves (up to $k_{eval}$), rolls them out one ply, and selects the branch that results in the worst outcome for the solving player. This mimics the game-theoretic concept of exploitability, ensuring the strategy is robust against suboptimal or unexpected opponent moves.
   • Metrics: Performance is measured by the final board state's evaluation (converted to win probability) relative to a "No Strategy" baseline and an "Optimal" engine baseline.

Key Contributions

1. Task Definition: The formal introduction of Chess Strategy Verbalization as a distinct task focused on generating pedagogically useful, natural language descriptions of engine strategies.
2. Framework Proposal: A novel pipeline combining chess engines for strategy extraction and LLMs for verbalization, coupled with a rigorous evaluation framework that tests strategy robustness beyond the main line.
3. Empirical Insights:
   • Natural language descriptions significantly improve the performance of both LLM and human players compared to having no guidance, though they generally underperform direct access to the raw JSON strategy tree (indicating a "lossy" compression).
   • Strategy vs. Concept Guidance: Providing the LLM with the concrete strategy tree (JSON) yields better results than providing only high-level concepts. Pure concept-based descriptions often lack the necessary detail for executable strategies.
   • Evaluation Beyond Main Line: Evaluating strategies only on the main line is insufficient; robustness against off-main-line opponent responses is critical for a strategy's true utility.
   • LLM vs. Human Alignment: There is a misalignment between what LLMs and humans find useful. LLMs often perform better with LLM-generated descriptions, while humans benefit more from human-written descriptions. Furthermore, LLM-generated descriptions may capture the main line well but fail to communicate off-main-line variations effectively for human readers.

Results
Experiments on the Lichess Puzzle dataset using models like o3 and gpt-oss-120b demonstrate that:

• Verbalized strategies improve solving player scores over a "No Strategy" baseline.
• The o3 verbalizer generally produces higher-quality descriptions than gpt-oss-120b.
• Self-reflection improves descriptions only when the verbalizer has sufficient chess knowledge to identify genuine omissions; otherwise, it can degrade performance.
• Increasing the evaluation parameter $k_{eval}$ (testing more opponent branches) consistently degrades the performance of all methods, revealing that many strategies are fragile and exploitable when opponents deviate from the main line.
• Natural language descriptions are significantly more compact than JSON representations while maintaining comparable performance, suggesting natural language is a viable medium for strategy communication.

Significance
The paper claims that natural language is a "promising and interpretable medium" for communicating strategic information to both human and LLM players. It highlights the necessity of moving beyond simple move justification to full strategy communication. Crucially, the work underscores the importance of evaluating strategies for robustness (exploitability) rather than just main-line performance and reveals a critical misalignment between LLM and human preferences in strategy descriptions. The authors conclude that while their approach generates useful descriptions, the field must address how to better align automated evaluations with human judgment and how to generate descriptions specifically tailored to human cognitive needs.