Application of machine learning to monster level prediction in tabletop RPG game design

Technical Summary: Application of Machine Learning to Monster Level Prediction in Tabletop RPG Game Design

Problem Definition
Designing balanced adversaries is a labor-intensive and critical component of Tabletop Role-Playing Game (TTRPG) development. In systems like Pathfinder Second Edition, monsters are defined by numerous numerical attributes (e.g., Strength, Armor Class, Hit Points, spell capabilities) that collectively determine their power, summarized as an ordinal "level." Currently, estimating a monster's level from its statistics relies on expert human judgment or extensive playtesting, both of which are error-prone and inefficient. The authors frame the task of predicting a monster's level from its attributes as a tabular ordinal regression problem, where the target variable is discrete, ordered, and imbalanced, with intervals between classes not necessarily representing equal increments in power.

Methodology
The study utilizes a dataset of 6,007 monsters derived from publicly available Pathfinder Second Edition sourcebooks. The data was curated to reflect the chronological design process, with publication dates serving as a proxy for the temporal evolution of game mechanics.

• Feature Engineering: Three feature sets were constructed:
  1. Basic (8 features): Core attributes (Str, Dex, Con, Int, Wis, Cha), Hit Points (HP), and Armor Class (AC).
  2. Extended (20 features): Adds Perception, saving throws, attack bonuses, expected damage calculations, and spell-related metrics (max spell level, spell DC, spell attack modifier).
  3. Full (33 features): Includes movement speeds, counts of immunities/resistances, and a granular breakdown of the number of spells available at each of the nine spell levels.
• Evaluation Protocols: To avoid data leakage and simulate real-world design workflows, the authors rejected random splitting in favor of:
  1. Chronological Split: Training on older data and testing on newer monsters.
  2. Expanding Window: A time-series approach where the model is trained on an initial subset, tested on the next batch, and then retrained with the new data added, repeating until all data is utilized.
• Models Evaluated: The study benchmarked 16 models across four categories:
  • Regression with Rounding: Linear models (Ridge, SVM) and tree-based models (Random Forest, LightGBM) with post-processing rounding.
  • Dedicated Ordinal Regression: Ordered Random Forest (ORF), Simple Approach to Ordinal Classification (SAOC), Ordered Logistic Regression (ORD), and Threshold-based models (Immediate-Threshold).
  • Probabilistic Models: Gaussian Process Ordinal Regression (GPOR).
  • Neural Networks: Multi-layer perceptrons (MLPs) adapted with ordinal-aware losses and architectures, including Cumulative Link Models (CLM), NNRank, OR-CNN, CORAL, CORN, and CONDOR.
• Metrics: Performance was assessed using Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE), specifically their macro-averaged versions (MAEM, RMSEM) to account for class imbalance. Additional metrics included accuracy, accuracy-at-1 (allowing a ±1 level error), and Somers' D for ordinal rank correlation.

Key Results

• Superiority of Tree-Based Ensembles: Tree-based ensemble methods, specifically Random Forest, LightGBM, SAOC (using Random Forest as a base), and ORF, consistently outperformed all other approaches. They achieved near-perfect ordinal ranking (Somers' D ≈ 0.99) and high predictive accuracy (Acc@1 ≈ 99%).
• Limitations of Neural Networks: Despite incorporating specific ordinal inductive biases (e.g., rank consistency constraints in CORAL, CORN, CONDOR), neural network models generally underperformed compared to tree-based ensembles. They exhibited higher variance and lower stability, particularly under the expanding-window protocol, suggesting that for this specific tabular dataset size and structure, ensembles are more robust.
• Feature Importance: The inclusion of extended and full feature sets generally improved performance. Correlation analysis revealed strong interdependencies between defensive capabilities (AC, HP, saving throws) and offensive/spell metrics, validating the feature engineering choices.
• Evaluation Strategy Impact: The expanding-window protocol yielded more pessimistic (and realistic) performance estimates compared to random splits, highlighting the importance of temporal validation in game design contexts where new content builds upon existing mechanics.

Key Contributions

1. Dataset Creation: The authors present the first publicly available dataset specifically for TTRPG monster-level prediction, derived from Pathfinder Second Edition, released under a permissive open-source license.
2. Problem Formalization: The work formally defines monster level prediction as a tabular ordinal regression task, providing a comprehensive benchmark of 16 distinct models.
3. Domain-Specific Evaluation: The paper introduces and validates chronological and expanding-window evaluation protocols tailored to the iterative nature of game design, moving beyond standard random splits.
4. Explainability: Through feature importance analysis and error distribution studies, the authors demonstrate that the best-performing models align with human intuition and the underlying rules of the game system.

Significance and Claims
The paper claims that machine learning can reliably approximate designer judgments for monster balancing, serving as an effective computer-aided tool for TTRPG system design. The results suggest that ML models can accelerate the creation of balanced adversaries, providing immediate feedback on monster power during the design phase. The authors position this work as a novel application of predictive ML in game design, distinct from AI used for playing games. They conclude that while there is room for improvement in accuracy, the current models are robust enough to warrant deployment in practical applications supporting game designers and Game Masters. The study emphasizes that this approach is particularly valuable for reducing the labor-intensive nature of creating monster compendiums and for enabling the rapid prototyping of unique monsters.