Technical Summary: DungeonBench

Problem Statement

Current artificial intelligence benchmarks often under-test rules-rich tactical reasoning: the capacity to make optimal decisions when geometry, timing, resource scarcity, objectives, and complex rule interactions occur simultaneously. While games like Go, StarCraft, and NetHack have advanced continuous control and planning, they often abstract away the heterogeneous action spaces and intricate rule dependencies found in domains like tabletop role-playing games. In tactical domains, a locally optimal action (e.g., high damage) can be strategically suboptimal if it depletes a scarce resource needed for a future encounter or fails to account for specific terrain and reaction mechanics. The challenge lies not merely in long-horizon planning or large branching factors, but in state-contingent option valuation under a strict, executable rule set.

Methodology

The DungeonBench Framework

DungeonBench is a benchmark and simulation engine built on the 2014 Dungeons & Dragons System Reference Document (SRD). It translates the vast, rule-heavy combat system of D&D into a measurable decision-making task. The system is designed to retain mechanics often abstracted away in simplified combat simulators, including:

• Complex Mechanics: Reactions, concentration, persistent zones, summons, legendary actions, and multi-classing features.
• 3D Tactical Space: A 2.5D grid system supporting verticality, flight, falling, and line-of-sight occlusion.
• Resource Persistence: Tracking hit points, spell slots, consumables, and short-rest timing across multiple encounters.

Decision Interface and Action Decomposition

To manage the massive action space inherent in D&D (e.g., selecting targets, spell levels, and 3D coordinates for area-of-effect spells), DungeonBench employs hierarchical action decomposition:

• Legal Option Generation: A rules engine generates a complete, indexed list of legal executable options for every decision point. Agents do not need to parse syntax or discover legal moves; they must rank and select from the provided list.
• Decomposed Choices: Instead of presenting a flat, massive action space, the interface breaks complex actions into sequences. For example, placing a Fireball involves selecting an X coordinate, then a Y coordinate conditioned on X, and a Z coordinate conditioned on (X, Y). Similarly, movement is presented first as a reachable destination, with route selection decomposed only if distinct routes yield different simulator-visible consequences (e.g., opportunity attacks or hazards).
• Observation: Agents receive a complete tactical observation including combatant statistics, map state, active effects, and the current decision context. The benchmark assumes full observability to isolate tactical valuation from perception or memory tasks.

Benchmark Tracks

The benchmark evaluates agents on two distinct tracks:

1. Encounter Track: Tests local tactical competence in single fights. Success is measured by winning the fight or achieving specific objectives (e.g., protecting a target, controlling a zone).
2. Day Track: Links multiple encounters through persistent party state and limited short rests. This track forces agents to trade off immediate tactical advantage against future survivability. Agents are given the sequence of future encounters to test their ability to budget resources (spell slots, hit dice, consumables) across a "day" of adventuring.

Evaluation Protocol

The evaluation focuses on frontier language models (e.g., GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro) acting as policies.

• Control: The benchmark party is controlled by the tested model, while all opposing forces are controlled by a fixed heuristic planner.
• Interface: Models interact via a structured text interface, selecting from engine-generated legal option IDs. They can use lookup tools to retrieve specific rule details (e.g., spell effects, movement costs) rather than having rules embedded in every prompt.
• Metrics:
  • Encounter: Win rate, final party hit-point fraction, survivor fraction, and rounds to termination.
  • Day: Full-clear rate (winning all encounters in a day), final resource state, and survivor fraction.

Key Contributions

1. DungeonBench Engine: A combat-rules engine and benchmark suite covering the combat-relevant 2014 D&D SRD, capable of resolving complex interactions like legendary actions, concentration, and 3D positioning without open-ended adjudication.
2. Dual-Track Evaluation: The introduction of Encounter and Day tracks to separately measure local tactical play and medium-horizon resource budgeting, highlighting the divergence between "winning a fight" and "winning the day."
3. Unified Policy Surface: A shared decision stream supporting heuristic controllers, language-model policies, and reinforcement learning agents, allowing for direct comparison of tactical option valuation across different model classes.

Results

Evaluation of frontier language models reveals a significant gap between local tactical competence and long-horizon resource management:

• Encounter Performance: Top models (Gemini 3.1 Pro and GPT-5.5) achieved win rates of approximately 82–83% on single encounters, preserving high party health and survivor fractions. Performance varied significantly among models, with some (e.g., DeepSeek V4) using more tokens and tool lookups without achieving higher win rates.
• Day Performance: Performance dropped precipitously on the Day track. The best models cleared only 2 out of 5 linked encounter days.
  • Resource Budgeting Failures: Models often failed to conserve resources (spell slots, hit dice) for later encounters, leading to "wipes" in the final rooms despite winning earlier fights.
  • Objective vs. Attrition: Failures were qualitatively different from Encounter failures. Some days were lost due to objective sequencing errors (e.g., failing to secure a map objective) rather than gradual attrition, while others showed models winning early fights but exhausting resources before the final boss.
• Decision Complexity: The Day track exposed agents to significantly longer horizons (328–391 decisions per episode vs. 79–109 in Encounter) and larger option sets, particularly in Objective and Boss scenarios where local option counts could exceed 1,000.

Significance and Claims

The paper claims that full tactical observations do not saturate the benchmark. While frontier models can often win isolated encounters, they struggle with the discipline required for resource budgeting and rule-aware tactical planning over linked sequences.

The significance of DungeonBench lies in its ability to:

• Isolate Tactical Reasoning: By providing complete state information and legal options, it removes perception and memory as confounding variables, focusing evaluation on the valuation of choices under complex rules.
• Expose Strategic Blind Spots: The Day track demonstrates that high performance in single episodes does not guarantee competence in environments where immediate gains must be weighed against future constraints.
• Provide a Realistic Testbed: By retaining the "texture" of D&D combat (conditions, reactions, verticality, and resource scarcity) without simplifying them away, DungeonBench offers a rigorous environment for testing AI agents on heterogeneous, state-dependent decision-making.

The authors conclude that DungeonBench provides a measurable, repeatable framework for evaluating agents on the specific challenge of rules-rich tactical reasoning, distinguishing between local optimization and global strategic planning.