Technical Summary: DSLE - A Learning Environment for Dark Souls Boss Encounters

Problem Statement

Reinforcement learning (RL) and neuroevolution research have historically relied on standardized environments (e.g., Atari, ViZDoom, StarCraft II) that expose structured state information through purpose-built interfaces. However, many modern commercial games lack such interfaces, limiting their use as reproducible benchmarks for real-time, third-person action role-playing games (RPGs). While prior work (DSAPI) demonstrated the feasibility of interacting with Dark Souls: Remastered via screen capture and computer vision, that system was limited to a single boss encounter, relied on visual state estimation, and lacked automated reset capabilities. There remains a gap for a reproducible benchmark that supports real-time reactive combat, visual observations, automated resets, and repeatable episodes across a diverse set of challenges within an unmodified commercial game.

Methodology

The authors introduce the Dark Souls Learning Environment (DSLE), a containerized platform that exposes all 22 boss encounters of Dark Souls: Remastered through a Gymnasium-style interface.

Environment Architecture

• Execution: DSLE runs the unmodified Windows game inside an Ubuntu-based Docker container using Wine. Rendering is handled via a virtual X11 display, with DXVK translating DirectX calls to Vulkan for GPU execution.
• State Extraction: Unlike previous vision-only approaches, DSLE utilizes direct memory access (via /proc/<pid>/mem) to retrieve structured game state (player/boss health, death flags, lock-on status). This ensures reliable reward computation and termination checks.
• Automation: The system automates episode resets by swapping scenario save states, navigating menus via template matching, and transferring control to the agent. It supports headless execution and distributed training across multiple isolated instances on a single machine.
• Interface:
  • Observation: Default input is a grayscale frame (600×800) from the player's perspective, including the HUD.
  • Action Space: A discrete set of 14 actions (movement, attacks, healing, dodging, parrying). Actions are held for a fixed duration (250ms), decoupling control rate from rendering frame rate.
  • Rewards: Sparse terminal rewards (win/loss) combined with shaped intermediate rewards (damage dealt/taken).

Experimental Protocol

To establish a baseline and assess difficulty, the authors defined DSLE-5, a representative subset of five bosses covering distinct control challenges:

1. Asylum Demon: Tutorial melee fight.
2. Capra Demon: Spatially constrained arena with multi-targets.
3. Chaos Witch Quelaag: Environmental hazard fight.
4. Ornstein and Smough: Multi-target, dual-phase fight.
5. Gwyn, Lord of Cinder: Fast-paced final boss.

Five baseline methods were evaluated on DSLE-5 under a fixed protocol (100,000 steps, 5 seeds, no per-boss tuning):

1. Random Policy: Uniform random action selection.
2. Expert System: A simple rule-based controller (heal when low, move/attack otherwise).
3. PPO & DQN: Deep RL agents trained from visual input using standard architectures (Mnih et al., 2015).
4. SCOPE (Sparse Cosine Optimized Policy Evolution): A neuroevolutionary baseline using a derivative-free approach with discrete cosine transform sparsification and CMA-ES optimization.

Key Contributions

1. DSLE Platform: A fully containerized, reproducible benchmark for Dark Souls: Remastered that supports all 22 boss encounters, automated resets, and direct memory access for state verification.
2. DSLE-5 Benchmark Suite: A curated subset of five bosses designed to represent the spectrum of difficulties in the game (melee, spatial, hazard, multi-target, and high-pressure combat).
3. Comprehensive Baseline Evaluation: The first systematic evaluation of multiple agent families (random, scripted, deep RL, and neuroevolution) on this domain, establishing that current methods struggle significantly beyond the tutorial level.
4. Stochasticity Analysis: Demonstration that despite fixed save states, the game environment exhibits inherent stochasticity (e.g., diverging health trajectories after identical action sequences), necessitating feedback-based control rather than open-loop planning.

Results

The evaluation reveals a steep difficulty gradient where current methods fail to generalize beyond the tutorial boss:

• Asylum Demon (Tutorial):

  • Expert System: Achieved a 63% win rate.
  • SCOPE: Achieved a peak 43% win rate (best generation), outperforming previous NEAT results (35%).
  • PPO/DQN: Showed negligible learning, with win rates ≤0.33%.
  • Random: 0% win rate.

• Harder Bosses (Capra, Quelaag, O&S, Gwyn):

  • Zero Wins: None of the five methods (including the expert system and SCOPE) defeated any of the four harder bosses.
  • Learning Failure: PPO and DQN showed no measurable learning within the 100,000-step budget (approx. 9–41 wall-clock hours per run). Reward curves remained pinned near the death penalty floor.
  • Failure Modes: Agents failed in two distinct regimes:
    1. Cramped/Multi-target: Immediate death (median survival 7–19s) in arenas like Capra Demon and Gwyn.
    2. Hazard/Range: Long survival (52–68s) but zero damage dealt in fights like Quelaag, where agents were killed by environmental hazards or ranged attacks before engaging.

• Extended Study: An exploratory study using "advantaged" level-50 character stats allowed SCOPE to win a few early-game bosses (e.g., Capra Demon at 96%), but the majority of the 22-boss suite remained unwon, confirming that difficulty extends well beyond equipment scaling.

Significance and Claims

The paper claims that DSLE provides a rigorous, reproducible benchmark for studying adaptive control policies in real-time, high-dimensional, and sparse-reward environments. The results demonstrate that Dark Souls: Remastered possesses a multi-tiered difficulty structure that current state-of-the-art model-free RL and neuroevolution methods cannot solve without significant architectural changes or curriculum learning.

The authors emphasize that the failure of agents is not merely a lack of compute but a fundamental inability to handle the specific control pressures of the game (timing, positioning, target switching, and hazard avoidance) within realistic budgets. The paper positions DSLE as a stepping stone for future research into:

• Methods for sparse, long-horizon, real-time control.
• Curriculum and transfer learning across the graded boss sequence.
• Imitation learning from human or scripted demonstrations.
• Multi-task and continual learning across distinct but related tasks.

The work concludes that while DSLE opens the door to these advanced research directions, the current landscape of game-playing agents remains unsolved for the majority of the benchmark's challenges.