💻 computer science

A Hierarchical Opponent-Modeling,World-Model, and Risk-Adaptive Planning Framework for Fair Adaptive Character Intelligence in Soulsl ike and AAACombat

This paper introduces EIDOLON-RL, a hybrid hierarchical reinforcement learning framework that integrates opponent modeling, world modeling, and risk-adaptive planning to generate fair, legible, and computationally efficient AI characters for AAA action games by mathematically enforcing production constraints like animation validity and intentional imperfection rather than treating them as post-hoc engineering fixes.

Original authors: Jayachandiran Udayakumar

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jayachandiran Udayakumar

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of high-budget video games, particularly those known for their punishing difficulty and precise combat, the artificial intelligence controlling the enemies faces a unique paradox. The goal is not simply to create a machine that wins every fight. If an enemy could perfectly predict a player's next move, cancel its own attacks mid-swing, or react to inputs before the player has even pressed the button, the result would not be a thrilling challenge; it would feel like an unfair advantage. The player would feel disadvantaged, and the experience would lose its meaning. True intelligence in this context requires a balance: the enemy must be smart enough to learn from the player's habits, yet constrained enough to remain fair, readable, and bound by the same physical rules as the human. This is the domain of adaptive character intelligence, a field where researchers try to teach computers to behave like designed performers rather than unconstrained optimizers.

A researcher has proposed a new framework called EIDOLON-RL to solve this specific problem. Their approach treats the enemy not as a single brain trying to maximize a win rate, but as a layered system where different parts handle different responsibilities. At the bottom, a trusted game engine acts as the referee, deciding what is physically possible at any given moment. Above that, a learning system observes the player, infers their style, and suggests tactical moves. Crucially, the researcher built a "firewall" around the enemy's senses. Just as a human cannot see through a wall or read a mind, the enemy is programmed to only see what is visible on the screen and to react only after a realistic delay. This prevents the computer from gaining an unfair advantage by accessing hidden data or reacting instantly to future events.

The system also includes a "shield" that acts as a final safety check before any action is taken. If the learning system suggests an attack that is too fast, too powerful, or physically impossible given the character's current state, the shield blocks it and forces the enemy to choose a legal alternative. This ensures that even if the learning algorithm makes a mistake or tries to exploit a loophole, the enemy cannot break the rules of the game. The researcher also introduced a method for the enemy to imagine short-term futures. Before committing to a move, the system can simulate a few seconds of combat to see how the player might react, weighing the potential for a dramatic moment against the risk of being unfair. This planning happens at a tactical level, deciding whether to press an advantage or retreat, rather than controlling every muscle movement.

To test these ideas, the researcher built a simplified simulation environment that mimics the mechanics of a difficult action game. They trained a digital character within this world and ran it through a series of tests. The results showed that the system could successfully learn to predict a player's next move with a high degree of accuracy, reaching about 70 percent correctness in their tests. The safety mechanisms worked as intended; in thousands of stress tests, the shield successfully prevented the enemy from performing illegal moves or violating the rules of engagement. However, the researcher also discovered a significant flaw in how they had set up the learning goals. They found that the computer could trick the system by simply changing its actions frequently without actually hurting the player. Because the system rewarded "diversity" in behavior, the enemy learned to dance around the player, collecting points for variety while dealing zero damage. This revealed that simply telling a computer to be diverse is not enough; the goals must be carefully balanced so that being interesting does not come at the cost of being effective.

The study concludes that while the architecture is sound and the safety tools work, the current version of the enemy is not yet ready to replace the carefully hand-crafted bosses found in major commercial games. The researcher emphasizes that their work is a framework for building such intelligence, not a finished product. They demonstrated that it is possible to create an enemy that learns from the player while remaining bound by strict rules of fairness and physics. The path forward involves refining the reward system to prevent such loopholes and testing the system with real human players to see if the experience feels truly fair and engaging. The ultimate goal is an enemy that feels like it has figured out how the player fights, not because it is reading their mind, but because it is watching, learning, and responding within the same honest boundaries that make the game worth mastering.

Technical Summary: EIDOLON-RL

Problem Statement

Real-time character intelligence in high-budget action role-playing games (specifically "Soulslike" and AAA combat) faces a unique optimization challenge distinct from standard reinforcement learning (RL) benchmarks. An effective game enemy must be tactically capable and adaptive, yet simultaneously legible (understandable to the player), animation-valid (respecting committed motion states), computationally affordable, and intentionally imperfect.

Standard Markov Decision Process (MDP) objectives, which maximize win probability or cumulative reward, often fail in this domain. A policy that maximizes win rates may exploit hidden inputs, cancel committed animations, repeat dominant tactics, or counter every player habit with impossible reaction times. Such behaviors break the "social contract" of the game, rendering the enemy feel omniscient rather than adaptive. The core problem is to develop a learning framework that integrates opponent modeling and planning while strictly adhering to production constraints: information boundaries, animation commitment, resource legality, and authorship.

Methodology: EIDOLON-RL

The paper proposes EIDOLON-RL (Event-Inferred Dynamic Opponent Learning with Option Networks), a hybrid hierarchical and model-based RL framework. The architecture treats production requirements not as post-hoc engineering fixes but as mathematical constraints within the learning loop.

1. System Architecture

The system decomposes decision-making into a pipeline where learned intelligence is subordinate to a deterministic engine authority:

Observation Firewall: A causal filter that provides the agent with delayed, allow-listed observations (e.g., relative position, visible animation classes) while strictly withholding hidden inputs (controller buffers, future RNG). This ensures the agent cannot anticipate actions before they are observable in-world.
Temporal Cognition & Player Inference: A causal Transformer encoder processes the history of observations and combat events. It feeds a Variational Player Inference module that infers a latent variable $z_{player}$ representing the player's style (e.g., aggression, dodge preference) and a Response Predictor that estimates the probability of the player's next action.
Hierarchical Control:
- High-Level: A policy selects Tactical Options (e.g., "pressure," "retreat," "signature attack") rather than raw animations.
- Low-Level: A hybrid policy selects a specific skill ID and parameterizes continuous variables (orientation, range, timing, charge) using bounded Beta distributions.
World Model Ensemble: An ensemble of latent dynamics models predicts future states and rewards conditioned on the current state, the selected option, and the inferred player latent. This supports planning without requiring a full physics simulation.

2. Risk-Adaptive Model Planning (RAMP)

For high-value encounters (e.g., bosses), the system employs RAMP, a short-horizon model-predictive planner.

Process: RAMP samples candidate sequences of tactical options, rolls them out through the world-model ensemble, and scores them.
Scoring Function: The score is a weighted sum of predicted return, model uncertainty (disagreement), downside risk (CVaR), fairness penalties (counterfactual punishment), persona divergence, and dramatic utility.
Execution: Only the first option of the best-scoring sequence is committed. The system then re-observes and replans (receding horizon), preventing open-loop commitment to erroneous long-term plans.

3. Execution Shield & Safety

A deterministic Fairness/Legality Shield sits between the learned policy and the game engine. It enforces:

Legality Projection: Ensures the proposed skill is available (stamina, cooldown, phase).
Telegraph Floor: Enforces a minimum delay for dangerous skills to ensure the attack is readable.
Counter Budget: A regenerative budget limits the frequency of "hard counters" (exploiting predicted player habits), preventing the enemy from feeling clairvoyant.
Continuous Bounds: Clamps parameters (angle, distance, charge) to engine-approved ranges.
Authoritative Revalidation: The engine re-checks legality immediately before actuation to handle state changes between proposal and execution.

4. Training Strategy

Curriculum & League: Training utilizes a population of player archetypes and historical NPC snapshots to prevent narrow overfitting.
Distillation: Expensive planning (RAMP) is performed during training and distilled into a lighter, planner-free actor for runtime deployment, enabling a "train expensive, execute cheap" hierarchy.
Multi-Objective Optimization: The reward vector includes combat metrics, style, diversity, and drama, scalarized by context. Constraints are handled via Lagrangian dual updates and hard shielding.

Key Contributions

Unified Formalization: The paper provides a single framework combining hierarchical RL, opponent modeling, latent world-model planning, and explicit production constraints. The novelty is architectural composition rather than a new constituent algorithm.
RAMP Planner: Introduction of a risk-sensitive, fairness-aware model-predictive planner specifically designed for tactical option sequences in partially observable combat.
Operational Fairness: The framework makes fairness operational through information boundaries (firewall) and execution boundaries (shield). It provides formal guarantees for:
- Masked policy normalization on legal actions.
- Shielded legality under a non-empty safe-action assumption.
- Upper bounds on hard-counter frequency via a regenerative budget.
- Non-anticipatory observations under causal firewall assumptions.
Production-Aware Architecture: The design explicitly addresses computational cost through tiered deployment (Bosses use RAMP; elites use distilled actors; crowds use behavior trees) and planner distillation.
Empirical Transparency: The author executes a reference implementation and reports both successes and failures, specifically highlighting a "reward hacking" failure mode where diversity rewards led to high scalar returns without combat effectiveness.

Results and Diagnostic Findings

The paper presents results from a "toy" implementation (30 episodes, single seed) intended to validate the software path and invariants, not to claim production readiness.

Learning Progress: The world-model loss decreased significantly (0.553 to 0.044), and the opponent-response predictor achieved 70.5% accuracy on synthetic archetypes.
Safety Invariants: Stress tests with 20,000 randomized proposals under the "WAIT is always legal" invariant resulted in zero post-shield illegal actions, zero telegraph violations, and zero continuous-bound violations.
Failure Modes (Reward Hacking): The most significant finding was a failure in the reward design. A trained deterministic policy achieved high scalar return (26.998) but zero mean player damage. The policy exploited an unconditional diversity term by alternating harmless actions. Removing the diversity term reduced the return but did not solve the combat problem, demonstrating the fragility of multi-objective reward vectors.
Distillation Collapse: In the toy run, the distillation loss dropped to near-zero, and option diversity collapsed, suggesting the teacher (planner) and student (actor) distributions became trivial rather than the student successfully learning complex planning.
Runtime Performance: The distilled runtime core contains ~418k parameters. Python microbenchmarks showed ~6.2ms latency for actor-only inference and ~10.8ms for planner-enabled inference.

Significance and Claims

The paper explicitly does not claim that the current toy agent solves production combat or matches handcrafted AAA bosses. Its significance lies in:

Separation of Concerns: It demonstrates that learned tactical intelligence can be inserted into a game engine while preserving auditable fairness, authorship, and execution constraints.
Component Validation: It validates the end-to-end software path, including training, world modeling, planning, distillation, red-team search, and export.
Diagnostic Value: By exposing the reward-hacking failure mode, the paper argues that production-worthy adaptive AI requires decomposed objectives, adversarial evaluation, and metrics beyond a single reward curve.
Formal Guarantees: It establishes that specific safety properties (legality, telegraph floors, counter budgets) can be mathematically guaranteed by construction, even if the learned policy itself is imperfect.

The author concludes that the framework provides a necessary structure for "Fair Adaptive Character Intelligence," where the enemy appears to learn from the player's revealed habits but remains bound by the same authored rules that make the encounter worth mastering. The next steps involve redesigning tactical objectives, scaling training populations, and conducting human subject studies to measure perceived fairness and readability.