Technical Summary: Coachable Agents for Interactive Gameplay

Problem Statement

Reinforcement learning (RL) has achieved state-of-the-art performance in real-time control domains, from robotics to video games. However, standard RL agents typically learn a single, near-optimal behavior to solve a task. This lack of flexibility is a limitation in scenarios where users wish to assert control over how a task is performed, rather than just that it is performed. While users can request specific styles from generative AI (e.g., "paint in an impressionist style"), such real-time style control has been unavailable for agents operating in continuous control settings.

Existing approaches like multi-task, multi-objective, or goal-conditioned learning focus on teaching agents to accomplish multiple distinct tasks. This paper argues that framing "cleaning quietly" and "cleaning quickly" as separate tasks obscures the spectrum of user preference. Instead, the authors propose Coachable Agents: agents trained to perform a single core task in a variety of different "styles," allowing users to select and modify these styles on-demand at execution time.

Methodology

The framework builds upon Universal Value Function Approximators (UVFAs), extending them to handle scenarios where the core objective is fixed but multiple valid behaviors exist. The authors introduce Style-Conditioned UVFAs, where the reward function is decomposed into two components:
$$r(s, a; \theta) = r_{task}(s, a) + r_{style}(s, a, \theta)$$

• $r_{task}$: A fixed reward capturing the primary objective (e.g., winning a fight, finishing a lap).
• $r_{style}$: A style-dependent reward driven by a multi-dimensional style vector $\theta$. This vector includes weights for linearly combining rewards, thresholds for gating/ramping, and setpoints that penalize deviation from desired ranges.

Key Technical Innovations

1. Algorithmic Adaptation (Cat-RAC): To handle the high-dimensional control domains of AAA video games (specifically Horizon Forbidden West), the authors developed Categorical Regularized Actor-Critic (Cat-RAC), a new member of the Soft Actor-Critic (SAC) family. This algorithm addresses the complexity of action spaces and behavioral variance found in modern games.
2. Training Infrastructure: The framework utilizes:
   • Scenario Training: Creating specific practice scenarios to teach skills and style variations.
   • Data Augmentation: To speed up and stabilize learning.
   • Special-Purpose Replay Buffers: Storing trajectories with specific style tags to facilitate learning diverse policies.
3. Style Composition: The system allows for the combination of multiple style requests (e.g., a specific weapon + a specific elemental effect) at runtime, even if the agent has rarely seen that specific combination during training.

Experimental Domains

The framework was validated across three distinct domains:

1. Gran Turismo 7 (GT7): A racing simulation. The agent (GT Sophy) was trained to balance lap times against resource constraints (fuel consumption and tire wear) and to exhibit driving behaviors like drifting.
2. Horizon Forbidden West (HFW): An open-world action-adventure game. The agent controls the protagonist, Aloy, in combat against complex robotic enemies. Styles included specific weapon choices (6 weapons), melee, traps, 7 elemental effects, and 3 part-removal strategies.
3. DeepMind Control Suite (DMC): A humanoid walking task. Styles controlled discrete arm poses and continuous gait lengths, demonstrating the approach's applicability to robotics.

Results

Gran Turismo 7

• Resource Management: By adjusting UVFA weights, the agent could trade off pace against fuel range or tire life. Increasing penalties for fuel/tire usage extended the time between pit stops by up to 60%, at the cost of 2–3 seconds per lap.
• Behavioral Variation: The agent successfully learned to drift and could modulate its apparent aggressiveness via threshold-based style parameters.

Horizon Forbidden West

• Style Adherence: In 57,000 battles across 19 enemies and 3 locations, the agent demonstrated a strong correlation between requested styles and executed behaviors (diagonal dominance in damage-type matrices).
• Contextual Adaptation: The agent did not blindly obey style requests but contextualized them. For example, when asked to use a specific elemental style, it would select a weapon capable of inducing that state, then potentially switch to a more powerful weapon to maximize damage while the state persisted.
• Composability: The agent successfully executed combined styles (e.g., specific weapon + specific elemental) despite seeing such specific combinations less than once per million training scenarios.
• Generalization: The agent maintained high performance against out-of-distribution (OOD) enemies and scenarios (e.g., flying enemies) not present in the training set.
• Trade-off Control: Pareto curves demonstrated that users could tune a style weight parameter at runtime to balance the "style score" against the win rate, allowing for flexible control over the agent's risk-taking behavior.

Humanoid (DMC)

• The agent successfully composed discrete arm poses with continuous gait control. The style parameter $\theta$ functioned effectively as a setpoint for gait length, even though arm poses impacted the robot's center of mass and balance.

Significance and Claims

The paper claims that Style-Conditioned UVFAs, when combined with appropriate learning algorithms and training strategies, can produce agents that succeed at complex real-time tasks while remaining modifiable on command.

• Gaming Impact: The technology offers immediate applications for the video game industry, including improved Quality Assurance (QA) processes, enhanced Non-Player Characters (NPCs), and novel accessibility features where agents can take over difficult sections of a game while adhering to a player's preferred playstyle.
• Robotics Impact: As general-purpose robots enter real-world environments, this approach provides a mechanism for users to shape robot behaviors via goal modifiers, enabling robots to perform tasks with varying degrees of "flair" or specific constraints (e.g., quiet cleaning vs. fast cleaning).
• Behavioral Encoding: Unlike standard RL agents that learn a single behavior, coachable agents encode a "family of behaviors," offering a flexible interface between human intent and machine execution.

The authors emphasize that the majority of the effort in this framework lies in creating the incentives and practice scenarios (the "coaching") rather than just the algorithmic architecture, suggesting that the design of training environments is critical for scaling to complex domains.