PCGRLLM: Large Language Model-Driven Reward Design for Procedural Content Generation Reinforcement Learning

Technical Summary: PCGRLLM

Problem Statement
The design of reward functions is a critical bottleneck in training Deep Reinforcement Learning (DRL) agents for Procedural Content Generation (PCG). Traditionally, this process relies heavily on human expertise, time-consuming reward shaping, and pre-defined environmental features, creating barriers to scalability and accessibility. While recent work has explored using Large Language Models (LLMs) to generate reward functions (e.g., ChatPCG), existing approaches often lack a refinement process that incorporates the actual outcomes of trained agents. Consequently, it remains uncertain whether the generated policies accurately reflect the intended reward conditions, limiting the ability of LLMs to effectively bridge high-level narrative instructions with low-level reward code.

Methodology
The paper proposes PCGRLLM, an extended architecture designed to automate reward generation for PCG via a feedback-driven loop. The framework operates through three primary sequential steps:

1. Reward Refinement: The LLM iteratively refines a reward function based on textual instructions (stories) and feedback from previous iterations. The process distinguishes between an initial generation phase (using a template) and subsequent refinement phases where the LLM analyzes a parent reward function and its associated feedback.
2. Self-Alignment: Before training the agent, the generated reward function undergoes a "few-shot" interaction with the environment using a random agent. This step calculates the mean and variance of reward values to ensure the function produces trainable signals within an appropriate range, adjusting weights or formulas to prevent issues like overly narrow ranges or excessive sparsity.
3. Feedback from Generated Content: A DRL agent (trained via Proximal Policy Optimization) generates content (game levels) based on the aligned reward function. The LLM then analyzes the resulting content against the original instructions to identify discrepancies (e.g., missing enemies or incorrect spatial layouts). This analysis generates specific feedback to guide the next iteration of reward refinement.

To enhance the exploration of the reward space, the framework integrates three reasoning-based Prompt Engineering (PE) techniques:

• Chain-of-Thought (CoT): A linear, step-by-step reasoning approach.
• Tree-of-Thoughts (ToT): Explores multiple reasoning directions with backtracking capabilities, selecting parent nodes based on fitness scores.
• Graph-of-Thoughts (GoT): An extension of ToT that utilizes auxiliary information (top-performing reward nodes) to improve sample efficiency and reasoning depth.

The system was evaluated on a text-to-reward task in a 2D dungeon environment, where agents were trained to generate levels satisfying specific narrative constraints (e.g., "Player must encounter a Bat to get a Key").

Key Contributions
The authors identify three main contributions:

1. Enhanced Reward Generation Architecture: The introduction of a feedback mechanism that allows the policy to align more effectively with instructions by observing actual agent outcomes, addressing the limitations of prior self-alignment-only methods.
2. Reasoning-Based Refinement Prompting: The application of state-of-the-art PE techniques (ToT, GoT) to improve the exploration of the reward space, moving beyond simple linear generation.
3. Comprehensive Modular Evaluation: Extensive experiments, including ablation studies, that provide insights into the capabilities of LLMs in content generation, specifically regarding code generation, content evaluation, and causal reasoning.

Experimental Results
The framework was evaluated across four scenarios involving varying conditional objectives and spatial constraints, using OpenAI's gpt-4o as the backend.

• Feedback Effectiveness: The inclusion of the feedback mechanism significantly improved performance compared to zero-shot generation and self-alignment-only baselines. In the Graph-of-Thought setting, the full PCGRLLM framework achieved the highest scores, demonstrating that feedback is crucial for correcting shortcomings in imperfect reward functions.
• Feedback Quality: Experiments comparing "No," "Generic," and "Specific" feedback revealed that specific, content-tailored feedback is essential for sustained performance improvement. Generic feedback yielded minimal gains, while the absence of feedback led to performance stagnation or decline.
• Prompt Engineering: While no statistically significant differences were found across all metrics, the Graph-of-Thought (GoT) method showed a descriptive trend of outperforming CoT and ToT in later iterations. GoT's backtracking mechanism allowed it to recover from performance drops caused by misguided exploration, ultimately achieving the highest final scores.
• Human Comparison: The results showed a complementary pattern: human-designed rewards performed better on tasks with multiple conditional objectives, whereas PCGRLLM excelled in tasks requiring spatial reasoning under constraints.
• Self-Evaluation Limitations: The study found that LLM-based self-evaluation of fitness (without heuristic oracles) often led to performance degradation. The LLM's tendency to hallucinate or lack objectivity in scoring generated content misguided the search direction, highlighting a current limitation in autonomous evaluation.

Significance and Claims
The paper claims that PCGRLLM demonstrates the potential to reduce human dependency in game AI development while supporting and enhancing creative processes. By framing reward generation as an iterative refinement loop involving self-alignment and feedback, the system achieves performance comparable to humans in specific spatial reasoning tasks. The authors emphasize that using LLMs as designers of reward functions (rather than direct content generators) offers explicit, composable control over the learning process, ensuring constraints are met and trade-offs are transparent.

However, the authors remain modest regarding the current state of autonomous evaluation. They acknowledge that the reliance on LLMs for self-evaluation of content quality currently introduces noise and hallucination, which can hinder the optimization process. Future work is suggested to focus on improving the objectivity of these self-evaluation signals through comparative reference points and self-consistency reasoning.