🤖 AI

Human-Aligned Procedural Level Generation Reinforcement Learning via Text-Level-Sketch Shared Representation

This paper introduces VIPCGRL, a novel deep reinforcement learning framework that enhances human-aligned procedural level generation by integrating text, level, and sketch modalities through a shared embedding space and contrastive learning to better interpret designer intent and produce controllable, human-like outputs.

Original authors: In-Chang Baek, Seoyoung Lee, Sung-Hyun Kim, Geumhwan Hwang, KyungJoong Kim

Published 2026-07-20

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: In-Chang Baek, Seoyoung Lee, Sung-Hyun Kim, Geumhwan Hwang, KyungJoong Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to be a game designer. You want it to build fun, playable levels, but you also want it to understand how you think. This is the world of Procedural Content Generation, where computers automatically create game worlds instead of humans drawing every single wall and monster. For a long time, these robots were like strict math students: they could follow a list of numbers (like "put 50 walls here") perfectly, but they couldn't understand a vague idea like "make it feel spooky" or "draw a path that looks like a snake." They also tended to build levels that felt cold and robotic, lacking the messy, creative flair of a human designer. The big question in this field is: How do we teach an AI not just to follow rules, but to think and create like a human partner?

This paper introduces a new system called VIPCGRL (Vision-Instruction PCGRL) that tries to solve this by giving the AI a "shared language" for different types of instructions. Think of it like teaching a robot to speak three dialects at once: Text (words), Levels (the actual game map), and Sketches (rough drawings). The researchers built a special "translator" that turns all three of these into the same secret code. This allows you to tell the AI to "make a narrow path" using words, or by drawing a quick scribble, or by showing it a finished map, and the AI understands them all as the same idea.

But the real magic isn't just understanding; it's about style. The authors noticed that even when AI follows instructions, the result often looks nothing like something a human would make. To fix this, they gave the AI a new kind of teacher's grade. Instead of just checking if the level is playable, the AI gets a bonus score if its creation looks and feels similar to levels made by real human designers. They tested this on a 2D game environment with 5,000 human-designed levels and 2,500 AI-generated ones. The results suggest that this new method makes the AI's levels much more "human-like" and easier to control with sketches or drawings, without losing the ability to follow specific rules. It's a step toward a future where you and an AI can co-create games together, with the AI understanding your rough sketches and creative vibes just as well as your typed commands.

The Problem: Robots Who Can't "Get" You

Imagine you are playing a video game, and you want to change the level. You might say, "I want a long, winding path with a few monsters hiding in the corners." A traditional game AI might hear "long path" and "monsters," but it might build a path that is technically long but feels boring and straight, or place monsters in spots that make no sense. It follows the math, but it misses the soul of the design.

Current tools for generating game levels (called PCGRL) are great at following strict numbers. If you say "make the map 16x16 tiles," they do it perfectly. But they struggle with two things:

Limited Input: They usually only understand numbers or very specific text. They can't really "see" a rough drawing you made on a napkin.
Robotic Style: Even when they follow the rules, the levels they build often look like they were made by a machine—too perfect, too symmetrical, or just lacking the creative "flair" of a human designer.

The researchers wanted to build an AI that acts more like a creative partner than a calculator. They wanted an AI that could take a text description, a rough sketch, or even a partial level map, and turn it into a new level that feels like it was designed by a human.

The Solution: A Universal Translator for Game Design

The team created VIPCGRL, which stands for Vision-Instruction Procedural Content Generation via Reinforcement Learning. The core idea is to teach the AI to speak three different languages at the same time: Text, Level Maps, and Sketches.

To do this, they built a "Shared Latent Space." Imagine a giant, invisible library where every idea is stored as a unique number code.

If you write "a narrow path," the AI turns that text into a code.
If you draw a narrow path on a piece of paper, the AI turns that sketch into a code.
If you show a map with a narrow path, the AI turns that map into a code.

The magic is that the AI is trained so that the code for "narrow path" is almost identical, no matter if it came from a word, a drawing, or a map. This is done using a technique called Quadruple Contrastive Learning. It's like a game of "find the matching pairs." The AI looks at a text instruction, a level map, a sketch, and the style of the creator (Human vs. AI). It learns to pull things that are similar closer together in its memory and push things that are different further apart.

For example, if you show the AI a text saying "narrow path" and a sketch of a narrow path, it learns that these two codes belong in the same neighborhood. But if you show it a sketch of a "wide open field," it learns that this code belongs in a completely different neighborhood. Crucially, it also learns to tell the difference between a "Human Style" code and an "AI Style" code, so it knows what to aim for.

The Secret Sauce: The "Human-Likeness" Reward

Just having a translator isn't enough; the AI still needs to know what to build. In standard training, an AI gets a reward (a point) only if it follows the rules (e.g., "did you make the path long enough?").

The researchers added a new rule: The Human-Likeness Reward.
Every time the AI builds a level, they compare it to a library of levels made by real humans. They use the shared code system to measure how "close" the AI's level is to a human's level.

If the AI builds a level that looks and feels like a human designed it, it gets a bonus point.
If it builds something that looks robotic, it gets fewer points.

This encourages the AI to not just follow the rules, but to copy the style of human designers. It's like a student who not only answers the math problem correctly but also writes the solution in the neat, creative handwriting the teacher loves.

What They Found: Better, More Human, and More Flexible

The team tested their system on a 2D game level generation task. They compared VIPCGRL to older methods that only used text or numbers.

1. It Sounds Like a Human (Human-Likeness):
When they measured how much the generated levels looked like human-made ones, VIPCGRL scored significantly higher.

In a test called TPKL-Div (which checks small, local patterns like how tiles are arranged), VIPCGRL was much closer to human designs than the old methods.
In a test called ViT-Sim (which checks the big-picture look), it also improved.
Most importantly, when real humans played the levels and rated them, they preferred the VIPCGRL levels. The humans rated the "Human-Likeness" of VIPCGRL at 4.25 out of 7, compared to 2.90 for the previous best method. The difference was statistically significant, meaning it wasn't just luck.

2. It Understands Sketches and Maps (Multi-Modal Control):
The AI was trained mostly on text instructions, but because of the shared language, it could also take sketches and level maps as input during the test.

Even though the AI had never been trained specifically on sketches, it could generate levels based on a drawing.
The controllability (how well it followed the instruction) was slightly lower for sketches than for text, but it was still much better than random guessing.
This proves that the "shared library" works: the AI can translate a drawing into the same code as a text instruction and build the right level.

3. The Trade-Off:
There was a tiny cost. As the researchers increased the "Human-Likeness" reward (making the AI try harder to look human), the AI became slightly less perfect at following the strict numerical rules (like hitting an exact number of monsters). However, the paper notes this trade-off was very small. The AI stayed very good at the task while becoming much more creative and human-like.

What This Means for the Future

The paper suggests that by teaching AI to understand multiple ways of giving instructions (text, drawings, maps) and by rewarding it for copying human style, we can create tools that are much better for game designers. Instead of just being a calculator that follows orders, the AI becomes a creative partner that understands your vibe.

The researchers admit there are limits. Their "sketches" were computer-generated from maps, not real hand-drawn scribbles from artists. They also note that the system is trained in two steps (first learning the language, then learning to build), which might make it a bit slower to adapt to new, weird situations. But the results show a clear path forward: if we want AI to help us create, we have to teach it to think and feel like a human, not just a robot.

Technical Summary: VIPCGRL – Human-Aligned Procedural Level Generation via Text-Level-Sketch Shared Representation

1. Problem Statement

Procedural Content Generation via Reinforcement Learning (PCGRL) aims to automate game level design, often serving as a tool for human designers in co-creative workflows. However, existing PCGRL systems face two primary limitations:

Limited Control Modalities: Current approaches rely heavily on numerical condition vectors or text-only inputs. They lack the ability to accept visual inputs (such as level maps or sketches), which are crucial for expressing spatial intent and structural constraints intuitively.
Lack of Human Alignment: While existing methods can satisfy specific task conditions (e.g., "create a map with 5 regions"), the generated levels often fail to capture the structural and stylistic nuances of human-authored content. The generated outputs may be functionally correct but stylistically distinct from human designs, limiting their utility in collaborative design scenarios where "human-likeness" is a key requirement.

The paper identifies a gap in current frameworks: they optimize for task-specific objectives but lack mechanisms to align generated content with human design styles or to integrate diverse input modalities (text, vision, sketches) into a unified control space.

2. Methodology: VIPCGRL

The authors propose VIPCGRL (Vision-Instruction PCGRL), a deep reinforcement learning framework designed to generate 2D game levels that are both controllable and human-aligned. The framework operates through a two-step training process involving a shared latent representation and a similarity-based reward.

A. Dataset Construction

To train the system, the authors constructed a multi-modal dataset consisting of triples: (Instruction, Level, Sketch).

Human Data ( $D_{human}$ ): 5,000 samples created by four game designers, pairing natural language instructions with 16x16 level maps. These levels were transformed into "sketch-style" images using edge detection, spline interpolation, and Gaussian blurring.
AI Data ( $D_{AI}$ ): 2,500 samples generated by a pre-trained IPCGRL model. These were paired with modified instructions (e.g., prefixed with "AI-style") to explicitly distinguish between human and AI design patterns.
Spatial Context: Both level and sketch inputs were augmented with x-y coordinate channels (inspired by CoordConv) to provide explicit spatial context.

B. Shared Latent Representation (Quadruple Contrastive Learning)

A core component is a shared embedding space learned via quadruple contrastive learning. This method aligns representations across three modalities (Text, Level, Sketch) while simultaneously distinguishing between two styles (Human vs. AI).

Encoders: Three encoders map inputs to 64-dimensional vectors:
- $f_\psi$ : Text encoder (pre-trained CLIP + trainable MLP).
- $f_\phi$ : Level encoder (Residual CNN + MLP).
- $f_\omega$ : Sketch encoder (Residual CNN + MLP).
Loss Function: The system employs a multi-positive InfoNCE objective. It pulls together embeddings that share the same task condition and style (e.g., a text instruction, its corresponding human level, and its sketch) while pushing apart embeddings that differ in either condition or style.
Result: This creates a unified space where different modalities describing the same design intent are clustered, and human/AI styles are separable.

C. Conditional Reinforcement Learning with Human-Alignment Reward

The DRL agent is trained using Proximal Policy Optimization (PPO) with a modified reward function that combines task completion with style alignment.

Control Reward ( $r_{ctrl}$ ): Standard PCGRL reward measuring the improvement toward a specific goal (e.g., reducing the difference between current monster count and target count).
Similarity Reward ( $r_{sim}$ ): An auxiliary reward derived from the shared latent space. At each step, the agent compares its current level state ( $s_t$ ) against a reference human-designed level ( $s_{human}$ ) conditioned on the same task. The reward is the relative increase in cosine similarity between the agent's state and the human reference in the embedding space.
Total Reward: $r_t = r_{ctrl} + \alpha \cdot \lambda_{sim} \cdot r_{sim}$ , where $\lambda_{sim}$ balances the trade-off between task performance and human-likeness.

3. Key Contributions

Human-Aligned DRL Framework: A novel PCGRL framework that explicitly optimizes for human-likeness via a similarity-based alignment reward, moving beyond simple condition satisfaction.
Multi-Modal Shared Representation: The introduction of a unified latent space integrating text, level maps, and sketches, enabling consistent conditional control across modalities.
Quadruple Contrastive Learning: A learning objective that jointly aligns multiple modalities and captures human-AI stylistic differences within a single embedding space.
Empirical Validation: Demonstration that the approach improves human-likeness while maintaining controllability, validated by both quantitative metrics and human evaluations.

4. Experimental Results

The authors evaluated VIPCGRL on a 2D level generation task with five distinct structural constraints.

Quantitative Metrics:
- Human-Likeness: VIPCGRL significantly outperformed baselines (CPCGRL, IPCGRL) in TPKL-Div (Tile Pattern KL-Divergence) and ViT-Sim (Vision Transformer Similarity). Increasing the similarity reward coefficient ( $\lambda_{sim}$ ) further enhanced human-likeness with only marginal trade-offs in controllability (Progress metric).
- Multi-Modal Generalization: An agent trained solely on text inputs successfully generalized to unseen Level and Sketch inputs during inference, achieving human-likeness scores comparable to text-conditioned generation.
Human Evaluation:
- In a user study with 26 participants, VIPCGRL received significantly higher scores for Human-Likeness and Text-Alignment compared to baselines.
- The model also demonstrated strong alignment with sketch inputs, despite sketches not being used during training, confirming the effectiveness of the shared representation.
Ablation Studies: Reducing the data volume for any single modality (Text, Level, or Sketch) negatively impacted performance across other modalities, confirming that the modalities contribute complementarily to the shared embedding space. Text data reduction had the most pronounced effect, suggesting text acts as a primary anchor for alignment.

5. Significance and Claims

The paper claims that VIPCGRL addresses the fundamental gap in PCGRL where generated content often lacks the stylistic fidelity of human designers. By introducing a shared representation and a similarity-based reward, the framework enables:

Co-creativity: AI agents that can interpret diverse human intents (via text or sketches) and generate content that aligns with human design patterns.
Controlled Stylistic Alignment: The ability to tune the degree of "human-likeness" via the $\lambda_{sim}$ parameter without completely sacrificing task performance.
Modality Agnosticism: The capability to switch between input modalities (text, level, sketch) at inference time without retraining the policy.

The authors modestly note limitations, specifically the two-step training process (frozen encoders during DRL) which may limit adaptability to intermediate states, and the use of procedurally generated sketches rather than real-world designer sketches. They suggest future work could explore end-to-end training and broader sketch styles.