Technical Summary: Knowledge-Conditioned, Single-Pass LLM Synthesis of Executable Unity Game Scenes

Problem Statement

Current demonstrations of Large Language Models (LLMs) generating Unity C# game scripts rely heavily on iterative repair loops, where models regenerate code until it compiles. This workflow conflates the model's intrinsic parametric knowledge with the corrective power of the repair loop or human intervention. Furthermore, existing evaluations often use unstable genre labels or free-form natural language, which lack the reproducibility required for cumulative research.

This paper addresses the question: What is the intrinsic capability ceiling of single-pass LLM generation of executable game artifacts when no human feedback or iterative repair is allowed? The study isolates the model's stored knowledge from practitioner domain expertise, treating the compiler as a strict, objective gate. If a script does not compile on the first pass, it is a failure attributable solely to the model.

Methodology

Experimental Design

The study evaluates the generation of Goal Playable Concepts (GPCs)—playable counterparts of 26 specific gameplay design patterns defined by Björk and Holopainen. These patterns serve as a theoretically grounded, reproducible vocabulary for game design, replacing unstable genre labels.

• Scale: 10,400 generation records.
• Models: Four open-weight models from different families (7B-Qwen2.5, 16B-DeepSeek, 22B-Codestral, 30B-Qwen3).
• Generation Modes:
  • Editor-style: Scripts that build scenes at edit time using UnityEditor APIs.
  • Runtime-builder: Scripts that construct scenes programmatically at play time using MonoBehaviour (tested only on the 30B model).
• Conditioning Levels: Four levels of Intermediate Representation (IR) conditioning:
  • No Schema: Direct generation from pattern description.
  • With Schema (Free/Min/Full): Generation of a JSON IR first, followed by C# code generation. The schema strictness varies from top-level field names to full field definitions and constraints.
• Procedure: Each record is a single generation pass. The generated C# script is compiled within a pre-built Unity project. No compiler errors are fed back to the model.

Failure Taxonomy Construction

The authors categorize the 99 distinct C# compiler error codes observed across 90,673 error occurrences into two mutually exclusive classes:

1. Grounding Errors (18 codes): Domain-specific failures requiring Unity engine knowledge to resolve. These involve invented or misused Unity types, namespaces, and APIs (e.g., CS0246: type or namespace not found; CS1061: no such member).
2. Hygiene Errors (81 codes): Domain-independent structural defects diagnosable without Unity knowledge. These include syntactic errors like missing semicolons, unmatched braces, and undeclared variables (e.g., CS1003, CS1002).

Key Results

1. Universal Compilation Failure

No generated C# script compiled into a runnable scene across any of the 10,400 records. This uniform failure eliminates survivorship bias, allowing for an exhaustive census of failure modes.

2. Error Composition and Model Behavior

• Grounding vs. Hygiene Distribution: The split between Grounding and Hygiene errors varies significantly by goal pattern and model configuration.
  • High Grounding Patterns: Concepts coupled to Unity's perception and physics APIs (e.g., Stealth, Rescue, Exploration, Survive) predominantly fail with Grounding errors. The model invents non-existent engine components (e.g., GuardAI, Pathfinding) to satisfy the design semantics.
  • High Hygiene Patterns: Concepts reducible to general state manipulation (e.g., Capture, Collection) predominantly fail with Hygiene errors. The generated scripts collapse structurally (syntax errors) before reaching the layer where Unity knowledge is tested.
• Model Scale and Identity: Increasing model size (from 7B to 30B) does not yield a runnable scene. Instead, it shifts the composition of failure. Larger models (like 30B-Qwen3-Ed) reach the compiler more often and fail at the Grounding layer (missing API knowledge), whereas smaller models often fail at the format gate (sanitizer rejection) or the Hygiene layer (syntax).
• IR Conditioning Effects: Stricter schema conditioning does not resolve failures but relocates them.
  • For smaller models (7B, 16B), strict schemas increase sanitizer rejection rates (up to 89%), preventing code from reaching the compiler.
  • For larger models (22B, 30B-Ed), stricter schemas improve format compliance, allowing more records to reach the compiler. However, this shifts the error profile toward Grounding errors, as the model is forced to produce syntactically valid code that still lacks correct engine references.
• Generation Mode: Switching from Editor-style to Runtime-builder mode (tested on 30B) redistributes errors but does not eliminate Grounding failures. In some cases (e.g., Collection), the Runtime mode introduced severe Hygiene spikes without improving compilation success.

3. Per-Pattern Error Profiles

The study establishes an empirical ordering of the 26 goal patterns based on the engine knowledge their instantiation demands.

• Stealth (Grounding share ~0.98 in some configurations) fails primarily because the model invents engine-specific types for vision and detection that do not exist.
• Capture (Grounding share ~0.0 in some configurations) fails primarily due to structural syntax errors, as the logic can be expressed in plain C# without specific engine APIs.

Significance and Claims

The paper claims to provide the first systematic census of LLM failure modes in engine-coupled code generation at the granularity of individual compiler errors. Its primary contributions are:

1. A Grounding/Hygiene Taxonomy: A formal distinction between domain-specific (engine knowledge) and domain-independent (structural) errors, derived from 90,673 observed compiler diagnostics.
2. An Error-Driven Lens on Design Complexity: The study demonstrates that the semantic complexity of a gameplay concept can be empirically ordered by the type of error it elicits. Patterns requiring specific engine APIs (perception, physics) expose the model's lack of parametric engine knowledge, while patterns relying on general logic expose structural generation weaknesses.
3. The Single-Pass Ceiling: The findings establish that, within the 7B–30B open-weight tier, the bottleneck for single-pass generation is missing engine-specific knowledge, not model scale, generation mode, or structural conditioning.
4. Practical Diagnostic for Designers: The error profiles offer a concrete guide for game designers and developers. They indicate which gameplay concepts (e.g., Stealth) are currently beyond the reach of unaided single-pass LLM generation and require external engine knowledge support (e.g., retrieval-augmented generation or fine-tuning), versus which concepts (e.g., Capture) might be amenable to syntactic post-processing or grammar-guided decoding.

The authors conclude that while iterative repair loops are productive in practice, they mask the intrinsic limitations of current LLMs. The single-pass evaluation reveals that without external engine knowledge injection, current models cannot synthesize executable Unity scenes, regardless of the complexity of the design pattern or the size of the model.