Technical Summary: Mage – Multi-Axis Evaluation of LLM-Generated Executable Game Scenes

Problem Statement

Current evaluation standards for Large Language Model (LLM) code generation rely heavily on compile-pass rates and unit test success (e.g., pass@k on standalone functions). The authors argue that for multi-component, domain-specific artifacts—specifically executable game scenes in Unity—this metric is actively misleading. A generated script may compile successfully while failing to instantiate a functional scene, rendering visible content without triggering win conditions, or implementing logic that never executes. Existing benchmarks fail to capture the divergence between syntactic correctness (compilation) and functional correctness (domain fidelity). Furthermore, prior attempts at semantic-level Intermediate Representation (IR) conditioning resulted in 0% compilation rates, leaving open the question of what level of structural grounding is sufficient for executable code generation.

Methodology

The paper introduces Mage, a benchmark and evaluation protocol designed to assess LLM-generated Unity 2022 game scenes across four distinct axes.

1. Benchmark Construction

• Dataset: 26 hand-crafted Unity 2D mini-games derived from the Goal Playable Concept (GPC) corpus, covering diverse goal patterns (e.g., resource collection, stealth, competitive play).
• Ground Truth: Each pattern is realized as a complete, flattened Unity scene (prefabs expanded to GameObjects) with a lossless JSON representation of the scene graph.
• IR Extraction: An automated pipeline extracts Intermediate Representations (IRs) at two granularity levels:
  • Behavior-only: Script logic, method bodies, inspector field values, physics configurations, and win/lose condition chains.
  • Full-scene: Behavior-only data plus passive physics objects, camera configuration, boundaries, and rendering components for all GameObjects.
• Models: Four open-weight LLMs (7B–30B active parameters): Qwen3-Coder-30B, Gemma-4-26B, DeepSeek-Coder-V2-Lite, and Qwen2.5-Coder-7B.
• Conditions: Three prompting strategies were tested across 858 generation attempts (3 seeds per pattern):
  1. No-schema (Direct NL-to-C#): Natural language pattern description only.
  2. Behavior-only IR: Ground-truth derived JSON containing script logic.
  3. Full-scene IR: Ground-truth derived JSON containing full scene structure.
• Control: All conditions shared identical API mapping rules and prompt templates to isolate the effect of IR conditioning.

2. Evaluation Protocol (Mage)

The protocol evaluates generation quality along four axes:

1. Compile Success: Whether the generated C# file loads in Unity without errors (excluding stale reference warnings).
2. Runtime Success: Whether the compiled scene executes GameBuilder.Awake() and exits cleanly without crashing.
3. Structural Fidelity: A multi-dimensional F1 score comparing the generated scene's GameObjects, components, scripts, and tags against the ground truth.
4. Mechanism Adherence: A static analysis metric measuring the correctness of the win/lose path (steps, effects, and conditions) via set F1 scores.

Key Results

Compile-Correctness Divergence

The study reveals a systematic anti-correlation between compile-pass rates and functional correctness:

• Direct NL-to-C# (No-schema): Achieved the highest mean runtime-pass rate (43%), with the strongest model (Qwen3) reaching 80%. However, these scenes were structurally and behaviorally vacuous. The mechanism adherence F1 score collapsed to ~0.12 (a baseline floor), and tag F1 was 0.00 across all models. The models produced small, locally consistent programs that satisfied the parser but lacked the target pattern's structure.
• IR Conditioning: Conditioning on IR (Behavior-only or Full-scene) significantly reduced runtime-pass rates (Mean: 14% and 21% respectively) but drastically improved functional correctness. Mechanism adherence F1 rose to 0.82–1.00 for the strongest models, and structural fidelity improved across all dimensions.

Granularity Saturation

Within the IR-conditioned settings, there was no statistical difference between Behavior-only and Full-scene granularity. McNemar's exact tests yielded p = 1.0 across all four models, indicating that for this benchmark, behavior-level information is sufficient to saturate the input-level conditioning; adding passive scene geometry (Full-scene) did not yield further gains in functional correctness.

Error Analysis

Compile failures were dominated by semantic errors (e.g., inaccessible members, type resolution, scope conflicts) rather than syntax errors.

• No-schema: Errors were rare; models avoided complex structures.
• IR Conditioning: Errors shifted to specific semantic failures introduced by the IR. For instance, CS0246 (type/namespace not found) appeared exclusively under IR conditions (469 occurrences), indicating failures in mapping IR entity names to Unity APIs. CS0122 (inaccessible member) spiked in Behavior-only IR (862 occurrences) due to models misapplying access modifiers exposed in the IR.

Contributions

1. Evaluation Methodology: A four-axis protocol (Compile, Runtime, Structural Fidelity, Mechanism Adherence) that exposes the systematic divergence between compile rates and functional correctness in domain-specific code generation.
2. Benchmark and Dataset: A release of 26 Unity goal patterns, two automatically extracted IR granularity levels, 858 generation outputs, and full replay logs for independent verification.
3. Empirical Characterization: A controlled study demonstrating that:
   • Compile rates are anti-correlated with functional correctness in this domain.
   • Structural IR conditioning recovers domain-faithful structure at the cost of runtime success rates.
   • Input-level granularity saturation occurs between behavior-only and full-scene IRs.
4. Three-Factor Decomposition: A framing lens (Domain Completeness, API-Mapping Adequacy, LLM Execution Fidelity) to interpret why structural grounding fails to produce correct code, attributing failures to semantic resolution and API translation rather than syntax.

Significance and Claims

The paper claims that compile-pass rate is an inadequate and often misleading signal for evaluating LLMs on multi-component, domain-specific artifacts. The authors argue that relying solely on compilation metrics leads to the false conclusion that models are performing well when they are actually generating "structurally vacuous" code.

The significance of the work lies in establishing that multi-axis evaluation is necessary to detect the divergence between syntactic validity and functional correctness. The authors posit that for tasks involving heavy domain-specific frameworks (like game engines), benchmarks must report at least three axes: parser-level pass, runtime-level pass, and domain-fidelity pass. The study suggests that while structural IR conditioning improves functional correctness, it introduces new failure modes related to API mapping and type resolution, highlighting the need for better translation mechanisms between IRs and domain APIs.

The paper concludes modestly, noting limitations such as the single-engine scope (Unity), the lack of human playability measurement (only runtime execution was verified), and the interpretive nature of the three-factor decomposition, which requires further causal investigation.