GameUIAgent: An LLM-Powered Framework for Automated Game UI Design with Structured Intermediate Representation

This paper introduces GameUIAgent, an LLM-powered framework that automates game UI design by converting natural language descriptions into editable Figma files via a structured JSON intermediate representation and a neuro-symbolic pipeline with self-correction, while establishing key empirical principles like the Quality Ceiling Effect and Rendering-Evaluation Fidelity Principle through extensive evaluation.

The Gist

Imagine you are the director of a massive video game. You have a brilliant idea: "I want a rare, glowing sword card that looks like it's made of fire, with a golden border and a shiny star."

In the old days, a human artist would spend hours drawing this, then another artist would draw a slightly less rare version, and another would make a common version. It's slow, expensive, and hard to keep consistent.

GameUIAgent is like hiring a super-smart, tireless robot assistant who can take your text description and instantly draw these cards for you. But here's the catch: if you just ask a robot to "draw a sword," it might draw a stick figure or forget the fire. This paper introduces a system that fixes those mistakes automatically.

Here is how it works, broken down into simple analogies:

1. The Blueprint (The "Design Spec JSON")

Instead of asking the robot to just "make a picture," the system forces the robot to write a detailed blueprint first (called a JSON file).

• The Analogy: Think of this like an architect drawing a floor plan before a builder starts laying bricks. The robot says, "I will put a red rectangle here, a blue circle there, and a text box saying 'Fire Sword'."
• Why it helps: If the robot makes a mistake, we can fix the blueprint without having to redraw the whole picture from scratch.

2. The Three-Step Assembly Line

The system doesn't just generate the blueprint and stop. It runs it through a strict three-step factory line:

• Step A: The Creative Writer (LLM): The robot writes the initial blueprint based on your text.
• Step B: The Strict Editor (Post-Processing): This is a computer program that checks the math. Did the robot say the sword is 500 inches wide? The editor shrinks it to fit. Did it forget to add the "Rare" star? The editor adds it automatically. It's like a spell-checker that also fixes your grammar and adds missing punctuation.
• Step C: The Art Critic (VLM): A different AI (a "Vision-Language Model") looks at the finished card and gives it a grade from 1 to 10. It checks: "Is the text readable? Do the colors match? Does it look cool?"

3. The "Reflection Controller" (The Self-Correcting Loop)

This is the magic part. If the Art Critic gives the card a low score (say, a 4/10), the system doesn't just give up.

• The Analogy: Imagine a student taking a test. If they get a question wrong, a normal student might just move on. But GameUIAgent is like a student who says, "Wait, I got the math wrong. Let me fix just the math part, keep the rest, and try again."
• The system takes the Critic's feedback ("The text is too small") and sends it back to the Creative Writer to fix only that specific problem. It repeats this loop until the score is high enough.
• Safety Net: The system keeps the "best version" it has ever seen. Even if the robot tries to fix a mistake and accidentally makes it worse, the system rolls back to the previous good version. It guarantees the design never gets worse.

4. The Two Big Surprises (What the Researchers Found)

The paper discovered two very important rules about how AI art works, which are like "laws of physics" for game design:

A. The "Quality Ceiling" (The Tired Critic)

• The Finding: If the robot starts with a really bad design, the self-correction loop can fix it easily. But if the design is already almost perfect, the robot can't really improve it much more.
• The Analogy: Imagine you are trying to clean a dirty window. If the window is covered in mud, a little scrubbing makes a huge difference. But if the window is already 99% clean, scrubbing harder won't make it sparkle any more; you've hit the "ceiling." The researchers found that the critic's ability to see tiny flaws is the limit, not the robot's ability to draw.

B. The "Rendering Trap" (The Paradox of Shiny Things)

• The Finding: Sometimes, making the picture look more realistic (adding shadows and gradients) actually makes the AI Critic hate it more.
• The Analogy: Imagine a house with a crooked wall. If the wall is painted flat white, the crook is hard to see. But if you add fancy, shiny wallpaper and dramatic lighting, the crooked wall becomes glaringly obvious!
• The Lesson: You can't just add "pretty" effects to a broken design. You have to fix the structure (the layout) before you add the fancy lighting, or the AI will just see the flaws more clearly and give you a lower score.

Summary

GameUIAgent is a tool that turns text into game art by:

1. Writing a strict blueprint.
2. Fixing the math and adding game rules automatically.
3. Having an AI judge grade it.
4. Letting the robot re-do the work only on the parts the judge didn't like, over and over, until it's perfect.

It solves the problem of making hundreds of game items (like swords, potions, and character cards) that all look consistent, even when they have different levels of rarity (Common vs. Legendary), without needing a human artist to draw every single one.

Technical Summary

1. Problem Statement

Game User Interface (UI) design is a labor-intensive, manual process requiring consistent visual assets across hierarchical rarity tiers (e.g., Common, Rare, SSR, UR) in games like gacha titles. Current generative AI solutions face three main limitations:

• Lack of Editability: Most tools generate static raster images rather than editable vector designs (e.g., Figma files).
• Domain Specificity: Existing methods target standard web interfaces and fail to adhere to game-specific rules, such as rarity hierarchies and thematic coherence.
• Quality Verification: Single-shot generation lacks iterative self-correction, and LLMs often fail to self-correct when acting as their own judges, leading to "visual emptiness" or structural degradation in complex designs.

2. Methodology: GameUIAgent Framework

The authors propose GameUIAgent, a neuro-symbolic framework that bridges natural language descriptions and professional design tools via a six-stage pipeline.

A. Core Architecture

The system decouples creative generation from deterministic rendering using a Design Spec JSON as a structured intermediate representation (IR).

1. Prompt Engineering: Centralized prompts define templates (Character Card, Item Thumbnail, Skill Panel) and design principles.
2. LLM Generation: An LLM generates the Design Spec JSON, defining a recursive node tree (Frames, Rectangles, Ellipses, Text) with geometry, styles, and hierarchy.
3. Intelligent Post-Processing: A deterministic module repairs the JSON (normalizing colors, clamping dimensions), injects data (calculating stat bar widths), and enhances rarity (adding tier-specific visual decorators like glow, borders, and star badges).
4. Rendering: The JSON is rasterized via a Figma plugin or a lightweight Python previewer.
5. VLM Quality Review: An independent Vision-Language Model (VLM), specifically GPT-4o, evaluates the rendered design across five dimensions: Layout, Consistency, Readability, Completeness, and Aesthetics (scored 1–10).
6. Reflection Controller (RC): An agentic loop that uses the VLM's scores to generate targeted repair prompts. It employs Best-Result Tracking to ensure the final output is never worse than the initial generation (non-regressive).

B. Key Technical Components

• Design Spec JSON: A schema-aligned, tool-agnostic format that supports gradients, shadows, and auto-layout constraints, acting as the "source of truth" between generation and rendering.
• Rarity Progression System: A post-processing rule set that automatically scales visual complexity (e.g., from simple borders for 'N' tier to multi-layered golden frames for 'UR' tier) based on the input rarity.
• VLM-as-Judge: The system uses an external VLM critic rather than the generator LLM for feedback, mitigating self-correction failure modes.

3. Key Contributions

1. End-to-End Pipeline: A novel framework converting natural language to editable Figma game UIs with guaranteed non-regressive quality improvement.
2. Failure Taxonomy: Identification of two task-structural failure modes in LLM-based UI generation:
   • Rarity-Dependent Degradation: Models struggle with complexity; some degrade in validity as rarity increases (complexity overload), while others over-generate on simple inputs.
   • Visual Emptiness: Syntactically valid JSON that renders as blank frames due to zero-area nodes or indistinguishable colors.
   • Finding: JSON validity is necessary but insufficient for design quality.
3. Empirical Principles:
   • Quality Ceiling Effect: Iterative self-correction gains are bounded by the "headroom" below a quality threshold. As initial quality approaches the evaluator's distinguishability limit, further refinement yields diminishing returns ($r = -0.96$).
   • Rendering-Evaluation Fidelity Principle: Partial rendering enhancements (e.g., adding gradients without fixing layout) can degrade evaluation scores by amplifying structural defects (like overlaps) that were previously hidden by flat colors.

4. Experimental Results

The system was evaluated on 110 test cases across three LLMs (DeepSeek V3, Gemini 2.0 Flash, GPT-4o-mini) and three UI templates.

• Cross-Model Analysis:

  • DeepSeek V3 outperformed others with 98% JSON validity and an average VLM score of 8.0/10.
  • Gemini and GPT-4o-mini showed significant validity gaps (88% and 56%, respectively) and lower perceptual scores, confirming that structural reliability is the primary differentiator.
  • Text Contrast Ratio was identified as the strongest predictor of VLM quality ($r=0.51$).

• Ablation Studies:

  • Structured Prompts: Removing the schema caused quality to collapse ($\Delta = -3.1$), proving schema/domain rules are more critical than raw model capability.
  • Post-Processing: Disabling it reduced quality by 1.4 points, primarily due to readability issues.
  • Few-Shot Scaffolding: Removing exemplars reduced Node Count by 38% and Color Diversity by 42%, revealing a "perceptual blind spot" where VLMs fail to detect compositional richness.

• Reflection Controller (RC) Performance:

  • Achieved a mean improvement of +0.96 points with 100% non-regressive outcomes.
  • Quality Ceiling: A strong negative correlation ($r = -0.96$) was found between initial quality and improvement potential. Designs starting above a threshold ($\theta=7.5$) saw negligible gains.
  • Rendering Impact: Adding gradients without layout correction dropped scores by 1.0 point (amplifying overlaps), while "Layout-aware" rendering recovered and surpassed the baseline by +2.52 points.

5. Significance and Implications

• Production Viability: GameUIAgent provides a practical solution for automating game asset creation, ensuring consistency across rarity tiers while maintaining editability in professional tools (Figma).
• Theoretical Advances:
  • The Quality Ceiling Effect suggests that in visual agentic systems, evaluator capability (headroom) is the limiting factor for scaling, not just generator capacity. This parallels test-time compute scaling laws.
  • The Rendering-Evaluation Fidelity Principle warns that improving visual fidelity without structural correctness can invert reward signals, a critical insight for training RL agents in visual domains.
• Future Directions: The framework establishes a foundation for LLM-driven visual agents, highlighting the need for synchronized improvements in generation, rendering, and evaluation mechanisms. The authors note that future work requires human expert validation to correlate VLM scores with professional design judgment.