BLOCK: An Open-Source Bi-Stage MLLM Character-to-Skin Pipeline for Minecraft

This paper introduces BLOCK, an open-source bi-stage pipeline that leverages a large multimodal model to generate consistent 3D character previews and a fine-tuned FLUX.2 model with a novel EvolveLoRA curriculum to decode these previews into pixel-perfect Minecraft skins.

The Gist

Imagine you want to turn a photo of your favorite celebrity, a drawing of a fantasy hero, or even a picture of your cat into a Minecraft skin. You want the result to look exactly like the character, but with that blocky, pixelated style that fits perfectly into the game's 3D world.

The problem? It's surprisingly hard. If you just ask a super-smart AI to "make a skin," it usually fails. It might put the face on the back of the head, swap the left arm with the right leg, or create a texture that looks great as a picture but breaks the game's rules.

Enter BLOCK, a new open-source tool that solves this by acting like a two-step assembly line instead of trying to do everything at once.

Here is how BLOCK works, explained with some everyday analogies:

The Problem: The "One-Step" Trap

Think of trying to make a Minecraft skin in one step like asking a master chef to cook a meal, wrap it in a specific box, and mail it to a customer all in one single motion.

• The chef (the AI) is great at cooking (understanding the character).
• But they are terrible at wrapping (following the strict Minecraft grid rules).
• The result? A delicious meal that spills out of the box because the chef didn't know how to fold the paper.

The Solution: The BLOCK Pipeline

BLOCK splits the job into two specialized workers, each doing what they are best at.

Stage 1: The "Translator" (The MLLM)

The Job: Turn a messy, real-world photo into a clean, standardized "blueprint."
The Analogy: Imagine you have a photo of a person in a weird pose, wearing a hat, with a messy background. You need to turn this into a flat, two-panel drawing (front view and back view) that looks like a Minecraft character sheet.

• How it works: BLOCK uses a powerful AI (like a super-intelligent translator) to look at your photo and say, "Okay, I see a guy in a red shirt. I will draw him standing straight, facing forward on the left, and facing backward on the right, with a clean white background."
• Why it's needed: This removes all the "noise" (weird angles, shadows, backgrounds) and gives the next step a perfect, standardized instruction sheet.

Stage 2: The "Architect" (The FLUX.2 Model)

The Job: Turn that clean blueprint into the actual 64x64 pixel skin file.
The Analogy: Now that you have the perfect blueprint, you need a master mason to build the wall. But this mason has a very strict rule: The bricks must fit a specific grid.

• The Challenge: Minecraft skins are tiny (64x64 pixels). If the blueprint has too many tiny details, they get lost when squished into that tiny grid.
• The Innovation (EvolveLoRA): This is the secret sauce. Instead of teaching the mason to build the wall from scratch, BLOCK teaches them in three easy steps:
  1. Step 1 (Text-to-Image): Teach the mason what a "red shirt" or "blue pants" looks like in general.
  2. Step 2 (Image-to-Image): Show the mason a picture of a character and ask them to draw the skin.
  3. Step 3 (Preview-to-Skin): Finally, show the mason the "blueprint" from Stage 1 and ask for the final skin.
  • Why this matters: By building on the previous lessons, the AI doesn't get confused. It learns the basics first, then the details, then the final strict rules. This is called EvolveLoRA (Evolving Low-Rank Adaptation).

The Final Step: The "Sieve"

Once the Architect builds the 512x512 pixel skin, BLOCK runs it through a deterministic decoder. Think of this as a sieve or a stamp. It takes the large, detailed image and shrinks it down to the strict 64x64 size, ensuring that every pixel lands exactly where the game engine expects it to be. No more swapped arms or floating heads!

Why This Matters

• It's Open Source: Anyone can use it, not just big tech companies.
• It's Robust: It handles weird photos, anime characters, and real people equally well.
• It's Pixel-Perfect: It guarantees the skin will actually work in the game without breaking the 3D model.

The Limitations (The "Bad" Cases)

The authors admit that sometimes the "Translator" (Stage 1) isn't perfect. If the blueprint it creates is too detailed or messy, the "Architect" (Stage 2) might struggle to squish it into the tiny grid, resulting in a blurry or lost detail. It's like trying to fit a high-definition photo onto a tiny stamp; some details just have to be sacrificed.

In a Nutshell

BLOCK is like a factory that takes a messy photo, turns it into a perfect, standardized drawing, and then uses a specialized robot to stamp that drawing onto a tiny, game-ready skin. By splitting the job into "Understanding the Character" and "Building the Skin," it solves a problem that was previously too messy for AI to handle alone.

Technical Summary

1. Problem Statement

Generating valid, pixel-perfect Minecraft skins (64×64 UV texture atlases) from arbitrary character concepts (e.g., photos or illustrations) is a significant challenge for current AI models.

• The Core Difficulty: While modern Multimodal Large Language Models (MLLMs) excel at understanding character references and following layout instructions, they struggle with the strict structural constraints of Minecraft skins.
• Failure Modes: Direct one-stage generation often results in:
  • Structural Errors: Misaligned UV regions (e.g., swapping front/back views, incorrect limb placement).
  • Constraint Violations: Textures that violate the rigid blocky geometry or overlay rules of Minecraft.
  • Data Scarcity: There is a lack of large-scale datasets pairing arbitrary character images with their corresponding valid skin atlases.
  • Input Noise: Real-world inputs vary wildly in pose, camera angle, and style, making it difficult for a single model to learn a stable mapping to a canonical UV space.

2. Methodology: The BLOCK Pipeline

The authors propose BLOCK, a two-stage pipeline that decomposes the complex generation task into manageable sub-problems: Semantic/Style Synthesis followed by Structured UV Translation.

Stage 1: Character → 3D Preview (Semantic Synthesis)

• Goal: Convert an arbitrary character reference into a consistent, canonical "Minecraft-style" dual-panel preview (front and back views).
• Model: Uses Gemini Nano Banana Pro (identified as the only tested model capable of consistent success on this specific task).
• Input Modes:
  • Mode-I: Takes a character reference, a layout anchor (fixing framing), and a pose anchor (enforcing a strict Minecraft standing pose).
  • Mode-II: Adds a style reference (e.g., an anime or realistic skin preview) to control the rendering style while maintaining character identity.
• Output: A clean, dual-panel image (front left, back right) with a slightly oblique camera angle to expose details, a pure white background, and strict adherence to Minecraft pose constraints.
• Role: This stage handles semantic understanding, style transfer, and normalization, effectively "compressing" the noisy input into a standardized format.

Stage 2: Preview → Skin Atlas (Structured Translation)

• Goal: Translate the Stage-1 preview into a valid 512×512 skin atlas (which is an 8× upscaling of the final 64×64 skin).
• Model: A fine-tuned FLUX.2 model.
• Process:
  1. Image-to-Image Generation: The model takes the dual-panel preview as a conditioning image and generates the corresponding UV atlas.
  2. Deterministic Decoding: The 512×512 output is downsampled via nearest-neighbor interpolation to 64×64, enforcing the strict Minecraft skin structure (head, torso, limbs, overlay layers).
• Role: This stage focuses purely on the high-fidelity, pixel-perfect translation from a 3D-like view to the 2D UV coordinate system, ensuring structural validity.

Key Innovation: EvolveLoRA

To train the Stage-2 model efficiently and stably, the authors propose EvolveLoRA, a progressive curriculum learning strategy using Low-Rank Adaptation (LoRA). Instead of training from scratch, the model evolves through three phases, initializing each new phase with the weights from the previous one:

1. Phase I (Text-to-Image): Trains on raw skin atlases with automatically generated text descriptions (e.g., "head is red, body is blue"). This injects Minecraft-specific priors into the model.
2. Phase II (Image-to-Image): Trains on paired data where the input is a front/back character render and the output is the skin atlas. This bridges the gap between text and visual conditioning.
3. Phase III (Preview-to-Atlas): The final phase trains on the actual Stage-1 style 3D previews. The model learns to map the oblique, stylized previews to the UV atlas.

• Benefit: This curriculum significantly improves convergence stability and efficiency, preventing the model from failing when faced with the complex domain gap of 3D previews immediately.

3. Key Contributions

1. BLOCK Pipeline: An open-source, two-stage architecture that successfully decouples semantic understanding (MLLM) from structural UV translation (Diffusion/Flow model), solving the brittleness of one-stage generation.
2. EvolveLoRA: A novel progressive adapter initialization curriculum (T2I → I2I → Preview-to-Skin) that enables stable fine-tuning of large models on complex translation tasks with limited compute.
3. Data & Tooling:
   • Construction of a scalable data pipeline using the Minecraft-Skins-20M dataset.
   • Release of prompt templates, fine-tuned weights, and a deterministic decoder to ensure reproducibility.
4. Benchmarking: Identification of current failure modes in MLLMs regarding UV constraints and providing a robust solution for pixel-perfect skin generation.

4. Results

• Qualitative Performance: The system successfully generates skins from diverse inputs, including real-world photos, anime characters, and complex concepts with fine-grained details.
• Handling Complexity: It maintains character identity (clothing, colors) while strictly adhering to Minecraft's blocky geometry and UV layout rules.
• Failure Cases: The paper acknowledges that if Stage-1 fails to "compress" details aggressively enough (leaving too much high-frequency noise), the deterministic downsampling in Stage-2 may result in aliasing or loss of detail. However, the overall success rate is significantly higher than direct generation attempts.

5. Significance and Future Work

• Significance: BLOCK demonstrates that decomposing complex generative tasks into semantic normalization and structural translation is a viable strategy for constrained domains like game asset generation. It makes high-quality, reproducible Minecraft skin generation accessible to the open-source community.
• Future Directions:
  • End-to-End Optimization: Moving toward direct view-to-skin generation without the intermediate MLLM stage.
  • Model Distillation: Reducing the computational cost by distilling the FLUX.2 capabilities into smaller models (e.g., 4B parameters) for wider deployment.
  • Dataset Standardization: Advocating for standardized benchmarks and curated datasets to better evaluate UV-structure correctness and overlay validity.

In summary, BLOCK represents a practical breakthrough in bridging the gap between general-purpose AI image generation and the rigid, technical requirements of game asset creation.