Tokenization Allows Multimodal Large Language Models to Understand, Generate and Edit Architectural Floor Plans

The paper introduces HouseMind, a multimodal large language model that unifies the understanding, generation, and editing of architectural floor plans by employing discrete room-instance tokens to bridge visual layouts with symbolic reasoning, thereby achieving superior geometric validity and controllability.

The Gist

Imagine you want to build a house, but instead of hiring an architect, you ask a very smart robot to draw the floor plan for you. You say, "I want a big living room in the middle, a kitchen to the north, and a bedroom to the south."

For a long time, AI was like a painter who could make a picture look pretty but didn't understand how a house actually works. It might draw a kitchen that floats in the air, a bedroom with no door, or a bathroom that is inside the living room. It was good at making pixels look nice, but bad at understanding the logic of space.

Enter HouseMind, a new AI system that changes the game. Here is how it works, explained simply:

1. The Problem: The "Pixel Painter" vs. The "Architect"

Think of old AI models as Pixel Painters. They look at a floor plan and try to guess what color goes where, pixel by pixel. They are like someone trying to recreate a map by copying every single dot on a piece of paper. If they miss one dot, the road might disappear. They struggle to understand that a "kitchen" needs to be next to a "dining room" or that a "bedroom" needs a door.

HouseMind is different. It acts like a Master Architect who speaks a special language of space.

2. The Secret Sauce: "Legos" instead of "Pixels"

The paper's big idea is Tokenization.

Imagine you have a giant box of LEGO bricks.

• Old AI: Tries to build the house by painting every single square inch of the wall.
• HouseMind: Uses pre-made LEGO bricks.
  • One brick is a "Kitchen."
  • One brick is a "Living Room."
  • One brick is a "Wall."
  • One brick is a "Door."

Instead of looking at a blurry image, HouseMind breaks the floor plan down into these discrete "Room Tokens" (like LEGO bricks). It turns the complex drawing into a simple list of words and codes, like a sentence:
<LivingRoom> <Kitchen> <Wall> <Bedroom>

3. How HouseMind "Thinks"

HouseMind is a Multimodal Large Language Model (MLLM). You can think of it as a super-smart translator that speaks two languages fluently:

1. Human Language: "Put the kitchen next to the living room."
2. Space Language: The list of LEGO bricks (tokens) that make up the floor plan.

Because it treats the floor plan like a sentence, it can use the same logic it uses to write a story to design a house.

• If you say, "The kitchen is to the left of the living room," the model understands the relationship between the bricks, not just the colors.
• It knows that a "Bathroom" usually needs a "Wall" around it and a "Door" to enter.

4. The Three Superpowers

The paper shows HouseMind doing three things that other AIs struggle with:

• Understanding (The Detective):
  You show it a floor plan, and it can tell you exactly what is happening. "Ah, I see a large living room in the center, with a kitchen to the northeast and a bedroom to the southwest." It doesn't just guess; it reads the "sentence" of the floor plan.

• Generating (The Creator):
  You give it a text prompt: "Design a house with 3 bedrooms and a big balcony." It doesn't just paint a picture; it assembles the LEGO bricks in the correct order to build a logical, usable house. It ensures the rooms connect properly and fit inside the outline.

• Editing (The Renovator):
  This is the coolest part. You can say, "Remove the balcony and add a small study."

  • Old AI: Might try to "paint over" the balcony, often messing up the walls or making the study float.
  • HouseMind: It simply takes the "Balcony" brick out of the list and swaps it for a "Study" brick. It knows exactly how to rearrange the remaining bricks so the house still makes sense.

5. Why This Matters

• It's Logical: It doesn't just make things look pretty; it makes things work. The rooms connect, the doors open, and the flow is logical.
• It's Fast & Small: Because it uses these "LEGO bricks" (tokens) instead of processing millions of pixels, it runs very fast and can even run on a single computer (locally), not just on massive supercomputers.
• It's Controllable: You can tell it exactly what to change, and it will do exactly that, without breaking the rest of the house.

The Bottom Line

HouseMind is like giving an AI a set of magic LEGO instructions instead of a paintbrush. It teaches the computer to understand that a house isn't just a picture; it's a puzzle of connected spaces. By turning floor plans into a language the AI already understands, it can finally design homes that are not just visually correct, but logically sound and ready to build.

Technical Summary

1. Problem Statement

Architectural floor plan design is a cognitively demanding task requiring the joint reasoning of geometry (spatial dimensions, boundaries), semantics (room functions), and spatial hierarchy (adjacency, circulation). Current AI systems face four major limitations:

• Lack of Global Coherence: Diffusion and autoregressive models often treat layout synthesis as a purely visual process, generating locally plausible but globally incoherent plans (e.g., inconsistent room adjacencies).
• Black-Box Nature: Large Vision-Language Models (VLMs) often lack interpretability and fine-grained spatial controllability.
• Task Fragmentation: Existing frameworks struggle to unify understanding, generation, and editing within a single architecture.
• Deployment Constraints: Most state-of-the-art systems are computationally expensive and difficult to deploy locally.

The core challenge is bridging the gap between continuous geometric layouts and discrete symbolic reasoning to enable controllable, interpretable, and coherent spatial design.

2. Methodology: HouseMind

HouseMind is a unified, lightweight, and locally deployable Multimodal Large Language Model (MLLM) framework. It treats floor plans as a language of space by discretizing continuous geometry into tokens.

A. Hierarchical Tokenization (VQ-VAE)

The framework converts continuous floor plan images into discrete token sequences using two Vector-Quantized Variational Autoencoder (VQ-VAE) branches:

1. Outline Discretization: A CNN encoder extracts features from the global building boundary (outline mask) and quantizes them into an Outline Token Sequence ($z_o$). This captures the global envelope.
2. Conditional Room Discretization: A conditional encoder processes individual room masks ($x_{r_i}$) conditioned on the outline ($x_o$). This generates Room Token Sequences ($z_{r_i}$) that capture both room geometry and spatial adjacency context.
3. Unified Vocabulary: The system constructs a unified vocabulary containing:
   • Outline tokens (geometry).
   • Room instance tokens (geometry + semantics).
   • Semantic label tokens (e.g., <room_Kitchen>, <room_LivingRoom>).

The final representation is an interleaved sequence: $Z = [z_o, \ell_{r_1}, z_{r_1}, \dots, \ell_{r_N}, z_{r_N}]$.

B. Three-Stage Multimodal Alignment & Instruction Tuning

HouseMind aligns spatial tokens with linguistic representations through a progressive training pipeline:

1. Stage 1: Embedding Initialization: Spatial codebooks from the VQ-VAE are mapped to unique trainable token embeddings within the LLM's vocabulary. This establishes a one-to-one correspondence between spatial codes and text tokens.
2. Stage 2: Multimodal Pre-training: The model is trained on large-scale paired data (text descriptions + outline/room tokens) using an autoregressive objective. It learns to predict the next token in mixed text-spatial sequences, enabling bidirectional alignment between language and geometry.
3. Stage 3: Instruction Tuning (SFT): The model is fine-tuned on curated instruction data to master three core tasks:
   • Understanding: Inferring room functions, topology, and relations from a layout.
   • Generation: Synthesizing a complete layout from a text prompt and an outline.
   • Editing: Modifying an existing layout based on natural language instructions (e.g., "add a bathroom to the west").

3. Key Contributions

• Unified Framework: HouseMind is the first model to unify floor plan understanding, generation, and editing into a single sequence-modeling architecture.
• Room-Instance Tokenization: By discretizing layouts into room-instance tokens rather than raw pixels, the model bridges symbolic reasoning and continuous geometry, enabling fine-grained control over spatial structure.
• Efficiency & Deployability: The compact architecture (based on Qwen3-0.6B) allows for real-time inference and local deployment on a single consumer GPU (RTX 3090), unlike heavy diffusion or large-scale VLM baselines.
• Structured Reasoning: The approach explicitly models room-to-room relationships and topological constraints, avoiding the "black-box" generation issues of pure image-based models.

4. Experimental Results

The authors constructed a unified benchmark based on the RPLAN dataset (78,000+ samples) covering all three tasks.

Quantitative Performance

• Understanding: HouseMind achieved near-perfect success rates (1.0) and significantly outperformed strong VLM baselines (LLaVA, Qwen-VL, InternVL) in Room Match Rate (99.8%), Location Accuracy (96.9%), and Adjacency Accuracy (99.0%). It reduced room area error to <0.6 $m^2$.
• Generation: HouseMind-G/O achieved superior Micro/Macro IoU (0.71/0.65) and FID (1.91) compared to diffusion models (ChatHouseDiffusion) and other LLM-based generators. It demonstrated better topological correctness (Node F1: 0.994).
• Editing: HouseMind-E showed high precision in structural modifications, with a $\Delta$IoU of 0.608 and minimal degradation to the original layout's structural integrity, whereas image-editing baselines often introduced irrelevant elements or failed to follow constraints.

Qualitative Analysis

• Coherence: Unlike baselines that generate "blob-like" or disconnected rooms, HouseMind produces layouts with consistent boundaries and logical circulation.
• Controllability: The model successfully executes complex instructions (e.g., "place a kitchen to the northeast of the living room") without hallucinating extra rooms or violating the building outline.
• Ablation Studies: Removing any of the three training stages (Embedding Init, Pre-training, SFT) resulted in significant performance drops, confirming the necessity of the progressive alignment strategy.

5. Significance and Future Directions

• Paradigm Shift: HouseMind demonstrates that tokenization is a critical mechanism for linking LLMs with spatial design intelligence. It shifts the paradigm from visual generation (pixel-based) to structure-aware reasoning (token-based).
• Practical Impact: The ability to run locally and handle editing tasks makes it viable for integration into professional architectural workflows, moving beyond static generation to interactive co-design.
• Limitations & Future Work: Current limitations include a focus on simple room additions/deletions (not complex topological transformations) and the lack of modeling for detailed components like doors, windows, and furniture. Future work aims to expand the dataset, integrate structural co-design, and align with human aesthetic preferences.

In conclusion, HouseMind establishes a new standard for AI-driven architectural design by proving that discrete tokenization enables LLMs to reason about spatial semantics and geometry with high fidelity, controllability, and efficiency.