🤖 AI

LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?

The paper introduces LEGO-Puzzles, a benchmark evaluating the multi-step spatial reasoning capabilities of Multimodal Large Language Models (MLLMs) through elementary tasks and assembly planning, revealing that even the strongest models significantly underperform compared to humans and struggle to maintain accuracy as planning complexity increases.

Original authors: Kexian Tang, Junyao Gao, Yanhong Zeng, Haodong Duan, Yanan Sun, Zhening Xing, Wenran Liu, Kai Chen, Kaifeng Lyu

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kexian Tang, Junyao Gao, Yanhong Zeng, Haodong Duan, Yanan Sun, Zhening Xing, Wenran Liu, Kai Chen, Kaifeng Lyu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to build a complex castle out of blocks, but you can only talk to it and show it pictures. This is the world of Multimodal Large Language Models (MLLMs). Think of these models as super-smart digital brains that can read text and "see" images, trying to understand the world just like humans do. One of the trickiest things for any brain to learn is spatial reasoning—the ability to understand how objects fit together in 3D space, knowing which block goes on top, which one needs to be turned around, and how the whole structure changes as you add pieces. While these AI models are getting better at chatting and describing pictures, scientists are still wondering: Can they actually plan a sequence of moves to build something? It's one thing to look at a finished tower and say, "That's tall," but it's a whole different challenge to figure out the exact order of 50 steps to build it from scratch. If we want robots to fix our cars or help us assemble furniture, they need to master this kind of step-by-step thinking.

Enter LEGO-Puzzles, a new "exam" designed by researchers to test exactly this skill. Instead of using real plastic bricks, the researchers created a massive digital playground using LEGO instructions to see how well the smartest AI models in the world can handle spatial puzzles. They broke the test down into two main levels. The first level, the Elementary Set, is like a warm-up. It asks the AI simple questions about a single picture: "Which block is taller?" "Are these two pieces touching?" or "If I turn this piece, what does it look like from the side?" This tests basic "spatial understanding." The second level, the Planning Set, is the real boss battle. Here, the AI isn't just answering questions; it has to act like a master builder. It is given a starting pile of blocks and a picture of the final goal, and it must generate a step-by-step plan to get there. The researchers tested 29 of the most advanced AI models available, including the biggest names like GPT-5 and Gemini, and even asked human experts to take the same test.

The results were a bit of a reality check for the AI world. Even the strongest models struggled with the basics. In the elementary questions, the best AI models fell at least 20% behind human performance. For example, when asked to judge the height of blocks in 3D space, many models got confused, looking at the flat 2D picture on the screen instead of imagining the true 3D shape. But the trouble got much worse as the tasks got longer. When the AI had to plan a sequence of steps using multiple-choice questions, their accuracy started to drop significantly once the plan exceeded 3 steps, falling below 50%. By the time the plan required 8 steps, the accuracy of the best models dropped to 0%. In contrast, human participants solved every single planning task perfectly, no matter how long the chain of steps was.

The researchers also tried something even harder: asking the AI to not just describe the steps, but to actually draw the intermediate pictures of the building process. They wanted to see if the AI could imagine what the castle looks like after step 1, then step 2, and so on. The results were stark. Even for a short plan of just 3 steps, the models failed completely at generating the correct visual sequence. They could sometimes draw a picture that looked like a LEGO brick, but they couldn't place it in the right spot or get the orientation correct. The paper notes that because the models failed so badly at just 3 steps for image generation, the researchers didn't even test longer horizons for this specific task. This suggests that while these models are great at recognizing patterns, they lack the ability to "imagine" the consequences of their actions over time. They can't hold a mental map of the building process in their heads without losing track.

In short, LEGO-Puzzles reveals that while our current AI models are impressive, they are still very far from having true spatial intelligence. They can't reliably plan a multi-step assembly, and they fail miserably when asked to visualize the future states of an object. The paper concludes that we have a long way to go before AI can truly understand the physical world well enough to help us build things, fix broken items, or navigate complex environments on its own. The gap between human intuition and machine calculation in the physical world is still wide, and this new benchmark gives us a clear map of exactly where the models are stumbling.

Technical Summary: LEGO-Puzzles

Problem Statement

While Multimodal Large Language Models (MLLMs) have demonstrated progress in spatial intelligence, their capability for multi-step sequential spatial reasoning remains largely unexplored. Existing benchmarks often focus on static spatial understanding (e.g., identifying object attributes or relations in a single image) or single-step action planning. However, real-world applications such as robotic control, autonomous driving, and automated assembly require the ability to perceive a scene, imagine the consequences of hypothetical actions, and sequentially apply spatial understanding across multiple steps. Current evaluations lack a systematic framework to test MLLMs from basic spatial perception to complex, long-horizon planning.

Methodology

The authors introduce LEGO-Puzzles, a benchmark designed to evaluate MLLMs' spatial reasoning capabilities through a progressive framework inspired by LEGO construction. The benchmark consists of two primary task sets derived from open-source LEGO projects with step-by-step assembly instructions:

1. The Elementary Set

This set contains over 1,100 carefully curated Visual Question-Answering (VQA) pairs spanning 11 distinct tasks, organized into three levels of increasing complexity:

Level 1: Spatial Understanding: Tests basic perception of 3D relationships, including height distinction, adjacency, rotation angles, and multiview prediction.
Level 2: Single-Step Sequential Reasoning: Requires reasoning about immediate assembly actions, such as determining rotation status, identifying correct assembly positions, predicting the next state, and identifying dependencies.
Level 3: Multi-Step Sequential Reasoning: Assesses the ability to reason across multiple sequences, including backward reasoning (identifying intermediate steps from a target), ordering assembly sequences, and detecting outliers in a sequence.

2. The Planning Set

This set evaluates the model's ability to generate explicit multi-step plans for assembling a target structure. It includes two task types:

Plan-k-Step-VQA: A multiple-choice task where models must select and order $k$ correct intermediate steps from a set of $2k$ options (including distractors) to transition from an initial state to a target state. The planning horizon $k$ ranges from 1 to 8.
Plan-k-Step-Generation: A generative task where models must produce images of intermediate assembly states step-by-step in a multi-turn dialogue, conditioned on previous outputs, to simulate a full-stage planning trajectory.

Data Curation

The dataset was constructed using a pipeline involving:

Data Collection: Gathering diverse open-source LEGO projects (animals, vehicles, furniture) and filtering for moderate complexity.
Rendering & Transformation: Using Studio1 to render projects into PDFs with fixed camera viewpoints, while applying transformations (e.g., lighting adjustments, deliberate errors) to create specific task scenarios.
Metadata Extraction & Unification: Extracting structured information (pieces, objects) and organizing assets for QA generation.
Quality Control: A multi-stage human-in-the-loop process including duplicate filtering, image quality verification, and annotation validation by three trained annotators.

Key Contributions

Comprehensive Evaluation Framework: A benchmark covering a spectrum from basic spatial understanding to multi-step planning with progressively increasing complexity.
Natural yet Challenging Tasks: Tasks constructed from real human-designed LEGO projects that pose significant challenges for frontier MLLMs, moving beyond simplified or highly abstract environments found in prior work.
Multi-Turn Image Generation Evaluation: A novel evaluation of MLLMs' ability to generate planned intermediate states as images, a capability crucial for real-world applications like robotic manipulation but previously untested in spatial reasoning benchmarks.

Experimental Results

The authors evaluated 29 state-of-the-art MLLMs, including proprietary models (e.g., GPT-5, Gemini-2.5-Pro) and open-source alternatives (e.g., Qwen, InternVL, LLaVA series).

Elementary Set Performance: Even the strongest models (GPT-5) lag behind human performance by at least 20% on elementary tasks. Many open-source models perform only marginally better than random guessing. Specific failures include:
- Height Task: Models often rely on 2D projections rather than true 3D perspective.
- Rotation Task: Models struggle to reliably perceive orientation changes.
- Multi-Step Reasoning: Accuracy drops significantly as tasks move from single-step to ordering or backward reasoning.
Planning Set Performance (VQA): As the planning horizon $k$ $k$ increases, accuracy degrades rapidly.
- For $k > 3$ , accuracy for all models drops below 50%.
- At $k = 8$ , accuracy approaches 0% for most models, whereas human participants achieve 100% accuracy.
- Proprietary models show better "set match" (identifying correct elements) than open-source models but still fail to order them correctly.
Planning Set Performance (Image Generation): When switching from multiple-choice to image generation, performance collapses.
- Even for $k=3$ , accuracy drops to 0%.
- Models often generate images with correct general appearance but fail to place LEGO pieces in correct positions or orientations.
- Errors frequently appear in the very first step of generation, indicating a lack of consistent visual imagination and temporal reasoning.

Significance and Claims

The paper claims that LEGO-Puzzles reveals critical limitations in current MLLMs' spatial reasoning capabilities. The findings highlight that:

Current models struggle with long-horizon planning and sequential spatial reasoning, failing to track and integrate spatial transformations across multiple steps.
There is a substantial gap between visual understanding and precise generation; models can recognize spatial relationships in static images but cannot reliably apply this reasoning to generate coherent, step-by-step visual outputs.
The benchmark serves as a proxy for real-world spatial understanding, validated by strong positive correlations (Pearson $r > 0.9$ ) between performance on LEGO-Puzzles and natural image benchmarks (3DSRBench).

The authors conclude that substantial advances are needed to bridge the gap between current MLLMs and human-level spatial intelligence, particularly in the domains of sequential reasoning and generative planning.