🤖 AI

Rule-Compliant Visual Spatial Planning for Multimodal Large Language Models

This paper introduces RuleMaze, a controllable benchmark for evaluating rule-compliant visual spatial planning in multimodal large language models, along with a hybrid rule generation framework and a Disentangled Multimodal Planning (DMP) method that significantly improves rule adherence and generalization by separating perception, execution, and verification.

Original authors: Yu Chen, Ting Lei, Yaoyi Li, Jia Cai, Zhecen Wu, Yang Liu

Published 2026-08-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yu Chen, Ting Lei, Yaoyi Li, Jia Cai, Zhecen Wu, Yang Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where a computer can look at a picture and understand not just what is in it, but how to move through it. This is the promise of multimodal large language models, a new generation of artificial intelligence that combines the ability to read and reason with the ability to see. These systems have already learned to describe images, answer questions about photos, and solve visual puzzles. But there is a gap in their knowledge. While they can navigate a simple path, they struggle when the path is blocked by a specific set of instructions that change from moment to moment. In the real world, an autonomous vehicle or a robot helper must not only find a way from point A to point B, but also obey a constantly shifting set of rules, such as "do not turn left on red" or "avoid the wet floor." Current models often fail here, treating rules as suggestions rather than hard constraints, or forgetting them entirely as they plan their next move.

To understand why this is so difficult, researchers Yu Chen and his team at Peking University and Yinwang Intelligent Technology created a new testing ground called RuleMaze. They built a digital environment filled with mazes, but these were not just puzzles of walls and open spaces. Each maze came with a unique set of natural language rules, written in plain English, that dictated exactly how a character could move. Some rules were simple, like "you cannot step on pink squares." Others were complex, requiring the character to remember where it had been or to change its behavior based on its current location, such as "if you are standing on an orange square, your next step must be upward." The researchers generated thousands of these scenarios using a method that automatically turned human-written rules into strict, machine-checkable logic. This allowed them to test whether an artificial intelligence could truly understand and follow a rule while navigating a visual space, rather than just guessing the right path based on patterns it had seen before.

The team discovered that simply asking a powerful model to look at the maze and write down a list of moves did not work well, especially when the rules were new or complicated. When the rules changed, the models often forgot the instructions or violated them without realizing it. To solve this, the researchers proposed a new way of thinking called Disentangled Multimodal Planning. Instead of forcing the model to do everything at once—seeing the picture, thinking about the rules, and deciding the next step all in one go—they broke the task into three separate, clear jobs. First, the model uses a tool to look at the maze and identify where it is and what special symbols are nearby. Second, it uses a tool to try a move and see what the new picture looks like. Third, and most importantly, it uses a dedicated tool to check if the move it just made follows the rules. This checking tool acts like a strict referee that instantly says "yes" or "no" to every single step before the model is allowed to continue.

This approach changed the outcome dramatically. In their tests, the new method allowed the models to follow rules with a success rate of 90 percent on tasks they had never seen before, a massive improvement over previous methods which often failed completely on new rules. The researchers found that when the model was forced to check its work at every single step, it stopped making the kinds of mistakes that happen when a system tries to plan a whole journey in its head without verifying each step. The model learned to pause, look at the rule, check the move, and only then proceed. This worked even when the rules were difficult, involving multiple conditions or requiring the model to remember items it had picked up along the way. The study suggests that for artificial intelligence to be truly reliable in complex, rule-bound environments, it cannot just be a smart guesser; it must be equipped with a system that forces it to verify its own actions against the constraints of the world it is navigating. By separating the act of seeing, the act of moving, and the act of checking, the researchers showed that machines can learn to follow instructions with a level of discipline that was previously out of reach.

Technical Summary: Rule-Compliant Visual Spatial Planning for Multimodal Large Language Models

1. Problem Definition

Multimodal Large Language Models (MLLMs) have demonstrated significant capabilities in visual reasoning and language understanding. However, their ability to perform rule-compliant visual spatial planning remains underexplored. This specific capability requires an agent to simultaneously:

Perceive complex visual spatial layouts.
Interpret explicit natural-language rules (which may be dynamic or previously unseen).
Generate multi-step action plans that strictly adhere to these constraints.

Current benchmarks and methods often assume unconstrained planning or optimize for single objectives without enforcing external rule compliance. Real-world applications, such as autonomous driving in new regulatory environments or embodied intelligence, require agents to operate under changing, context-specific constraints rather than fixed global objectives. Existing approaches struggle with this due to:

Data Scarcity: Real-world scenes rarely provide ground-truth action sequences for dynamically varying rule sets, making supervised learning expensive and difficult to scale.
Methodological Limitations: End-to-end formulations often entangle perception, execution, and rule reasoning. This leads to brittle behavior when models encounter multi-step constraints or novel rules, as the model must relearn stable capabilities whenever rules change.

2. Methodology

The paper proposes a two-pronged solution: a controllable benchmark for evaluation and a disentangled framework for planning.

2.1 RuleMaze: A Controllable Benchmark

To address the lack of systematic evaluation, the authors introduce RuleMaze, a dataset of procedurally generated maze-rule pairs.

Language–Logic–Function Hybridization: To enable scalable rule construction without manual engineering, the authors propose a pipeline:
1. Rule Ideation: LLMs generate diverse, linguistically rich natural-language rules (e.g., "If in an orange zone, move up").
2. Logical Formalization: Rules are translated into structured logical representations (e.g., $Zone(t, orange) \rightarrow Move(t, up)$ ).
3. Validator Synthesis: LLMs automatically generate executable Python functions (validators) that verify whether a candidate trajectory satisfies the logical rule.
Constraint-Based Heuristic Matching: A strategy is employed to pair rules with mazes where the rule is meaningful. A maze is selected for a rule only if:
1. The rule restricts the set of valid trajectories (Restrictiveness).
2. At least one rule-compliant solution exists (Existence).
Dataset Scenarios:
- RegularMaze: Focuses on static goals and local constraints (e.g., avoiding colored zones).
- QuestMaze: Introduces semantic objects (keys, treasures) and state-dependent rules, requiring memory and handling dynamic goal conditions.
Difficulty Levels: Rules are categorized as Easy, Medium, or Hard based on the syntactic complexity (number of logical connectives) of their formalizations.

2.2 Disentangled Multimodal Planning (DMP)

To improve generalization and rule compliance, the authors propose DMP, a framework that modularizes the planning process rather than relying on end-to-end training.

Tool-Based Architecture: DMP equips the MLLM with a set of interpretable, callable tools organized into three categories:
1. Perception Tools: Extract symbolic information from the maze image (e.g., identifying current cell type, start/goal locations).
2. Execution Tools: Deterministically update the visual state by applying movement actions (e.g., ExecuteMove).
3. Verification Tools: Assess rule compliance and goal achievement using the validators synthesized during dataset construction (e.g., VerifyRule, VerifyEndpoint).
Iterative Decision Process: The model acts as a controller that alternates between internal reasoning and external tool invocation. At each step, it generates a program to select and parameterize a tool.
Key Advantage: By externalizing stable components (perception, execution, verification) into tools, the controller only needs to learn when and how to orchestrate them. This allows the system to generalize to new rules without retraining the core controller, as new rules simply require new validators.

3. Key Contributions

RuleMaze Benchmark: A principled benchmark featuring diverse procedurally generated mazes and systematically constructed rule sets, grounded by executable verification tools to ensure unambiguous evaluation.
Language–Logic–Function Hybridization: A pipeline for the scalable, automated construction of rule sets with controlled complexity, eliminating the need for manual rule engineering.
Disentangled Multimodal Planning (DMP): A framework that enhances rule-following by separating perception, execution, and verification into reusable tools, enabling interpretable, multi-step reasoning under constraints.
Empirical Validation: Extensive experiments demonstrating that DMP significantly outperforms end-to-end textual planning baselines, particularly in zero-shot generalization to unseen and complex rules.

4. Experimental Results

The authors evaluated DMP (using a Qwen2.5-VL-3B backbone) against baselines including Direct prompting, Chain-of-Thought (CoT), Supervised Fine-Tuning (SFT), and proprietary models (Gemini 2.5 Pro/Flash).

RegularMaze Performance:
- Seen Rules: DMP achieved 98.0% Exact Match (EM), outperforming SFT (94.0%).
- Unseen Rules: DMP achieved 90.0% EM, a substantial improvement over SFT (70.3%) and significantly higher than prompting-based methods (which struggled near 0% for open-source models).
- Complexity: DMP maintained high performance across Easy, Medium, and Hard rule complexities, whereas performance gaps widened for baselines as complexity increased.
QuestMaze Performance:
- In the more complex semantic environment, DMP achieved 91.4% EM on seen rules and 88.0% on unseen rules.
- DMP improved unseen rule performance by +31.7% over SFT (88.0% vs. 56.3%).
- While proprietary models like Gemini 2.5 Pro showed stronger performance on QuestMaze than on RegularMaze, DMP still outperformed them, especially in hard rule scenarios.
Qualitative Analysis: Baseline models often failed by violating rules after entering constrained zones or heuristically waiving rules when blocked. DMP successfully utilized step-wise verification tools to detect potential violations and revise plans before execution.

5. Significance and Claims

The paper claims that RuleMaze establishes a principled framework for studying grounded and interpretable rule-based spatial planning in MLLMs. The significance of the work lies in:

Addressing a Gap: It targets the critical but understudied capability of operating under explicit, dynamic rule constraints, which is essential for real-world embodied intelligence.
Scalability: The proposed hybridization pipeline solves the data bottleneck by automating the creation of complex rule-maze pairs.
Generalization: The DMP framework demonstrates that disentangling stable capabilities from the coordination logic allows models to generalize to previously unseen rules without retraining, offering a path toward more robust and adaptable multimodal agents.

The authors conclude that while current MLLMs show promise, structured approaches like DMP are necessary to achieve reliable, rule-compliant spatial planning in complex environments.