💻 computer science

OccluRank: Controllable Occlusion-Aware Layout-to-Image Generation by Adding Just an Ordinal Rank

OccluRank is a simple and controllable framework for layout-to-image generation that achieves precise occlusion-aware synthesis by augmenting bounding boxes with a single ordinal rank and employing an Order-aware Instance Interaction module, eliminating the need for complex geometric conditions or specialized inference procedures.

Original authors: Wenyang Hong, Yuan Wang, Yanbin Hao, Lanqing Xue, Ke Wang, Xiang Wang, Kuien Liu, Richang Hong

Published 2026-08-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenyang Hong, Yuan Wang, Yanbin Hao, Lanqing Xue, Ke Wang, Xiang Wang, Kuien Liu, Richang Hong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of computer vision, researchers have long taught machines to generate images from text descriptions. More recently, they developed a way to give these machines precise instructions about where objects should appear in a picture. By drawing a simple box around a specific area and labeling it, a user can tell the computer to place a cat there, a car there, or a tree there. This process, known as layout-to-image generation, has become a powerful tool for designers and creators who need to arrange scenes with exact control. However, a significant blind spot remains in this technology. While a box can tell the computer where an object is, it cannot tell the computer which object is in front of another. In the real world, objects often overlap; a person might stand in front of a bench, or a cup might sit on a table, hiding part of the surface beneath it. Current computer models struggle with this depth, often blending overlapping objects into a single, confused mess or placing them in the wrong order, making the scene look physically impossible.

A team of researchers has now introduced a new method called OccluRank to solve this specific problem of visual overlap. Their approach is surprisingly simple, adding just one piece of information to the standard instructions: a ranking number. Instead of relying on complex 3D models, depth maps, or multiple layers of geometric data, the system asks the user to assign a simple number to each object, indicating whether it should appear in the foreground or the background. A lower number means the object is closer to the viewer, while a higher number means it is further away. The researchers built a framework that uses this single ranking to guide the computer's decision-making process. Before the computer combines the different parts of the image, it uses a special module to let the objects "talk" to each other. If a motorcycle and a bookshelf are meant to overlap, the system checks their assigned numbers and ensures the motorcycle's features are processed as if it is sitting on top of the bookshelf, effectively hiding the part of the bookshelf that should be behind it.

To test if this method actually works, the team created a new dataset called OccluLayout. They built thousands of scenes using 3D computer graphics software, arranging furniture, animals, and vehicles in various overlapping configurations. Because they built these scenes from scratch, they knew the exact order in which every object appeared. They then used artificial intelligence to edit the backgrounds and add detailed descriptions to the objects, creating a massive library of training data where the correct answer was known with certainty. This allowed them to train their model without guessing or relying on imperfect estimates of depth. They also developed a new testing system, OccluLayout-Bench, which uses advanced language models to grade the results. Instead of just looking at the overall picture quality, these evaluators check specific details: did the computer draw all the requested objects? Did they stay inside their boxes? Did they keep their colors and textures? Most importantly, did the overlapping objects appear in the correct front-to-back order?

The results show that this simple ranking system is highly effective. When compared to other methods that try to solve the same problem, the new approach consistently produced images where the objects were more clearly defined and the overlapping relationships were correct. In tests involving complex scenes with multiple items, the model successfully preserved the visibility of objects that should be in front while correctly obscuring the parts of objects that should be behind them. It managed to do this without needing extra inputs like 3D coordinates or camera angles, relying only on the user's simple list of numbers. The images generated were not only structurally accurate regarding the order of objects but also maintained high visual quality, with the objects looking natural and their attributes remaining consistent.

The researchers found that the key to their success was the way the system handled the interaction between objects before combining them into a final image. By letting the objects exchange information based on their assigned ranks, the model learned to resolve conflicts in the overlapping areas. If the system was told that a zebra was in front of a refrigerator, it adjusted the features of the zebra to cover the refrigerator, rather than blending the two together. This capability held true even when the researchers tested the model on objects it had never seen before during training, suggesting that the method learned a general rule about how to handle depth rather than just memorizing specific pictures. The study demonstrates that by adding a single, intuitive layer of instruction, computers can be guided to create complex, layered scenes that respect the physical reality of how objects occupy space.

Technical Summary: OccluRank

Problem Statement

Layout-to-image generation enables explicit spatial control over object placement and instance-level semantics through bounding-box layouts. However, standard bounding boxes specify only the spatial extent of instances and fail to represent their occlusion order (i.e., which object appears in front of another). This limitation leads to incorrect occlusion relationships, merged objects, or attribute confusion in crowded scenes, human-object interactions, and layered compositions.

Existing approaches to address this face two primary challenges:

Occlusion-dependent instance interaction: Current methods often isolate instance semantics or use gated aggregation (e.g., IFAdapter) that reweights independently constructed features without explicitly modeling interactions based on visibility order. This can result in ambiguous layering or feature mixing.
Simple and controllable order specification: Methods that do incorporate order often rely on complex geometric inputs (e.g., 3D bounding boxes, camera parameters, masks) or specialized, computationally expensive inference procedures (e.g., iterative latent optimization, sequential denoising). There remains a need for a method that incorporates user-specified occlusion order through a simple condition compatible with standard layout-to-image generation.

Methodology: OccluRank

The authors propose OccluRank, a controllable, occlusion-aware layout-to-image framework built upon a pretrained Stable Diffusion XL (SDXL) model. The core innovation is augmenting each bounding box with a single ordinal rank to specify the intended front-to-back occlusion order.

Key Components

Ordinal Rank Conditioning:
- Each instance $i$ is assigned a rank $r_i \in \{1, \dots, O\}$ , where a smaller value indicates a more foreground position.
- These ranks are encoded as learnable embeddings and injected into the corresponding instance representations. This allows for explicit order control without requiring additional geometric inputs like depth maps or 3D parameters.
Order-aware Instance Interaction (OII) Module:
- OccluRank adopts an interaction-before-aggregation pipeline. Instead of directly aggregating independently constructed instance feature maps, the model keeps them separated initially.
- At each latent location covered by multiple instances, the rank-conditioned features are organized into a sequence along the instance dimension.
- A Transformer block processes this sequence, allowing overlapping instance features to interact and update jointly based on their semantic content and ordinal roles.
- This mechanism enables the specified order to guide information exchange, allowing an instance to adapt its representation based on which objects compete with it and which should appear in front.
Aggregation and Injection:
- After interaction, the updated features are aggregated into an Instance Semantic Map (ISM).
- The ISM is injected into selected cross-attention layers of the SDXL backbone via a gated residual connection, ensuring the generated image respects the specified layout and occlusion order.
Training Objective:
- The model is trained using a global denoising loss ( $\mathcal{L}_{global}$ ) and a region-focused local denoising loss ( $\mathcal{L}_{local}$ ).
- The local loss specifically targets the union of instance regions, providing stronger supervision for instance structures and occlusion boundaries where visibility order is expressed.

Datasets and Benchmarks

To support training and evaluation, the authors introduce two new resources:

OccluLayout: A synthetic training dataset constructed from controllable 3D scenes in Blender. Unlike datasets that infer occlusion from partially occluded images using auxiliary models, OccluLayout derives occlusion order, amodal bounding boxes, and amodal masks directly from known scene geometry. It includes 34,496 images with fine-grained instance descriptions and 71 object categories.
OccluLayout-Bench: A comprehensive evaluation benchmark comprising 1,000 held-out images. It evaluates five dimensions:
1. Instance Presence: Whether requested instances are recognizable.
2. Spatial Layout: Alignment with input boxes (Box mIoU).
3. Attribute Consistency: Color and Texture adherence.
4. Occlusion Order: Strict Pair and Strict Image metrics for correct visibility relationships.
5. Image Quality: Fréchet Inception Distance (FID).
- Evaluation is performed using multiple Multimodal Large Language Models (MLLMs) to ensure robustness.

Results

Experiments on OccluLayout-Bench demonstrate that OccluRank outperforms existing state-of-the-art methods (including IFAdapter, CreatiLayout, LaRender, VODiff, and OcclusionFormer) across most metrics:

Occlusion Control: OccluRank achieves the highest scores in Strict Pair and Strict Image metrics, significantly outperforming baselines like IFAdapter (improvements of 7.38%–10.61% in Strict Pair). This indicates a superior ability to resolve competing conditions in overlapping regions.
Instance Preservation: The model achieves the best Presence scores, reliably preserving target instances even under substantial occlusion.
Attribute Consistency: It maintains competitive or superior performance in Color and Texture metrics, ensuring that the specified attributes are preserved despite the complex occlusion handling.
Image Quality: OccluRank maintains a competitive FID score (62.746), comparable to strong baselines, demonstrating that the added control does not compromise overall image fidelity.
Ablation Studies: Removing the OII module, rank embeddings, or the local loss objective leads to degraded performance in occlusion ordering and boundary coherence, validating the necessity of each component.

Significance and Claims

The paper claims that OccluRank offers a simple and controllable solution to the occlusion-aware layout-to-image problem. By adding only one ordinal rank per instance, it achieves explicit visibility control without the need for complex geometric inputs or specialized inference-time optimization.

The authors emphasize that their Order-aware Instance Interaction (OII) module effectively models occlusion-dependent interactions before feature aggregation, allowing the specified order to guide high-dimensional content updates rather than just reweighting independent features. Furthermore, the introduction of OccluLayout and OccluLayout-Bench addresses the lack of reliable, geometry-derived occlusion supervision and comprehensive evaluation protocols in the field, providing a foundation for future research in controllable image generation.