💻 computer science

MS-CustomNet: Controllable Multi-Subject Customization with Hierarchical Relational Semantics

MS-CustomNet is a novel framework that enables zero-shot, fine-grained control over multi-subject image generation by allowing users to explicitly define hierarchical arrangements and spatial relationships while preserving individual subject identities, supported by a new MSI dataset and demonstrating superior performance in identity preservation and positional control.

Original authors: Pengxiang Cai, Mengyang Li

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pengxiang Cai, Mengyang Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical photo editor that can turn your text descriptions into real pictures. You can tell it, "Draw a cat on a mat," and it does. But what if you want something more specific? What if you want to say, "Put this specific cat (the one from your phone) sitting inside this specific bowl (the one from your kitchen), while that specific dog (your neighbor's) watches from behind the bowl"?

Current AI tools are like talented but slightly chaotic painters. They are great at painting a cat or a bowl, but if you ask them to put three specific things together in a very precise way, they often get confused. They might mix the cat's face with the dog's ears, or they might put the bowl floating in the sky instead of on the table. They struggle to understand the "rules" of how objects stack, hide behind each other, or interact.

Enter MS-CustomNet. Think of it as a master stage director for a play, rather than just a painter.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Concept Bleed"

Imagine you have a box of LEGO bricks. You want to build a castle, a spaceship, and a robot. If you just throw all the bricks in a pile and say "Build me a scene," the AI might accidentally glue the robot's arm onto the spaceship's wing. This is called "concept bleeding." Existing tools often fail to keep the characters distinct or to follow your instructions on who is standing in front of whom.

2. The Solution: The "Director's Script" (MS-CustomNet)

MS-CustomNet is a new system designed to be the ultimate director. It doesn't just guess; it follows a script you give it.

The Cast (Identity Preservation): You show the AI photos of your specific "actors" (a specific backpack, a specific cup, a specific dog). The system has a special memory trick that ensures the backpack stays a backpack and the dog stays a dog, even when they are moved to a new scene. It's like having a costume department that never loses the actors' unique features.
The Stage Directions (Spatial Control): This is the magic part. You don't just say "put them together." You give the AI a layout map. Think of this like a floor plan for a theater. You tell the AI: "The cup goes inside the bowl," or "The book goes on top of the stack." The AI reads this map and builds the scene exactly to those instructions, respecting who is in front of whom and who is hiding behind whom.

3. The Training Ground: The "MSI Dataset"

To teach this AI to be so good at following directions, the researchers had to build a new school. They created a massive dataset called MSI (Multi-Subject Interaction).

Imagine taking thousands of photos from a library (the COCO dataset).
They filtered out the boring ones and kept only the complex scenes with multiple objects interacting (like a dog playing with a ball near a person).
They then created "training exercises" where the AI had to learn how to take a photo of a dog, a ball, and a person, and rearrange them according to a new set of rules. This is like a student practicing with flashcards until they can solve any puzzle.

4. The Learning Strategy: "Baby Steps" (Curriculum Learning)

You wouldn't ask a toddler to run a marathon on day one. You start with walking, then jogging, then running.
The researchers used a similar strategy called Curriculum Learning.

Phase 1: They taught the AI to handle just two objects (e.g., a cup and a saucer).
Phase 2: Once the AI got good at that, they added a third object.
Phase 3: Finally, they let it handle up to five objects at once.
This prevented the AI from getting overwhelmed and confused, allowing it to learn complex interactions step-by-step.

5. The Result: A Perfectly Staged Scene

When you use MS-CustomNet, you get a result that feels like a movie set:

The Identity: The objects look exactly like the photos you gave it.
The Position: The objects are exactly where you told them to be.
The Interaction: If you said "the cup is inside the bowl," the AI knows to draw the rim of the bowl covering the top of the cup. It understands depth and layering.

In summary:
If current AI image generators are like a magician who pulls random rabbits out of a hat, MS-CustomNet is like a chef who follows your exact recipe. It takes the specific ingredients you provide, arranges them exactly how you want them on the plate, and ensures the final dish looks delicious and coherent. It gives you the power to be the director of your own visual story, with total control over who stands where and how they interact.

1. Problem Statement

While diffusion-based text-to-image generation has advanced significantly, multi-subject customization remains a critical challenge. Existing methods struggle to:

Preserve Identity: Maintain the distinct visual identity of multiple user-provided subjects simultaneously without "concept bleeding" or identity loss.
Control Composition: Provide explicit, user-defined control over the hierarchical structure (e.g., occlusion, layering, containment) and spatial relationships between subjects.
Limitations of Current Approaches:
- Optimization-based methods (e.g., DreamBooth) are computationally expensive and prone to overfitting.
- Encoder-based methods (e.g., CustomNet) often lack fine-grained control over multi-subject interactions.
- General multi-subject models (e.g., MS-Diffusion) rely on implicit prompt semantics, offering limited predictability for specific spatial arrangements (e.g., "Object A inside Object B").
- Data Scarcity: There is a lack of public datasets containing complex multi-subject interactions with explicit layout annotations.

2. Methodology: MS-CustomNet

The authors propose MS-CustomNet, a framework built upon a Latent Diffusion Model (LDM) designed for zero-shot integration of multiple subjects with explicit compositional control.

A. Dataset Construction: MSI Dataset

To address the data gap, the authors constructed the Multi-Subject Interaction (MSI) dataset derived from COCO:

Filtering: Images are filtered based on subject count ( $\alpha \ge 2$ ) and relative area prominence ( $\beta = 0.015$ ) to ensure salient interactions.
Reference Pool: A diverse pool of reference images is curated for each object category to prevent trivial copy-pasting.
Layout Generation: A composite layout map ( $M_L$ ) is generated by placing resized reference images onto a canvas based on COCO bounding boxes.
Hierarchical Ordering: An occlusion prior is applied to determine the compositing order (depth), ensuring smaller/foreground objects occlude larger/background objects, simulating realistic layering.
Final Structure: Each training tuple consists of: Reference images $\{x_{ref,k}\}$ , Text prompt $T$ , Layout map $M_L$ , and the target scene $y_s$ .

B. Model Architecture

The framework extends a pre-trained LDM (based on CustomNet) with specific conditioning mechanisms:

Category-Aware Subject Embeddings:
- Visual features are extracted via a CLIP image encoder.
- Category labels are encoded via a Category Encoder.
- These are concatenated and projected by a Category-Aware Projection Network ( $F_{proj}$ ) to create subject embeddings ( $f_{s,k}$ ). This ensures the model associates specific visual references with their semantic categories.
Layout Conditioning:
- The layout map $M_L$ is encoded by an AutoencoderKL to produce a conditioning tensor ( $c_L$ ).
- This tensor is concatenated to the noisy latent variable, providing explicit 2D position, scale, and layering guidance.
Text Conditioning: Standard CLIP text embeddings guide the background and scene semantics.

C. Training Strategies

Two novel strategies are employed to stabilize training and enhance performance:

Dual Stage Training (DST):
- Stage 1: Focuses on layout and interaction priming (learning placement and basic relationships).
- Stage 2: Focuses on identity and detail refinement (enhancing photorealism and feature fidelity) with a reduced learning rate.
Curriculum Learning on Subject Quantity (CLSQ):
- The model is trained progressively, starting with simpler compositions (fewer subjects) and gradually increasing the maximum number of subjects ( $K$ ) allowed in a batch. This prevents the model from being overwhelmed by complex multi-object interactions early in training.

3. Key Contributions

MS-CustomNet Framework: A novel architecture extending single-subject customization to the multi-subject domain, enabling explicit, user-defined control over compositional structure (layering/occlusion) and spatial placement.
MSI Dataset: A specialized dataset derived from COCO containing 14,537 complex multi-subject scenes with layout maps and reference pools, facilitating the training of relational semantics.
Advanced Training Regimens: Introduction of Dual Stage Training and Curriculum Learning on Subject Quantity to balance identity preservation with complex compositional learning.
Zero-Shot Capability: The system allows for the integration of multiple user-provided objects without per-subject fine-tuning, maintaining high fidelity and scene coherence.

4. Experimental Results

The method was evaluated on the MSI-Bench (100 multi-subject tasks) against state-of-the-art methods like IP-Adapter, CustomNet, $\lambda$ -ECLIPSE, and SSR-Encoder.

Quantitative Performance:
- Identity Preservation: Achieved a DINO-I score of 0.61 and CLIP-I of 0.79 in multi-subject tasks, outperforming or matching specialized single-subject methods.
- Positional Control: Achieved a YOLO-L score of 0.94, demonstrating superior accuracy in placing subjects at user-specified locations compared to baselines (e.g., SSR-Encoder scored 0.91).
- Object Count: High YOLO-Subj scores indicate accurate generation of the correct number and categories of objects.
Qualitative Performance:
- Ablation Studies: Showed that without layout maps ( $M_L$ ), spatial control failed; without DST, identity fidelity dropped; without CLSQ, background coherence suffered. The full model achieved the best balance.
- Complex Scenarios: Successfully generated scenes with specific hierarchical relationships (e.g., "a cake inside a bowl" vs. "behind a bowl," or stacked books with specific occlusion orders) which other methods failed to execute precisely.

5. Significance

Shift from Implicit to Explicit Control: MS-CustomNet moves beyond relying on the implicit semantics of text prompts to provide deterministic, user-directed control over the physical arrangement and interaction of multiple objects.
Bridging the Gap: It solves the "concept bleeding" and "identity loss" issues common in multi-subject generation while adding a layer of structural control previously unavailable in zero-shot frameworks.
Resource Contribution: The release of the MSI dataset provides a crucial resource for the community to train and evaluate models on complex compositional tasks, addressing a major bottleneck in the field.
Application Potential: The framework is highly relevant for applications requiring precise scene composition, such as advertising, game asset generation, and visual storytelling, where specific spatial relationships between objects are critical.