RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformers

The paper proposes RelaCtrl, a relevance-guided framework that optimizes control signal integration in Diffusion Transformers by dynamically tailoring layer configurations and introducing a Two-Dimensional Shuffle Mixer, achieving superior performance with only 15% of the parameters and computational complexity compared to PixArt-delta.

The Gist

Imagine you are a master chef (the Diffusion Transformer) trying to cook a perfect meal based on a customer's order (the text description). Sometimes, the customer gives you extra instructions: "Make it spicy," "Use only organic vegetables," or "Arrange it like a flower." These extra instructions are your control signals.

In the past, to follow these extra instructions, chefs would hire a whole new team of sous-chefs just to double-check every single step of the cooking process, from chopping onions to plating the dessert. This was effective, but it was expensive, slow, and wasted a lot of energy. They were checking every step, even the ones where the customer's extra instructions didn't really matter.

This paper, RelaCtrl, introduces a smarter way to cook. It's like hiring a smart, efficient sous-chef who knows exactly when and how to help, without wasting time on things that don't need attention.

Here is how it works, broken down into three simple ideas:

1. The "Relevance Score": Knowing When to Speak Up

The researchers discovered that not all steps in the cooking process are equally important for following the customer's extra rules.

• The Old Way: The sous-chef shouted instructions at the beginning, the middle, and the end of the cooking process, even if the chef was already doing the right thing.
• The RelaCtrl Way: They ran a test to see when the extra instructions mattered most. They found that the instructions were most critical during the middle stages of cooking (like seasoning the sauce). At the very beginning (chopping) and the very end (plating), the instructions mattered less.
• The Result: Instead of shouting at every step, the smart sous-chef only speaks up at the 11 most critical moments. This saves a huge amount of energy and time, yet the meal still turns out perfect.

2. The "Two-Dimensional Shuffle Mixer" (TDSM): The Efficient Helper

Even when the sous-chef does speak up, the old method was clumsy. It used a giant, heavy tool to mix ingredients, which took up a lot of space in the kitchen.

• The Old Tool: A massive, slow mixer that tried to stir every single ingredient with every other ingredient at once.
• The New Tool (TDSM): The researchers built a lightweight, magical shaker.
  • Imagine you have a deck of cards (the ingredients). Instead of looking at the whole deck, you randomly pick a few cards, shuffle them around, mix them, and then put them back in their original order.
  • Because you shuffled them randomly, the cards that were far apart in the deck can now "talk" to each other. This allows the sous-chef to understand the big picture without needing a giant, heavy machine.
  • This new tool does the same job as the giant mixer but is much smaller and faster.

3. The "Smart Placement": Putting the Right Tools in the Right Spots

Finally, the paper explains that the strength of the sous-chef's help should change depending on the moment.

• In the most critical moments (the middle of cooking), the sous-chef uses a strong, detailed plan (more computing power) to ensure the dish is perfect.
• In the less critical moments, the sous-chef uses a simpler, lighter plan.
• This ensures that no energy is wasted on steps that don't need it, and no energy is saved on steps that do.

The Big Picture: Why This Matters

Before this paper, adding "control" to AI image generators was like adding a heavy backpack to a runner. It made the runner slower and tired them out quickly.

RelaCtrl is like giving that runner a lightweight, aerodynamic suit.

• It's faster: It uses about 85% less extra computing power than previous methods.
• It's cheaper: It needs far fewer "parameters" (which you can think of as the size of the brain needed to do the job).
• It's just as good: The images it creates are just as high-quality and follow the instructions just as well as the heavy, slow methods.

In short: RelaCtrl teaches AI how to be a smart, efficient worker that knows exactly when to pay attention and how to do the job with the least amount of effort possible, without sacrificing the quality of the final result.

Technical Summary

1. Problem Statement

Diffusion Transformers (DiTs) have become the state-of-the-art architecture for text-to-image and text-to-video generation due to their scalability. However, integrating controllable generation (e.g., using edge maps, depth, or segmentation) into DiTs presents two critical challenges:

• High Computational Overhead: Existing methods, such as PixArt-δ, achieve control by duplicating a large portion of the network (e.g., the first 13 Transformer blocks). This results in a ~50% increase in parameters and computational complexity, making training and inference expensive.
• Inefficient Resource Allocation: Current approaches apply uniform control mechanisms across all layers, ignoring the fact that different layers in a DiT have varying levels of relevance to control signals. Blindly copying all layers leads to redundant computations in layers where control information has little impact on the final output.

2. Methodology

The authors propose RelaCtrl, a framework designed to optimize control signal integration through relevance analysis and architectural efficiency. The methodology consists of three core components:

A. ControlNet Relevance Score (CRS) Analysis

To address inefficient resource allocation, the authors first quantified the importance of each layer in a DiT-ControlNet architecture.

• Experiment: They trained a model with all blocks duplicated and systematically skipped individual control blocks during inference.
• Metrics: They measured the impact on generation quality using Fréchet Inception Distance (FID) and control accuracy using Hausdorff Distance (HDD).
• Finding: The relevance of control information follows a "rise-and-fall" trend. The most critical layers for control are concentrated in the early-to-middle layers (e.g., blocks 5–7), while deeper layers show diminishing returns. This contrasts with the assumption that simply copying the first $N$ blocks is optimal.

B. Relevance-Guided Allocation Strategy

Based on the CRS analysis, RelaCtrl employs a dynamic placement strategy:

• Selective Insertion: Instead of duplicating the first 13 blocks, control blocks are inserted only at the top 11 positions ranked by relevance.
• Efficiency: This reduces the number of control blocks by ~15% while maintaining performance comparable to the full duplication method.

C. Relevance-Guided Lightweight Control Block (RGLC) & TDSM

To further reduce the parameter count and computational cost of the selected control blocks, the authors replaced the standard Self-Attention and Feed-Forward Network (FFN) with a novel module:

• Two-Dimensional Shuffle Mixer (TDSM):
  • Mechanism: TDSM replaces the token mixer (Self-Attention) and channel mixer (FFN) with a single operation. It performs random channel selection followed by random token shuffling within 3D space.
  • Local Attention with Global Reach: After shuffling, local self-attention is computed. Because tokens are randomly shuffled, tokens from distant parts of the original sequence can end up in the same local window, enabling non-local modeling within a computationally cheap local attention framework.
  • Inverse Recovery: To ensure information is not lost during shuffling, an inverse shuffle operation is applied after attention to restore the original token order.
• Adaptive Capacity: The number of channel groups in TDSM is regulated based on the layer's relevance score. High-relevance layers use fewer groups (larger feature dimensions) to enhance modeling capability, while low-relevance layers use more groups to save computation.

3. Key Contributions

1. Relevance Discovery: The paper is the first to systematically analyze and demonstrate that control information relevance in DiTs is non-uniform, peaking in the shallow-to-middle layers.
2. RelaCtrl Framework: A strategy that optimizes control block placement based on relevance scores, reducing the number of required control blocks without sacrificing quality.
3. TDSM Architecture: A novel, lightweight mixer that efficiently replaces Self-Attention and FFN. It achieves non-local modeling capabilities through random shuffling, significantly reducing parameters and FLOPs.
4. Efficiency-Performance Trade-off: The method achieves superior performance with only 15% of the additional parameters and computational complexity compared to the baseline PixArt-δ.

4. Experimental Results

The authors evaluated RelaCtrl on the COCO validation set across four conditional tasks: Canny edges, HED (Holistically-Nested Edge Detection), Depth, and Segmentation.

• Quantitative Performance:
  • Control Accuracy: RelaCtrl achieved the best or near-best HDD (Hausdorff Distance) and mIoU scores across all tasks, outperforming SOTA methods like Uni-ControlNet, ControlNet-XS, and ControlNext.
  • Image Quality: It achieved superior FID and CLIP-Aesthetics scores compared to PixArt-δ and other baselines.
  • Text Consistency: High CLIP Scores confirmed strong alignment between the generated image and the text prompt.
• Efficiency Gains:
  • Parameters: RelaCtrl added only 45M parameters (7.4% increase over PixArt-α), compared to PixArt-δ's ~294M increase (48% increase).
  • Complexity: It added only 46.7 GFLOPs (8.6% increase), whereas PixArt-δ added ~270 GFLOPs (49% increase).
  • Inference: Inference time increased by only ~6.3%, demonstrating high practical efficiency.

5. Significance

RelaCtrl addresses a major bottleneck in the deployment of controllable Diffusion Transformers. By proving that not all layers are equally important for control and introducing a lightweight, non-local mixing mechanism, the paper enables:

• Cost-Effective Deployment: Making high-quality controllable generation feasible on consumer-grade hardware or in resource-constrained environments.
• Scalable Research: Providing a blueprint for efficient control in large-scale generative models, moving away from brute-force parameter duplication.
• Generalization: The relevance-guided strategy and TDSM design are shown to be effective across different DiT backbones (PixArt-α) and diverse control modalities (edges, depth, segmentation).

In summary, RelaCtrl represents a shift from "copying everything" to "copying smartly," leveraging layer-specific relevance and efficient architectural design to achieve state-of-the-art controllable generation with minimal computational overhead.