Diffusion Templates: A Unified Plugin Framework for Controllable Diffusion

Technical Summary: Diffusion Templates: A Unified Plugin Framework for Controllable Diffusion

1. Problem Statement

The current ecosystem of controllable diffusion models (e.g., ControlNet, LoRA, IP-Adapter, T2I-Adapter) is highly fragmented. While effective, these methods are typically developed as isolated, backbone-specific systems. This fragmentation leads to several critical bottlenecks:

• Incompatibility: Different methods use different training pipelines, parameter formats, and runtime hooks.
• Lack of Reusability: It is difficult to transfer a control capability from one base model (e.g., Stable Diffusion) to another (e.g., FLUX) without significant re-engineering.
• Compositionality Issues: Combining multiple controls (e.g., structural control + color adjustment + image editing) is a "handcrafted engineering problem." Integrating them often requires modifying the core denoising pipeline to resolve conflicts between competing conditioning pathways.

2. Methodology

The authors propose Diffusion Templates, a unified plugin framework that treats controllable capabilities as independent, reusable modules rather than backbone-specific modifications. The framework decouples base-model inference from capability injection through three core components:

• Template Cache (The Interface): Instead of prescribing a specific architecture, the framework defines a standardized intermediate interface. The most prominent carrier is the KV-Cache, which allows for sequence-level concatenation. However, the framework is extensible and can also support other formats like LoRA weights.
• Template Model (The Plugin): A modular unit that maps task-specific inputs (e.g., depth maps, scalar values, or reference images) into a Template Cache. Each model exposes two standardized interfaces: process_inputs (no-gradient preprocessing) and forward (gradient-based computation for training/inference).
• Template Pipeline (The Orchestrator): A unified runtime that loads one or more Template models, executes them to produce caches, merges those caches (e.g., via concatenation), and injects them into the base diffusion model.

Training Strategy: The framework uses a side-branch approach where the base model remains frozen, and only the new Template branches are optimized using the original pretraining loss.

3. Key Contributions

• A Unified Systems Abstraction: Introduced a plugin-based architecture for diffusion models, inspired by the modularity seen in LLM agent/tool-use frameworks.
• Decoupling of Layers: Successfully factorized controllable generation into two distinct layers: capability construction (Template Model) and capability consumption (Template Pipeline).
• A Diverse Model Zoo: Developed and released a wide array of Template models trained on the FLUX.2 backbone, demonstrating the framework's versatility across different input types (structural, scalar, and image-conditioned).

4. Results and Case Studies

The authors demonstrated the framework's effectiveness through a comprehensive "Model Zoo" covering various tasks:

• Structural Control: Depth, outline, pose, and normal maps via KV-Cache injection.
• Attribute Control (Scalar): Fine-grained control over brightness, color (RGB channels), sharpness (edge density), and age (human portrait semantics).
• Image-Conditioned Tasks:
  • Image Editing: Achieving 1.8× speedup compared to standard editing methods.
  • Super-Resolution: Recovering high-frequency details via template-based upscaling.
  • Content Reference: An "Image-to-LoRA" model that converts a reference image into a LoRA for style or identity transfer.
  • Local Inpainting: Combining learned model control with pipeline-level masking for precise edits.
• Aesthetic Alignment: Using LoRA as a carrier to align generations with human preference scales.
• Template Fusion: Successfully demonstrated that multiple templates (e.g., Super-Resolution + Sharpness + Aesthetic Alignment) can be composed simultaneously without manual pipeline redesign.

5. Significance

Diffusion Templates represents a shift from model-centric control to system-centric control. By providing a stable "contract" between plugins and backbones, the framework:

1. Lowers the barrier to entry for developing new control methods.
2. Enables seamless compositionality, allowing users to stack diverse capabilities (structural, stylistic, and attribute-based) effortlessly.
3. Future-proofs the ecosystem, as new, more powerful diffusion backbones (like video models) can adopt these templates if they adhere to the standardized cache interface.