💻 computer science

Video-Rate Streaming Stylization on a Vision-Aware MLLM-Conditioned Edit Diffusion: Asymmetric Batched Inference on a Distilled UNet + MLLM Text Encoder

This paper presents a video-rate streaming stylization pipeline that overcomes the text-encoder bottleneck in MLLM-conditioned edit diffusion by combining a distilled 0.39B U-Net with a 2.13B Qwen3-VL encoder, utilizing asymmetric batched inference and fused graph optimizations to achieve up to 74.1 fps on consumer GPUs.

Original authors: Yoshiyuki Ootani

Published 2026-06-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yoshiyuki Ootani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical art studio where a robot painter can instantly turn any video into an oil painting. Usually, the slowest part of this process is the painter itself (the "U-Net"), who takes time to mix colors and apply brushstrokes.

However, this paper describes a new setup where the painter has been super-trained to work incredibly fast—so fast that they are no longer the bottleneck. Instead, the new "slowpoke" is the Art Director (a massive AI called an MLLM). Before the painter can start, the Art Director has to look at the video frame and the user's instructions (e.g., "make it look like a Van Gogh") and write a detailed brief. Because the Art Director is so heavy and complex, they hold up the whole line, even though the painter is ready to go.

The authors built a system to solve this specific problem: How do you keep the video flowing smoothly when the Art Director is slow, but the Painter is fast?

Here is how they did it, using three main tricks:

1. The "Two-Track" Assembly Line (Asymmetric Pipelining)

Imagine a factory with two conveyor belts running side-by-side.

The Main Belt: The fast Painter works on the current frame.
The Side Belt: The slow Art Director works on the next frame's instructions while the Painter is busy.

By running these two tasks at the same time on different "tracks" (CUDA streams), the system hides the Art Director's slowness. While the Painter is busy painting Frame 1, the Art Director is already reading the instructions for Frame 2. By the time the Painter finishes, the instructions for the next frame are ready. This keeps the line moving without stopping.

2. The "Group Brief" Strategy (Batched Inference)

The Art Director is slow, but they get faster if they process a group of people at once rather than one by one.

Instead of asking the Art Director to write a brief for just one frame, the system gathers a batch of 8 or 16 frames.
The Art Director writes one massive, efficient brief that covers all of them.
This "bulk discount" on time means the Art Director spends less time per frame, even though they are processing more data at once.

3. The "Smart Refresh" Schedule (Periodic Conditioning)

Sometimes, the instructions don't need to be rewritten for every single frame. If the video is just a person walking, the "oil painting style" doesn't need to be re-calculated every millisecond.

The system uses a "refresh schedule." It calculates the detailed style instructions once, then reuses that same instruction for several frames in a row.
It only stops to recalculate the instructions when the scene changes significantly or after a set number of frames.
This saves a huge amount of time because the system doesn't waste energy re-asking the Art Director the same question over and over.

The Results: A Fast, Smooth Stream

The authors tested this on a single consumer graphics card (an RTX 3090 Ti).

Speed: They achieved a steady 27 to 30 frames per second (FPS). This is fast enough to watch a video in real-time without stuttering.
Latency: There is a slight delay (about 0.5 to 1 second) because the system has to buffer a few frames to make the "Two-Track" system work, but the video plays smoothly once it starts.
Quality: The system works well on videos it hasn't seen before (like different types of dance or parkour videos) and can handle different art styles, though it struggles a bit with very complex, high-contrast patterns like comic book dots.

What They Tried (and Failed)

The paper also lists a few ideas that didn't work, acting as "cautionary tales" for others:

Mixing old paint: Trying to mix the previous frame's paint directly into the new one made the video turn into a blurry, single-color blob.
Skipping the painter: Trying to guess the next frame by just warping the old one (skipping the painter entirely) resulted in a shaky, low-quality video.
Cutting corners: Removing parts of the Art Director's instructions to save time actually made the video worse, proving those instructions were necessary.

The Bottom Line

This paper isn't about making the robot painter faster; it's about reorganizing the factory so the slow Art Director doesn't stop the fast Painter. By running tasks in parallel, grouping instructions, and reusing them wisely, they managed to stream high-quality video stylization in real-time on a single home computer.

Technical Summary: Video-Rate Streaming Stylization on a Vision-Aware MLLM-Conditioned Edit Diffusion

Problem Statement
Real-time video translation using diffusion models has traditionally relied on text-conditioned image diffusion stacks where the U-Net denoiser is the per-frame computational bottleneck, and the text encoder is a lightweight CLIP-class network (~125M parameters). However, aggressive distillation of the denoiser (reducing it to 4-step or 1-step students) inverts this bottleneck. In "vision-aware" edit diffusion, where the system must process both a source image and a textual instruction, the text encoder is replaced by a heavy Multimodal Large Language Model (MLLM), such as Qwen3-VL (2.13B parameters).

In this "Small-Distilled-Denoiser plus Heavy-MLLM-Encoder" (SDD-MTE) regime, the MLLM text encoder becomes the critical path. Standard streaming techniques designed for denoiser-dominated pipelines fail here because:

Encoder Latency: The MLLM encoder is significantly larger than the denoiser and contains intrinsic CPU-GPU synchronization points (e.g., in rotary position embeddings) that prevent simple single-stream dispatch.
Temporal Consistency Overhead: Existing temporal adapters like ControlNet-LLLite rely on Python branches and tensor-identity checks that break torch.compile graph capture, causing naive implementations to run ~3.5× slower than the base U-Net.
Numerical Instability: Wide-range MLLM outputs (std=37, max=1160) overflow FP16 attention mechanisms, rendering TensorRT paths (relied upon by systems like StreamDiffusion) unavailable.

The paper addresses the challenge of achieving video-rate streaming throughput on a single consumer GPU within this inverted bottleneck regime.

Methodology
The authors propose a streaming pipeline built around three specific engineering mechanisms designed for the SDD-MTE regime, instantiated on a 0.39B distilled U-Net (DreamLite-mobile) paired with a 2.13B Qwen3-VL encoder.

Asymmetric Side-Stream / Main-Stream Pipelining:
- The pipeline splits into two concurrent CUDA streams. The Main Stream handles the compiled U-Net denoising and VAE decoding for the current batch ( $n$ ).
- The Side Stream concurrently processes the MLLM text encoding (TE) and VAE encoding for the next batch ( $n+1$ ).
- This design hides the MLLM encoder's intrinsic synchronization latency behind the U-Net computation.
- Batch Amortization: The system buffers $B$ frames (e.g., $B=8$ or $16$) before each pipeline call. Since the MLLM encoder scales sub-linearly with batch size (sub-linearity $s \approx 0.226$ at $B=8$ ) while the compiled U-Net scales near-linearly ( $s \approx 0.77$ ), batching significantly reduces the per-frame TE cost without proportionally increasing U-Net cost.
- Static-Prompt Caching (Optional): For streams with fixed instructions, the system can encode a single representative frame and broadcast its key/value tensors to the rest of the batch, or refresh the encoding only every $N$ batches.
Compile-Friendly ControlNet-LLLite Reformulation:
- To enable torch.compile optimization, the authors refactor the ControlNet-LLLite adapter to remove Python branches and tensor-identity changes that previously forced Dynamo recompilation on every frame.
- Key changes include registering condition embeddings as fixed-size buffers with in-place updates, removing training-only paths (dropout, dtype checks), and exposing a dedicated forward_inference method.
- This allows the entire U-Net + adapter stack to be folded into a single fused graph, achieving a 4.75× speedup over the eager reference implementation.
Periodic Conditioning Refresh and Hook Subset:
- Refresh Schedule: Instead of recomputing the LLLite conditioning embedding (derived from the warped previous frame) for every batch, the system rebuilds it every $N$ batches (e.g., $N=8$ ). This amortizes the cost of the conditioning CNN, which dominates the LLLite path's wall time.
- Hook Subset: The full 108-hook adapter is reduced to a subset of 38 hooks located only in the down_blocks. This reduces the conditioning encoder workload by ~65% while recovering spatial sharpness lost when using all 108 hooks (which tend to produce a "smooth purple" artifact).
- Async Flow: The Farneback optical flow calculation is threaded and kicked asynchronously during prefetch to overlap with the next side-stream slot.

Key Contributions

System Architecture: A novel asymmetric pipelining strategy that treats the MLLM encoder as a separate, concurrent workload to overcome the inverted bottleneck in SDD-MTE regimes.
Compiler Optimization: A refactored ControlNet-LLLite implementation compatible with torch.compile, enabling fused graph execution for temporal adapters.
Efficiency Mechanisms: A combination of batched TE amortization, periodic conditioning refresh, and hook subset selection that collectively reduce per-frame latency without significant quality degradation.
Empirical Validation: Comprehensive evaluation on consumer hardware (RTX 3090 Ti, 4090, 5090) demonstrating sustained throughput and temporal consistency across in-domain and cross-dataset scenarios.

Results
Evaluated on a single RTX 3090 Ti at 512×512 resolution with a batch size of $B=8$ and 1-step inference ( $K=1$ ):

Throughput: The pipeline sustains 27.4 FPS (p50 latency ~0.5s) and 29.6 FPS at $B=16$ (p50 latency ~1.0s) over a 480-frame run.
Hardware Scaling: Performance scales to 54.9 FPS on RTX 4090 and 74.1 FPS on RTX 5090.
Quality: The "champion" configuration (down_blocks hook subset + periodic refresh) achieves a warping error ( $\epsilon_w$ ) of 18.34, a 20% reduction in flicker compared to a no-LLLite baseline (22.97), while maintaining spatial detail (Sobel score 4.74 vs. 5.10 baseline).
Generalization: The released temporal adapter generalizes to 19 unused DAVIS-2017 sequences and 15 clips from 7 non-DAVIS sources with performance metrics within the in-clip noise of the in-domain champion.
Prompt Generalization: The model handles unseen style prompts (e.g., Van Gogh) well, though performance degrades for styles with high-frequency textures outside the training manifold (e.g., comic halftone).

Significance and Claims
The paper positions itself as a systems study rather than a benchmark superiority claim. It explicitly states that it does not claim superiority over StreamDiffusion-class systems (which use CLIP encoders) but locates its numbers against a same-stack re-run of StreamDiffusion to provide context.

The primary significance lies in demonstrating that video-rate streaming is achievable on a single consumer GPU even when the text encoder is a heavy MLLM, provided the system architecture is redesigned to accommodate the inverted bottleneck. The authors emphasize that standard denoiser-acceleration techniques are insufficient for this regime and that the proposed mechanisms (asymmetric pipelining, compile-friendly adapters, and amortized conditioning) are necessary to unlock real-time performance.

The paper also serves as a guide for practitioners by documenting negative results (e.g., naive latent mixing, TensorRT FP16 failures with MLLM outputs, and the pitfalls of using warping error alone as a metric for smoothing artifacts) to prevent common engineering dead ends in this specific domain.