Jano: Adaptive Diffusion Generation with Early-stage Convergence Awareness

Jano is a training-free framework that accelerates Diffusion Transformers by identifying heterogeneous convergence patterns in early denoising stages and applying adaptive token scheduling to achieve up to 2.4x speedup while preserving generation quality.

The Gist

Imagine you are an artist painting a massive, incredibly detailed mural. You have a magical brush that can paint the whole thing, but it's slow. To paint a single 5-second video clip, this brush takes over an hour to work its magic.

The problem is that the brush treats every part of the painting the same way. It spends the same amount of time and energy painting a complex, detailed face as it does painting a plain, empty blue sky.

JANO is a new "smart assistant" for this artist. It doesn't teach the artist a new way to paint (it's "training-free"), but it tells the artist where to focus their energy and where they can relax.

Here is how JANO works, broken down into simple concepts:

1. The "Eagle Eye" Observation (Early-Stage Complexity Recognition)

Usually, artists only know how complex a part of the painting is once it's finished. JANO is different. It looks at the very first few strokes (the "early steps") and instantly figures out which parts of the image are going to be simple and which will be complex.

• The Analogy: Imagine you are baking a cake. A normal baker bakes the whole cake for the same amount of time. JANO is like a baker who smells the batter after 5 minutes and says, "The chocolate part needs 40 more minutes, but the vanilla part is already done!"
• How it works: JANO looks at the "noise" (the blurry starting point) and predicts which areas will need a lot of attention (like a face or a moving car) and which will settle down quickly (like a wall or the sky).

2. The "Traffic Light" System (Convergence Levels)

Once JANO knows what's what, it sorts every tiny piece of the image (called "tokens") into three traffic light categories:

• 🔴 Red (Static): These are the boring, simple parts (like a blue sky). They are "done" very early. JANO tells the computer: "Stop working on this! Just keep the last version you made."
• 🟡 Yellow (Moderate): These are okay, but need a little more work. They get checked occasionally.
• 🟢 Green (Active): These are the complex, busy parts (like a person's face or a moving animal). They need the full attention of the computer for the entire process.

3. The "Smart Relay" (Adaptive Generation)

This is where the magic happens. Instead of the computer trying to paint the entire image at every single step (which is like trying to run a marathon while carrying a heavy backpack), JANO uses a smart relay system.

• The Old Way: The computer runs a full marathon, calculating every single pixel for every single step, even the ones that don't need changing.
• The JANO Way: The computer only runs the marathon for the "Green" (complex) parts. For the "Red" (simple) parts, it just grabs the finished result from a shelf (a "cache") and pastes it back in.

Think of it like a construction crew building a house.

• Without JANO: The whole crew is constantly re-painting the front door, even though it's already perfect, while also trying to build the roof.
• With JANO: The crew realizes the front door is done. They send the painters to the roof (the hard part) and just leave the door alone until the very end. They save massive amounts of time.

4. The "Magic Memory" (KV Cache)

You might ask: "If they stop working on the simple parts, how does the computer remember what the whole picture looks like?"

JANO uses a special memory trick called KV Cache. It's like having a sticky note on the wall. When the computer stops working on the "Red" parts, it writes down the current state on a sticky note. When it needs to look at the whole picture again to paint the "Green" parts, it just sticks the note back on the wall. This way, the computer doesn't have to re-calculate the simple parts; it just remembers them.

The Result: Faster, Not Worse

The paper tested JANO on the most advanced AI video and image generators available today.

• Speed: It made the generation 2 to 2.4 times faster. (A task that took 1 hour now takes about 25 minutes).
• Quality: Surprisingly, the quality didn't drop. In fact, because the computer focused more energy on the important parts (like faces), the main subjects sometimes looked better than before, even if the background was slightly less detailed.

Summary

JANO is like a smart project manager for AI art. Instead of making the AI work hard on everything equally, it identifies the "easy" parts early on, tells the AI to stop wasting time on them, and focuses all the computing power on the "hard" parts. The result is a super-fast AI that still creates beautiful, high-quality videos and images.

Technical Summary

1. Problem Statement

Diffusion models, particularly Diffusion Transformers (DiTs), have revolutionized generative AI but suffer from severe computational inefficiencies.

• The Bottleneck: Unlike Large Language Models (LLMs) that use causal attention with KV Cache, DiTs require full-sequence attention across all tokens at every timestep. This results in quadratic computational complexity relative to sequence length, making high-resolution video generation (e.g., 720p for 5 seconds) extremely slow (often >1 hour on a single A100).
• Limitations of Existing Methods: Current training-free acceleration techniques rely on feature caching (reusing intermediate results). However, these methods typically use uniform, content-agnostic strategies. They either cache too coarsely (failing to maximize speedup) or too finely (introducing high overhead for per-step monitoring). They fail to account for the fact that different regions of an image or video converge at different rates.

2. Key Insight & Motivation

The authors observe a heterogeneous convergence pattern in diffusion generation:

• Non-Uniformity: Different spatial regions require different numbers of denoising steps to stabilize.
  • Simple regions (e.g., uniform backgrounds) converge quickly.
  • Complex regions (e.g., facial features, detailed textures) require sustained computation.
• Correlation: There is a strong statistical correlation between regional content complexity and convergence speed.
• The Challenge: While this correlation is clear in the final output, it is difficult to predict during the early noisy stages of generation. Traditional methods like FFT fail to extract meaningful complexity features from early-stage latents dominated by noise.

3. Methodology: JANO Framework

JANO is a training-free framework that leverages early-stage complexity recognition to enable adaptive, region-aware generation. It consists of three main components:

A. Early-Stage Complexity Recognition

Instead of relying on noisy latent values directly, JANO derives a theoretical metric based on Flow Matching (FM) principles.

• Theoretical Basis: In Flow Matching, the model predicts a velocity field $v_\theta$. The authors prove that the difference in velocity fields between two points (trajectories) remains approximately constant if those points are destined to converge to similar final states.
• Distance Metric: They define a distance metric $D_{A,B}(t)$ based on the difference between the predicted velocity difference and the initial noise difference. Large values indicate divergent trajectories (complex regions), while small values indicate convergent trajectories (simple regions).
• Block-wise Analyzer: To minimize error accumulation, the method analyzes the second-order differences of temporal and spatial gradients over the first $\sim10\%$ of sampling steps (the "warm-up" phase). This generates a Complexity Score for each 3D block (frames $\times$ height $\times$ width).

B. Convergence-Aware Classification

Based on the complexity scores, tokens are categorized into three levels:

1. Level 1 (Static): Regions that converge early (e.g., backgrounds).
2. Level 2 (Moderate): Regions requiring intermediate steps.
3. Level 3 (Active): Regions needing the full denoising process (e.g., complex subjects).

C. Adaptive Generation with Interleaved Pipeline & KV Cache

JANO implements a three-phase generation pipeline to optimize resource allocation:

1. Warm-up Phase: Analyzes early outputs to generate a Convergence Level Map.
2. Interleaved Generation Phase:
   • Level 1 (Static): Activated at the lowest frequency (frozen for long periods).
   • Level 2 (Moderate): Activated at intermediate frequency.
   • Level 3 (Active): Continuously updated.
   • KV Cache Mechanism: To maintain global sequence dependency in DiTs while skipping computation for frozen tokens, JANO introduces a specialized KV Cache. It stores Key-Value pairs for frozen tokens and concatenates them with active tokens during attention computation. This avoids costly positional restoration, making the cache retrieval virtually overhead-free.
3. Cool-down Phase: A final refinement stage to ensure smooth transitions and fix artifacts caused by sparse updates.

4. Key Contributions

1. Theoretical Characterization: Identified and mathematically characterized the heterogeneous nature of convergence patterns in diffusion generation, proving that early-stage complexity can predict regional convergence requirements.
2. JANO Framework: Proposed a lightweight, training-free system integrating early-stage complexity recognition with an adaptive token activation strategy.
3. Efficient Pipeline: Designed an interleaved generation pipeline with optimized KV Cache management that allows for selective computation without breaking the full-attention mechanism of DiTs.

5. Experimental Results

The framework was evaluated on state-of-the-art models: Flux-1-dev (Image) and Wan2.1 (Video, 1.3B and 14B variants).

• Speedup:
  • Achieved an average 2.0× speedup, with peaks up to 2.4×.
  • Specific results:
    • Flux-1: 1.86× speedup (Latency reduced from 11.37s to 6.12s).
    • Wan-1.3B: 2.25× speedup.
    • Wan-14B: 2.30× speedup (Latency reduced from 1790s to 778s).
• Quality Preservation:
  • JANO maintained or even improved generation quality compared to baselines (ToCa, PAB, TeaCache).
  • Metrics (PSNR, SSIM, LPIPS, VBench) showed JANO outperformed other methods, particularly in Subject Consistency and Image Quality.
  • Notably, by reducing computation in simple backgrounds, JANO sometimes enhanced the salience of the main subject, leading to better consistency scores despite slight background simplification.
• Accuracy: The early-stage complexity recognition achieved a median accuracy of 0.73 with only 6-7 warm-up steps, significantly outperforming FFT-based baselines (0.13–0.25).

6. Significance

• Paradigm Shift: JANO challenges the conventional assumption that diffusion generation requires uniform processing for all tokens. It proves that content-aware, adaptive computation is viable and highly effective.
• Scalability: The method is particularly effective for large-scale models (like Wan-14B) where transformer computation dominates, offering a practical solution for real-time and interactive high-fidelity content generation.
• Training-Free: Unlike methods requiring fine-tuning, JANO is plug-and-play, making it immediately applicable to existing state-of-the-art models.

In summary, JANO provides a robust, theoretically grounded solution to the computational bottleneck of Diffusion Transformers by intelligently skipping redundant calculations in simple regions while focusing resources on complex, dynamic areas.