FrescoDiffusion: 4K Image-to-Video with Prior-Regularized Tiled Diffusion

1. Problem Statement

The paper addresses the challenge of generating ultra-high-resolution (4K) videos from single images using Image-to-Video (I2V) diffusion models. While current state-of-the-art I2V models perform well at native resolutions (typically 480p–1080p), they struggle with 4K inputs due to memory constraints and computational limits.

Existing solutions face two main failure modes when scaling to 4K:

1. Downscaling: Resizing a 4K image to the model's native resolution results in a loss of fine-grained details and structural complexity.
2. Tiled Denoising: Processing the image in overlapping tiles (patches) preserves local detail but often leads to global inconsistency. This manifests as "cross-tile drift," visible seams, and a breakdown of the global layout, particularly in complex scenes like frescos (large-scale artworks with multiple distinct characters, objects, and sub-scenes).
3. Video Super-Resolution (VSR): Standard VSR methods cannot hallucinate new semantic content or complex scene-level dynamics that were not present in the low-resolution input; they are limited to texture refinement.

The specific difficulty in the "fresco" setting is that different regions of the image have different roles: some are static (background/architecture), while others are dynamic (characters/movement). A uniform approach fails to balance the need for global coherence with local creativity.

2. Methodology: FrescoDiffusion

FrescoDiffusion is a training-free method that extends existing I2V models to 4K resolution by combining tiled denoising with a precomputed latent prior. The core innovation is a novel loss function that fuses local tile predictions with a global reference.

A. Core Pipeline

1. Global Latent Prior Generation:

   • The input 4K image is resized to the model's native resolution (e.g., 480×832).
   • The model generates a low-resolution video from this thumbnail.
   • This low-res video is upsampled in the latent space (using trilinear interpolation) to match the 4K canvas size, creating a global reference trajectory ($x_{\text{prior}}$). This prior captures the long-range temporal and spatial structure.

2. Prior-Regularized Tile Fusion:

   • The 4K latent canvas is processed in overlapping tiles.
   • Instead of simply averaging tile predictions (as in MultiDiffusion), the method minimizes a new energy loss function, $\ell_{\text{FD}}$, at every timestep:
     $$\ell_{\text{FD}} = \underbrace{\|\sqrt{\lambda} \odot [(x_t^{4K} - \sigma_t y^\star) - x_{\text{prior}}]\|_2^2}_{\text{Prior Regularization}} + \underbrace{\ell_{\text{MD}}(y^\star; t)}_{\text{Tile Consistency}}$$
   • $\ell_{\text{MD}}$: Ensures consistency between overlapping tiles (standard MultiDiffusion loss).
   • $\ell_{\text{prior}}$: Penalizes deviation from the upsampled global prior.
   • Closed-Form Solution: Because the loss is strictly convex, the optimal fused velocity field ($y_{\text{FD}}$) has a closed-form solution:
     $$y_{\text{FD}} = \frac{\sigma_t \cdot \lambda \odot (x_t^{4K} - x_{\text{prior}}) + \sum w_i \odot y_i}{\sigma_t^2 \cdot \lambda + \sum w_i}$$
     This allows for efficient fusion without iterative optimization.

3. Spatio-Temporal Scheduling (Regional Control):

   • The regularization strength $\lambda$ is not static. It is designed to control the trade-off between creativity (new details) and consistency (adhering to the prior).
   • Temporal Gating: $\lambda$ decreases over time (cosine schedule). High $\lambda$ in early steps ensures global structure; low $\lambda$ in later steps allows the model to add fine details.
   • Spatial Gating (R-FrescoDiffusion): A spatial activity map ($A(p)$) distinguishes between active regions (foreground, characters) and static regions (background).
     • Background: High $\lambda$ for longer, keeping the structure stable.
     • Foreground: $\lambda$ drops earlier, allowing the model to animate characters and introduce motion freely.

3. Key Contributions

1. FrescoDiffusion Algorithm: A novel, training-free framework for 4K I2V generation that solves the global coherence vs. local detail trade-off via prior-regularized tile fusion.
2. Regional Regularization: Introduction of a spatial gating mechanism that allows independent control over motion in active zones vs. static backgrounds, crucial for complex multi-scene images.
3. FrescoArchive Dataset: A new dataset of 371 complex, multi-scene 4K images specifically curated for evaluating fresco-to-video tasks, addressing the lack of suitable benchmarks for this domain.
4. Efficiency: The method is computationally efficient, outperforming baselines in runtime while maintaining high quality.

4. Experimental Results

The authors evaluated FrescoDiffusion on the FrescoArchive dataset and the standard VBench-I2V dataset, comparing against baselines like MultiDiffusion, DemoFusion, and DynamicScaler.

• Qualitative Performance:

  • Baselines (MultiDiffusion, DynamicScaler) suffer from structural drift, visible seams, or loss of global layout.
  • FrescoDiffusion preserves the global layout of the fresco while animating specific regions naturally.
  • R-FrescoDiffusion (with regional gating) shows superior preservation of background structure compared to the standard version.

• Quantitative Performance:

  • User Study: In a study with 47 participants, R-FrescoDiffusion was preferred over DynamicScaler and MultiDiffusion by 84–93% of users. It also significantly outperformed DemoFusion (69% preference).
  • Automatic Metrics: On FrescoArchive, FrescoDiffusion achieved the highest average scores in Subject Consistency, Motion Smoothness, and Prior Alignment.
  • Efficiency: The method is 45% faster than DemoFusion and significantly faster than DynamicScaler, thanks to the closed-form fusion update and reduced-step sampling (6 steps).

• Ablation Studies:

  • Removing the temporal schedule or spatial gating degrades performance.
  • The method allows users to navigate a Pareto frontier between creativity and prior similarity by adjusting the regularization strength ($\lambda_{\text{base}}$) and gating thresholds.

5. Significance and Impact

• Bridging the Resolution Gap: FrescoDiffusion provides a practical solution for animating ultra-high-definition content without requiring massive computational resources or retraining large foundation models.
• Artistic and Creative Applications: It is specifically tailored for "fresco" style content (murals, complex paintings, panoramic scenes), enabling new possibilities in digital art restoration, animation, and immersive media.
• Controllability: The explicit control over the trade-off between adhering to the input structure and generating new motion offers a new degree of freedom for generative AI users.
• Open Science: The authors commit to open-sourcing the algorithm and the FrescoArchive dataset, fostering further research in high-resolution video generation.

In summary, FrescoDiffusion represents a significant step forward in making 4K video generation accessible and coherent, particularly for complex, multi-scene imagery, by leveraging a smart fusion of low-resolution priors and high-resolution tiled denoising.