🤖 AI

Tiled Prompts: Overcoming Prompt Misguidance in Image and Video Super-Resolution

The paper proposes "Tiled Prompts," a unified framework for image and video super-resolution that generates tile-specific text prompts to overcome the localized guidance errors and hallucinations caused by using a single global caption in latent tiling pipelines, thereby significantly improving perceptual quality and fidelity with minimal overhead.

Original authors: Bryan Sangwoo Kim, Jonghyun Park, Jong Chul Ye

Published 2026-04-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bryan Sangwoo Kim, Jonghyun Park, Jong Chul Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "One-Size-Fits-All" Map

Imagine you are a master painter hired to restore a tiny, blurry, low-resolution photo of a massive city street into a crystal-clear, 4K masterpiece.

To do this, you can't paint the whole street at once; your canvas is too small. So, you decide to paint the street tile by tile. You take a small square of the image, paint it, move to the next square, and repeat until the whole street is done.

The Old Way (Global Prompt):
In the past, to help you paint, the client gave you one single sentence describing the whole street.

The Prompt: "A busy city street with a blue sky, a red bus, a coffee shop, and a dog."

The Problem:
When you are painting the top-left tile, which only shows a patch of sidewalk, the prompt tells you to paint a "blue sky" and a "red bus." You get confused!

Error of Omission: The prompt didn't mention the specific "cracked brick" pattern on this specific patch of sidewalk. You miss the detail.
Error of Commission: The prompt tells you to paint a "red bus," but there is no bus in this tile. You might accidentally paint a ghost bus or a weird red smudge because your brain is trying to follow instructions that don't fit the spot you are working on.

This confusion is called "Prompt Misguidance." The instructions are too broad for the tiny piece of the puzzle you are currently holding.

The Solution: The "Local Guide" System (Tiled Prompts)

The authors of this paper, Bryan Sangwoo Kim and his team, came up with a brilliant fix called Tiled Prompts.

Instead of giving you one sentence for the whole street, they hire a smart AI assistant (a Vision-Language Model) to stand next to you at every single tile.

How it works:

Zoom In: Before you paint a specific tile, the AI looks only at that tiny, blurry square.
Write a New Prompt: The AI instantly writes a brand-new, hyper-specific sentence just for that square.
- Tile 1 (Sidewalk): "Cracked gray concrete with a small puddle reflecting a streetlamp."
- Tile 2 (Bus): "The shiny red side of a bus with the number '42' clearly visible."
- Tile 3 (Sky): "Clear blue sky with a single white cloud."
Paint: You now have perfect instructions for exactly what is in front of you.

The Result:

No Confusion: You don't try to paint a bus on the sidewalk.
More Detail: You don't miss the "cracked concrete" because the AI specifically told you to look for it.
Consistency: When you stitch all the tiles back together, the whole image looks sharp, logical, and realistic.

Why This Matters for Videos

The paper also applies this to videos (moving pictures). This is even harder!

Imagine a video of a car driving down a street.

The Old Way: The AI gives you one prompt for the whole video: "A car driving."
The Problem: In frame 1, the car is far away. In frame 10, the car is close, and you can see the license plate. In frame 20, the car turns a corner. One sentence can't describe all those changing details. The AI might get confused and make the car flicker or change shape weirdly.

The Tiled Prompt Fix for Video:
The system looks at a tiny moving chunk of the video (a tile in space and time). It generates a prompt like: "Close-up of a spinning silver wheel with mud splatters."
This ensures the motion looks smooth and the details (like the mud) appear exactly where they should, frame by frame.

The "Magic" Behind the Scenes

The paper proves mathematically that this method works better by reducing "ambiguity."

Analogy: Think of the AI as a detective trying to solve a crime.
- Global Prompt: "The crime happened in a city." (Too vague! Could be anywhere.)
- Tiled Prompt: "The crime happened in the alley behind the bakery on 5th Street." (Precise! The detective knows exactly where to look.)

By giving the AI precise, local instructions, they stop it from "hallucinating" (making things up) or getting confused.

The Bottom Line

The authors created a system that stops AI image upscalers from getting confused by giving them too much information at once. Instead, they break the image into small pieces and give the AI a customized, specific instruction for each piece.

The Result: Sharper images, clearer text on signs, smoother videos, and fewer weird artifacts (like ghost buses on sidewalks), all without slowing down the computer much. It's like swapping a blurry, general map for a set of perfect, turn-by-turn GPS directions for every single street corner.

1. Problem Statement: Prompt Misguidance in Latent Tiling

Modern high-resolution image and video super-resolution (SR) often relies on latent tiling strategies to manage memory constraints, processing large inputs as a grid of smaller, overlapping tiles. While text-conditioned diffusion models have advanced SR by using prompts as semantic priors, current pipelines typically employ a single global caption ( $c_{global}$ ) for the entire image or video sequence.

The authors identify a critical failure mode in this approach called Prompt Misguidance, which occurs when a coarse global prompt is applied to specific local tiles. This leads to two types of semantic errors:

Errors of Omission: The global prompt fails to include critical, localized high-frequency details (e.g., specific text on a sign, unique textures) that are only visible in a specific tile. This leaves the reconstruction of that tile ambiguous.
Errors of Commission: The global prompt includes semantic concepts relevant to other parts of the image/video but irrelevant to the current tile (e.g., describing "sky" for a tile showing "pavement"). This injects off-target semantics, causing the model to hallucinate incorrect details.

In Video Super-Resolution (VSR), this problem is exacerbated by temporal misguidance, where a static global prompt cannot capture dynamic changes, motion, or scene evolution across frames.

2. Methodology: Tiled Prompts Framework

The authors propose Tiled Prompts, a unified framework that replaces the single global prompt with dense, tile-specific prompts generated by a Vision-Language Model (VLM).

Core Mechanism

Instead of conditioning the SR model on $c_{global}$ for all tiles, the framework generates a unique prompt $c^{(i)}_{local}$ for each latent tile $i$ .

Image SR: A VLM analyzes each low-resolution (LR) tile $x^{(i)}_L$ and generates a descriptive prompt specific to that region's content.
Video SR: To address temporal misguidance, the VLM is provided with both the local spatio-temporal block and the full LR video sequence. This allows the VLM to understand the global context while generating a prompt that describes the specific local dynamics (e.g., "spinning wheels" rather than just "car").

Theoretical Foundation

The paper provides an information-theoretic analysis to prove that Tiled Prompts reduce misguidance:

Mutual Information Gap ( $\Delta I$ ): The authors define $\Delta I$ as the gap in mutual information between the ideal local prompt ( $c^*$ ) and the actual prompt used. They prove that using a global prompt creates a larger $\Delta I$ compared to using a tile-specific prompt.
KL Divergence Bound: They demonstrate that the accumulated prompt misguidance is lower-bounded by the KL divergence between the ideal and actual posteriors. By minimizing $\Delta I$ through localized prompting, the framework mathematically forces the generative trajectory to converge closer to the true high-frequency details, reducing hallucinations and artifacts.

Algorithm

The inference process (Algorithm 1) involves:

Splitting the LR input into a grid of tiles.
Using a VLM to generate a specific prompt for each tile.
Running the diffusion-based SR model on each tile conditioned on its specific prompt.
Aggregating the results using Gaussian blending to ensure seamless boundaries.

3. Key Contributions

Problem Definition: Formalized the concept of "Prompt Misguidance" in the context of latent tiling, categorizing it into errors of omission and commission.
Unified Framework: Introduced Tiled Prompts, a method that leverages VLMs to generate dense, localized textual guidance for both Image SR (SISR) and Video SR (VSR).
Theoretical Insight: Provided a rigorous information-theoretic proof showing that tile-specific prompts minimize the mutual information gap and the theoretical lower bound of prompt misguidance.
State-of-the-Art Performance: Demonstrated that the method improves perceptual quality and fidelity with negligible computational overhead.

4. Experimental Results

The framework was evaluated on real-world datasets (Urban100, OST300, LSDIR for images; VideoLQ, RealVSR, MVSR for videos) using various backbone models (DiT4SR, STAR, OSEDiff, DiffVSR).

Perceptual Quality: Tiled Prompts consistently outperformed global prompt baselines across no-reference metrics (NIQE, MUSIQ, MANIQA, CLIPIQA).
Text-Image/Video Alignment: Metrics like CLIP Score, ImageReward, and VQAScore showed significant improvements, confirming that the generated prompts are semantically aligned with the local visual content.
Reference-Based Metrics: On high-resolution ground truth datasets (LSDIR2048, SEPE8K), the method achieved better PSNR, SSIM, and lower FID/LPIPS scores, indicating higher fidelity and fewer artifacts.
Video Consistency: In VSR, the method significantly reduced temporal inconsistencies and improved video quality metrics (FasterVQA, DOVER) by capturing dynamic content that global prompts missed.
Efficiency: The additional computation for VLM-based prompt generation incurred negligible overhead (e.g., ~2% increase in inference time for image SR), making it a practical solution.
Robustness: The performance gains were observed to increase with higher Classifier-Free Guidance (CFG) scales, suggesting that Tiled Prompts are particularly effective when strong guidance is required.

5. Significance

This work addresses a fundamental bottleneck in scaling text-conditioned generative models to ultra-high resolutions. By shifting from a "one-size-fits-all" global prompt to a "context-aware" tiled prompting strategy, the authors enable diffusion models to reconstruct fine-grained details (such as text, logos, and complex textures) that were previously lost or hallucinated.

The method is model-agnostic (works with various diffusion backbones) and unified (applicable to both images and videos), offering a practical and theoretically grounded solution for next-generation super-resolution systems. It highlights the critical role of localized semantic understanding in generative reconstruction tasks.