💻 computer science

Beyond Text Prompts: Visual-to-Visual Generation as A Unified Paradigm

This paper introduces **V2V-Zero**, a training-free framework that enables visual-to-visual generation by conditioning existing vision-language models on visual specification pages instead of text prompts, demonstrating that this unified paradigm achieves competitive performance while revealing current limitations in content generation and structural control across image and video models.

Original authors: Yaofang Liu, Kangning Cui, Meng Chu, Zhaoqing Li, Suiyun Zhang, Jean-Michel Morel, Xiaodong Cun, Haoxuan Che, Rui Liu, Raymond H. Chan

Published 2026-05-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yaofang Liu, Kangning Cui, Meng Chu, Zhaoqing Li, Suiyun Zhang, Jean-Michel Morel, Xiaodong Cun, Haoxuan Che, Rui Liu, Raymond H. Chan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Stop Describing, Start Showing

Imagine you are an architect trying to tell a builder what to construct.

The Old Way (Text-to-Image): You write a long, detailed letter describing the house. You say, "It should be blue, with three windows on the left, a red door, and a chimney shaped like a cat." The builder (the AI) has to read your words, guess what "blue" looks like, imagine the "cat-shaped chimney," and hope they get the layout right. Often, they get it wrong because words are fuzzy.
The New Way (Visual-to-Visual / V2V): Instead of a letter, you hand the builder a blueprint. This blueprint isn't a picture of the finished house to copy; it's a specification sheet. It has a blue paint swatch, a sketch of the cat-chimney, a diagram showing exactly where the windows go, and a photo of the red door. The builder looks at this sheet and builds exactly what is shown.

This paper introduces V2V-Zero, a method that lets you give AI generators these "blueprints" (visual pages) instead of text prompts.

How It Works: The "Magic Translator"

The researchers didn't build a new AI from scratch. Instead, they found a clever shortcut using existing technology.

The Two-Step Process:
- Imagine you have a Translator (a Vision-Language Model, or VLM) who is great at looking at pictures and understanding what they mean.
- Imagine you have a Builder (a Diffusion Model) who is great at making images but usually only listens to the Translator's spoken words.
- The Trick: The researchers realized that the Translator already turns pictures into a secret "code" (hidden states) that the Builder understands. Usually, the Translator only turns text into this code.
- The Innovation: They simply told the Translator, "Hey, look at this visual blueprint page instead of a text prompt. Give the Builder the code you made from the picture."
- Result: The Builder receives the blueprint's "code" and builds the image based on the visual instructions, without needing to be retrained. It's like swapping the language the Builder listens to, without changing the Builder's ears.

The "Zero-Shot" Superpower

The paper calls this "Zero-Shot" and "Training-Free."
Think of it like giving a chef a new recipe card. Usually, to teach a chef a new style, you have to send them to culinary school (training). Here, the researchers just handed the chef a new card with pictures instead of words. The chef already knew how to read the ingredients (the visual code); they just needed to be shown the ingredients in a different format. The chef didn't need to go to school; they just needed a new menu.

What They Tested (The "Simple-V2V Bench")

To see if this actually works, they created a test called Simple-V2V Bench. It's like a driving test for AI, but instead of driving a car, the AI has to follow visual instructions.

They tested seven types of "blueprints":

Inline Color: A tiny blue square in the instructions.
Visual Reference: A photo of a specific dog to copy.
Visual Text: A word written in a specific font to reproduce.
Counting: A picture showing exactly 3 apples.
Style: A painting to copy the style of.
Pose: A skeleton drawing of a person's pose.
Sketch: A rough drawing to turn into a real image.

The Results: The "Three-Tier" Reality

The results were a mix of "Wow!" and "Not yet." They found a clear hierarchy of difficulty:

The Easy Stuff (Strong): The AI was very good at binding attributes. If you showed a blue square, it made a blue object. If you showed a specific dog photo, it made that dog. It could also copy text styles well.
- Analogy: The builder is great at painting the walls the exact color you swatched.
The Hard Stuff (Unreliable): When asked to generate specific content based on complex visual cues, it struggled.
- Analogy: The builder sometimes paints the wrong number of windows or puts the door on the roof.
The Hardest Stuff (Difficult): Structural control (like following a pose sketch or a layout diagram) was the weakest link. Even the best commercial systems (like GPT Image 2) struggled to follow a stick-figure drawing perfectly.
- Analogy: The builder often ignores the floor plan and builds the house upside down, even if the colors are right.

The Video Extension

They also tried this on a video generator (HunyuanVideo). They handed it a visual blueprint and asked for a video. It worked, but the results were even lower than the images.

Analogy: Asking the builder to build a moving house based on a blueprint is even harder than building a static one. The builder got the colors right but messed up the movement.

The "Secret Sauce" Discovery

The researchers looked under the hood to see why it worked. They found that the AI wasn't actually "thinking" about the picture and turning it into a sentence in its head.

The Finding: 95% of the time, the AI was looking directly at the visual code from the blueprint. It wasn't ignoring the picture to write a description first; it was reading the picture's "DNA" directly.
Analogy: It's like the builder isn't reading a letter describing the house; they are looking at the blueprint's wiring diagram and following the wires directly.

Summary

This paper proposes a new way to talk to AI: Show, don't tell.

What it is: A method to use visual pages (blueprints with colors, sketches, and photos) as instructions instead of text.
How it works: It uses existing AI tools in a clever way, swapping text inputs for visual inputs without needing to retrain the models.
What it can do: It's excellent at copying colors, styles, and specific objects.
What it can't do yet: It still struggles with complex layouts, exact counts, and following detailed pose sketches.

The paper concludes that while we are currently in a "text-first" world, the technology is already ready to understand "visual-first" instructions, opening the door for a future where we design with pictures rather than paragraphs.

Technical Summary: Beyond Text Prompts: Visual-to-Visual Generation as A Unified Paradigm

Problem Statement

Current visual generation systems predominantly rely on a "text-first" interface, requiring users to serialize complex visual intents—such as spatial layouts, exact color swatches, typography, and structural sketches—into natural language prompts. The authors argue that this paradigm creates a bottleneck: natural language compresses critical visual signals (e.g., precise glyph shapes, exact spatial relations, and specific color bindings) that are more naturally and directly represented as visual artifacts. While modern text-to-image (T2I) and text-to-video (T2V) models have advanced rapidly, they often struggle to reconstruct intended visual constraints when those constraints are lost during the translation from visual intent to text tokens.

Methodology: V2V-Zero

The paper proposes Visual-to-Visual (V2V) generation, a paradigm where the user conditions a generative model using a structured visual specification page rather than a text prompt. This page acts as a conditioning document containing rendered text, sketches, color chips, reference images, and spatial diagrams, rather than an image to be edited or reconstructed.

To realize this without retraining, the authors introduce V2V-Zero, a training-free framework that leverages existing Vision-Language Models (VLMs) and diffusion generators. The core architectural insight is that modern T2I/T2V systems (e.g., Qwen-Image) already utilize a VLM conditioning encoder that maps both text and images into a shared hidden-state space consumed by the diffusion transformer (DiT).

Key Technical Components:

Frozen Architecture: V2V-Zero operates on frozen weights. It does not train adapters or fine-tune the generator.
Visual Page Encoding: A user-provided visual page $V$ is processed by a frozen VLM (e.g., Qwen2.5-VL). The VLM treats the page as an input sequence, potentially generating intermediate reasoning tokens.
Hidden State Injection: Instead of feeding text embeddings to the generator, V2V-Zero extracts the final-layer hidden states from the VLM and injects them directly into the generator's conditioning interface.
Conditioning Modes:
- Image-HS-only: Injects only the visual token states from the page.
- FULL-FINAL (Default): Injects both the visual token states and the final-layer hidden states of the VLM's generated reasoning tokens. This mode preserves the visual evidence while aligning the page with a generation-ready scene interpretation.
Mechanism: The framework relies on the observation that the DiT's cross-attention mechanism is already trained to attend to VLM hidden states. By replacing text prompts with visual-page hidden states, the generator accesses the same conditioning space without architectural changes.

Key Contributions

V2V Paradigm Formulation: The paper defines V2V as a unified generation paradigm where structured visual inputs serve as the primary conditioning signal, distinct from image editing or task-specific control modules.
V2V-Zero Framework: A zero-shot, training-free method that achieves competitive performance by routing visual-page hidden states through existing VLM-conditioned generators.
Simple-V2V Bench: A novel benchmark comprising seven visual-conditioning tasks (visual text, inline color, inline visual reference, counting, style transfer, pose control, and sketch reference) evaluated across seven models, including commercial systems (GPT Image 2, Seedream 5.0 Lite) and open-weight models.
Mechanistic Analysis: Empirical evidence showing that in the "reasoning path" (FULL-FINAL), the DiT assigns 95.0% of its conditioning-token attention mass to the visual-prefix hidden states rather than the generated reasoning tokens, confirming that the system relies on direct visual conditioning rather than text re-encoding.

Experimental Results

GenEval Benchmark:

V2V-Zero achieves an overall score of 0.85 on GenEval using a frozen Qwen-Image backbone.
This performance closely matches the backbone's optimized text-to-image (T2I) performance (0.87) without any fine-tuning, demonstrating that visual conditioning can effectively replace text prompts for compositional tasks.

Simple-V2V Bench:

V2V-Zero scores 32.7/100 overall.
Capability Hierarchy: The benchmark reveals a three-tier structure:
1. Attribute Binding: Strong performance (e.g., 76.9/100 on inline color), approaching commercial baselines.
2. Content Generation: Unreliable performance for complex semantic generation.
3. Structural Control: Significant difficulty in pose control (13.3/100) and sketch reference (16.6/100), even for state-of-the-art commercial systems.
Video Extension: Applying the same V2V-Zero method to HunyuanVideo-1.5 yields a score of 20.2/100, demonstrating the framework's transferability to T2V generation, though with lower performance than image generation.

Significance and Claims

The paper claims that V2V-Zero identifies a latent visual-to-visual route already present in VLM-conditioned generators. The significance lies in demonstrating that:

Zero-Shot Viability: Existing models can accept visual specification pages as direct conditioning inputs without retraining, challenging the necessity of text-only interfaces.
Architectural Alignment: The results suggest that the "reasoning path" in these models is primarily visually routed; the generator attends to the visual representation of the page rather than relying on the VLM to convert the page into a textual rationale.
Future Direction: The work posits that visual specification should be treated as a "first-class conditioning language." While current zero-shot methods show promise, the paper suggests that native generators trained end-to-end to treat visual input as a primary conditioning signal represent the natural next step, particularly for improving structural control and precise visual instruction following.

The authors remain modest regarding the current limitations, acknowledging that while attribute binding is strong, structural control (pose, sketch) remains a difficult regime even for commercial systems, and the video extension is a preliminary validation rather than a comprehensive video benchmark.