💻 computer science

Geometrically Consistent Multi-View Scene Generation from Freehand Sketches

This paper introduces a novel framework for generating geometrically consistent multi-view 3D scenes from a single freehand sketch by leveraging a newly curated dataset, parallel camera-aware attention adapters, and sparse correspondence supervision to overcome the challenges of distorted 2D inputs and lack of training data, achieving superior realism, consistency, and inference speed compared to existing methods.

Original authors: Ahmed Bourouis, Savas Ozkan, Andrea Maracani, Yi-Zhe Song, Mete Ozay

Published 2026-07-21

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Ahmed Bourouis, Savas Ozkan, Andrea Maracani, Yi-Zhe Song, Mete Ozay

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an architect trying to build a house, but instead of blueprints, you only have a single, shaky doodle on a napkin. You've drawn a few lines for a roof, a squiggle for a door, and maybe a stick figure standing in front. To a human, this scribble is enough to imagine the house from the side, from the back, and even from above. Your brain automatically fills in the missing 3D details, understanding that the roof slopes down and the door has a back. But for a computer, this is a nightmare. Computers are usually very literal; they need precise photos or strict mathematical rules to understand depth. If you ask a standard AI to look at that napkin doodle and show you the house from a different angle, it often gets confused, producing a flat, distorted, or completely different building. This is the world of "generative AI," where computers try to create new images. The big challenge here is "geometric consistency"—making sure that if you walk around a computer-generated object, the back of the object matches the front, and the windows stay in the same place, even though the computer only saw a messy 2D drawing to start with.

This paper tackles a very specific and difficult version of that problem: Can we take a single, freehand sketch of a whole scene (like a street with a car and a person) and instantly generate a full 360-degree tour of that scene, where every angle looks real and fits together perfectly? The authors say yes, but they had to invent a new way to teach the computer how to "think" in 3D using only a 2D sketch. They built a system that doesn't just guess; it uses a special set of rules to ensure that if you see a dog's nose in the front view, the back of the dog's head appears in the back view, and they are in the exact right spot relative to each other. They call this "Sketch-to-Multi-View" generation.

The Napkin-to-3D Magic Trick

The researchers, working with Samsung Research and the University of Surrey, created a method that turns a single, shaky doodle into a full 3D movie. Imagine you draw a picture of a girl sitting on a horse. Usually, an AI might just make a pretty picture of that girl. But this new system takes that one drawing and spins it around, showing you the girl and horse from the front, the side, the back, and even from high up or low down, all in one go. The result is a set of images that look like you are walking around the scene, and crucially, the horse's legs don't disappear or change shape magically as you turn the corner.

Why Was This So Hard?

The authors point out that freehand sketches are the "hardest" kind of input for a computer. Unlike a photograph, which has shadows and perspective that tell the computer exactly where things are in 3D space, a sketch is full of "noise." The lines might be wobbly, the proportions might be weird, and the artist might have drawn a car that looks too big for the road. Previous methods tried to solve this by first turning the sketch into a perfect photo, and then turning that photo into 3D. But the authors argue this is like trying to translate a poem by first turning it into a grocery list and then back into a poem—it loses the soul and the details. They wanted to go straight from the messy sketch to the 3D world without that middle step.

How They Did It: The Three Secret Ingredients

To make this work, the team cooked up a recipe with three special ingredients that work together like a well-oiled machine.

1. The "Training Gym" (The Dataset)
First, they needed a place to teach the AI. There were no existing collections of "sketches paired with 360-degree views" because it's so hard to make them. So, they built their own gym. They took about 10,000 sketches and used a powerful AI to generate five different "realistic" photos for each sketch. Then, they acted like strict art critics, checking which generated photo matched the sketch best (using a score called mIoU). They picked the best one and used it to generate 33 different views of that scene. In the end, they curated a dataset of 9,222 high-quality examples. Think of this as the AI practicing on thousands of "what if" scenarios before it ever sees a real user's drawing.

2. The "Compass" (Camera-Aware Attention Adapters)
The second ingredient is a clever add-on to the AI's brain. Standard AI models don't really know what a "camera angle" is; they just see a grid of pixels. The researchers added a lightweight "adapter" (a small extra module) that acts like a compass. This compass tells the AI, "Hey, this part of the image is the front, and that part is the back." It uses a mathematical trick called "Projective Rotary Position Encoding" (PRoPE) to understand the relationship between different views. It's like giving the AI a mental map so it knows that if it's drawing the back of a chair, it shouldn't accidentally draw the front legs. This tiny addition only added about 2.7% more "brain power" (parameters) to the model but made a huge difference.

3. The "Spot Check" (Correspondence Supervision Loss)
The third ingredient is a strict teacher. Even with a compass, the AI might still get confused about which specific dots in the front view match up with dots in the back view. To fix this, the team used a technique called "Structure-from-Motion" (SfM). They took their generated 3D views and used a tool to find matching points (like the tip of a nose or a wheel) across different angles. Then, they taught the AI a "loss function" (a way to measure mistakes) that punished it if those matching points didn't line up. It's like a teacher checking a student's drawing of a cube from the front and the side, and saying, "If the corner of the roof is here in the front, it must be here in the side view, or you get a red mark." This forced the AI to learn the rules of 3D geometry explicitly.

The Results: Fast, Real, and Consistent

When they tested their new method, the results were impressive. They compared their "one-step" approach (sketch straight to 3D) against the old "two-step" methods (sketch to photo, then photo to 3D).

Speed: Their method was much faster, taking only about 50 seconds to generate a full set of views, while the old methods took minutes or even half an hour.
Realism: The images looked more realistic. In a test called FID (which measures how close the images are to real photos), their method scored 18.49, while the next best was over 46. A lower score is better, meaning their images were much closer to reality.
Consistency: Most importantly, the 3D structure held up. Their method got a "Corr-Acc" (geometric consistency) score of 0.199, beating the others. This means the objects in the generated scenes stayed in the right places as the camera moved around.

What They Didn't Do (And Why It Matters)

The authors were careful to rule out some common shortcuts. They showed that simply adding a "LoRA" (a standard way to tweak AI models) wasn't enough; without their special "compass" and "spot check," the 3D views fell apart. They also proved that their method didn't just work on the specific sketches they trained on. When they tested it on sketches from other datasets (like the TU-Berlin dataset, which has a very different drawing style), the AI still managed to create consistent 3D views. This suggests the AI actually learned the rules of 3D space, not just memorized the drawings.

The Bottom Line

This paper suggests that we can finally turn a simple, messy doodle into a full 3D world without needing a perfect photo or a slow, step-by-step process. By combining a massive new dataset, a smart "compass" for camera angles, and a strict "spot check" for matching points, the authors created a system that is faster and more accurate than previous attempts. While the system isn't perfect yet (it works best at 480x480 resolution and relies on generated training data), it opens the door for a future where anyone can sketch a scene on a napkin and instantly explore it in 3D, turning our wildest scribbles into virtual worlds.

Technical Summary: Geometrically Consistent Multi-View Scene Generation from Freehand Sketches

Problem Definition

The paper addresses the task of Sketch-to-Multi-View (S2MV) generation: synthesizing a set of geometrically consistent, photorealistic novel views representing a complete 3D scene, conditioned solely on a single freehand sketch and a text caption.

This problem presents three compounding challenges that existing methods fail to address simultaneously:

Data Scarcity: No existing dataset pairs freehand scene sketches with geometrically consistent multi-view images. Freehand sketches introduce non-linear distortions and abstraction that conflict with standard 3D rendering pipelines.
Geometric Reasoning from Distorted Input: Unlike photographs (which provide dense appearance and implicit 3D cues) or text (which lacks spatial constraints), freehand sketches are spatially informative yet geometrically unreliable. Models must infer coherent 3D structures from 2D inputs lacking strict projective alignment.
Cross-View Consistency: Generating multiple views simultaneously requires enforcing correspondences between views rather than conditioning on viewpoints independently. Existing sketch-to-3D methods typically require multiple input views or costly per-scene optimization, while multi-view diffusion methods usually start from high-fidelity photographs.

Methodology

The proposed framework synthesizes all $N$ views in a single end-to-end denoising process using a Video Diffusion Transformer (DiT), bypassing intermediate photorealistic generation or iterative refinement. The architecture consists of three core components:

1. S2MV Dataset Construction Pipeline

To address the lack of training data, the authors curate a dataset of 9,222 sketch-to-multi-view samples via an automated pipeline:

Multi-Seed Generation: For each sketch in the FS-COCO dataset, five photorealistic candidate images are generated using FLUX.2-dev.
Alignment Verification: Semantic segmentation masks are extracted from both the sketch and candidates. The candidate with the highest mean Intersection-over-Union (mIoU) against the sketch mask is selected to ensure spatial fidelity.
Multi-View Synthesis: The selected frontal view is processed by Qwen Image Edit Angles to synthesize $N$ views at varying azimuth and elevation.
Curation: Samples with poor alignment are filtered out, resulting in a rigorous training corpus.

2. Camera-Aware Attention Adapters (CA3)

To inject geometric inductive biases into the pretrained video DiT (Wan 2.1), the authors introduce lightweight parallel adapters:

Virtual Camera Construction: Since ground-truth camera parameters are unavailable for generated images, virtual cameras are constructed by placing viewpoints on a sphere of fixed radius around the scene origin.
Projective Rotary Position Encoding (PRoPE): The adapters modulate attention queries, keys, and values using projective matrices derived from the virtual camera extrinsics and intrinsics. This embeds epipolar constraints directly into the attention computation, allowing the model to reason about relative projective geometry between views.
Efficiency: The adapters add only ~2.7% parameters (approx. 35.4M) and utilize a zero-initialized parallel branch to preserve the pretrained generative priors.
Domain Adaptation: LoRA (Low-Rank Adaptation) is applied to the backbone's self-attention layers to adapt the video-trained features to the sketch domain.

3. Correspondence Supervision Loss (CSL)

To enforce cross-view geometric consistency, a sparse loss is introduced:

SfM-Derived Targets: Structure-from-Motion (COLMAP) is run on the pseudo ground-truth multi-view images to extract sparse 3D correspondences.
Sparse InfoNCE Loss: Instead of supervising the full attention matrix (which is memory-prohibitive), the loss applies a contrastive objective to the adapter's query and key projections at specific layers. It pulls together embeddings of corresponding spatial patches across views and pushes apart non-corresponding ones.
Curriculum Training: The correspondence loss weight ( $\lambda_{corr}$ ) is ramped up after an initial warmup phase to allow domain adaptation to stabilize first.

Key Results

The method is evaluated on the S2MV test set (477 samples, 33 views each) against two-stage baselines (FLUX.2-dev for sketch-to-photo, followed by SEVA or ViewCrafter for multi-view synthesis).

Geometric Consistency: The proposed method achieves a Corr-Acc of 0.199, significantly outperforming SEVA (0.161) and ViewCrafter (0.136). This represents a 23% improvement in geometric consistency.
Realism: The method achieves an FID of 18.49, a >60% improvement over baselines (46–48), indicating higher perceptual realism.
Efficiency: The single-stage approach provides a 3.7× inference speedup over SEVA and a 42× speedup over ViewCrafter.
Ablation Studies:
- Removing CSL degrades Corr-Acc (0.175) and LPIPS, confirming the necessity of explicit supervision for consistency.
- Removing CA3 causes catastrophic failure (PSNR drops to 5.0, FID to 266), proving that LoRA alone cannot handle multi-view geometry.
- Removing LoRA slightly degrades consistency (Corr-Acc 0.188), showing that domain adaptation complements geometric reasoning.
- Removing Frame Replication doubles the FID, highlighting the importance of preserving view identity through the VAE bottleneck.

Significance and Claims

The paper claims to present the first method capable of generating geometrically coherent, photorealistic multi-view scenes from a single freehand scene sketch in a single forward pass.

Novelty: It bridges the gap between sketch-based generation and multi-view synthesis, a task previously unattempted due to the lack of data and the difficulty of reasoning from distorted 2D inputs.
Paradigm Shift: The work demonstrates that combining camera-aware attention adapters with explicit correspondence supervision is an effective paradigm for lifting 2D generative models into geometrically grounded multi-view generators.
Accessibility: By enabling 3D content creation via simple freehand sketches without per-scene optimization or reference photographs, the method offers a lightweight and accessible interface for 3D generation.

The authors note limitations regarding the training scale (9k samples vs. millions for baselines), the use of generated rather than rendered ground truth, and the resolution (480×480), but assert that the proposed components are mutually reinforcing and essential for the task.